Dead T4, unusual symptoms

[prl]

Our in-use T4 died yesterday afternoon. It just stopped running while my wife was watching a recording. She thought that it had simply taken over to do recordings, but it wouldn't power on when we wanted to watch some recordings in the evening.

In its current state, when it's powered on from the rear mains switch the blue power/remote LED on the front panel switches to full brightness for about 1/2 sec, and then goes to the half-brightness level. The Booting... text on the OLED display isn't shown, the display stays black. There is no response to either the remote or the front panel power buttons.

I initially suspect a power supply problem, but I swapped the power supply with a known-good test T4. The in-use T4 stayed unable to be powered up and the test T4 functioned normally.

I've now also swapped the disks between the two, and the test T4 operates as I'd expect it to, with the power supply and the HDD from the in-use T4.

So I guess that means a mainboard (possibly flash memory?) or front panel problem. I'd be willing to try swapping the front panels to try to eliminate them as the cause (or demonstrate it), but I don't know how to disconnect the ribbon cable that connects the front panel with the mainmoard.

Has anyone seen these sorts of symptoms or have any suggestions for further tests that might isolate the problem? I guess my next step should be to see what (if anything) the serial out on the back says.

Also does anyone know how to disconnect/reconnect the ribbon cable between the front panel and the mainboard?

I'm beginning to think that the T4 is going to need a trip to WA to see Warkus

[prl]

I guess my next step should be to see what (if anything) the serial out on the back says.

Tried that. All I got on from the serial port was "..", which I think is just startup noise on the line. This seems to confirm that the boot loader isn't running (which was likely because the "Booting..." text never appeared on the front panel, either).

[Paul_oz53]

I guess my next step should be to see what (if anything) the serial out on the back says.

Tried that. All I got on from the serial port was "..", which I think is just startup noise on the line. This seems to confirm that the boot loader isn't running (which was likely because the "Booting..." text never appeared on the front panel, either).

I was about to send my T4 to Warkus because it got into a state where the "Booting..." text appeared but froze. I left it off for a few days before speaking to Warkus. I was about to package it up when I plugged it in one last time and lo and behold, it magically came back to life!

IanSav similarly plugged in his dead T4 after a very long time off supply and reported it too had risen from the dead

Only an uninformed guess but it seems that waiting until all residual charge is lost from mainboard capacitors mysteriously unlocks the T4. Depending on how determined you are, you could try shorting the caps manually or the extended off treatment but the latter is pretty unpredictable - Ian's took months whilst mine was mere days.

Failing that, Warkus.

[prl]

I have sort-of tried the "leave it off for an extended time" trick inadvertently, It was powered off overnight from about 8pm last night to about 10:30am this morning. Of course, that's a lot shorter time than you and (especially IanSav) left it off for, but I'd have thought the time constants for all the capacitors would be a lot less than 24 hours.

Also, my T4 is locking up at a different point in the start-up sequence than yours was.

But I'll give it a go leaving it for a few days before trying it again. Fortunately, I have a workable backup.

[IanSav]

Hi,

I suspect my T4 was left off far longer than one would technically expect. It is odd to find devices that need some real down time before they come back to life.

Regards,
Ian.

[prl]

Are there any recommended incantations?

[Paul_oz53]

I have sort-of tried the "leave it off for an extended time" trick inadvertently, It was powered off overnight from about 8pm last night to about 10:30am this morning. Of course, that's a lot shorter time than you and (especially IanSav) left it off for, but I'd have thought the time constants for all the capacitors would be a lot less than 24 hours.

Also, my T4 is locking up at a different point in the start-up sequence than yours was.

But I'll give it a go leaving it for a few days before trying it again. Fortunately, I have a workable backup.

I had left mine unplugged for shorter periods with no change in behaviour. Totally bricked - not able to recognise a usb and frozen display. Can't be absolutely sure now that mine did say "booting..." - may be a figment of my imagination with the passage of time. Regardless, no harm in trying it if there are workarounds available. My own setup has five Wizzes and I record everything on at least two machines so as to minimise the risk of missing out. Bit tedious swapping things in the IceTV app when I swapped the T4 and U4 over though.

Totally agree it is bizarre and I too would have thought the caps would have drained after a few hours. I've no way of knowing but residual charge holding something high is about the only theory I could think of to explain it rising from the dead. IanSav's case is even stranger!

If yours recovers too, I think I will start a new religion based on abstinence for former Wiz addicts! I'll need to consult the gods before I divulge the secret chant, lest I be denied the chance to become the oracle...

[prl]

Bit tedious swapping things in the IceTV app when I swapped the T4 and U4 over though.

Been there, done that, when we switched from DP-Lite + DP-P2 to T4 (at the time, My Series had about 400 entries). Today I just went through the week duplicating the in-use T4's recordings from it to the stand-in.

Totally agree it is bizarre and I too would have thought the caps would have drained after a few hours. I've no way of knowing but residual charge holding something high is about the only theory I could think of to explain it rising from the dead. IanSav's case is even stranger!

If yours recovers too, I think I will start a new religion based on abstinence for former Wiz addicts! I'll need to consult the gods before I divulge the secret chant, lest I be denied the chance to become the oracle...

We won't actually be abstaining, so if I'm successful resurrecting the old in-use T4, perhaps we can rule out abstinence and just stick to the chants and taking the auspices.

[Grumpy_Geoff]

..., perhaps we can rule out abstinence and just stick to the chants and taking the auspices.

Will the goats be safe?

[prl]

..., perhaps we can rule out abstinence and just stick to the chants and taking the auspices.

Will the goats be safe?

Perfectly. You may be thinking of haruspicy. Animals are generally not harmed in taking auspices.

[Grumpy_Geoff]

..., perhaps we can rule out abstinence and just stick to the chants and taking the auspices.

Will the goats be safe?

Perfectly. ...

Good, then can you send 'em over my way, as today fortnight I get connected to FTTN, I may need all the help I can get, I'm dreading any potential looming RSP & NBN cockups (there's already been one NBN cockup, and we all know the mess Turnbull & Abbot made of the MTM).

[prl]

I'm in the fortunate position that I'm on iiNet's own VDSL2 network in Canberra, not on the NBN. I get about 43Mb/sec down link sync speed and it's usually not congested (just got 40Mb/s actual download, which probably as good as it gets for that link speed).

Aussie Broadband's NBN line speed estimator says I might get between 32 and 79Mbps (that range is in itself not reassuring). If I stay with iiNet, it would cost me 55% more than my current (grandfathered) plan ($110 vs $70) and deliver me somewhere between 26% slower and 83% faster sync (such is the nature of the node lottery) than what I get now, which is fine for my needs (on ADSL2 I got 4Mb/s). I have no idea what congestion's like on the NBN in my area.

[Grumpy_Geoff]

So you're on the old TransACT network then? It sounds like you should stay on what you have, and hope you don't get kicked off it.

An address check gives me "the nominal access line speed of your service is likely to be between 25 and 58Mbps." (I'm pretty sure those are post-co-existence figures).
I expect my best will be about what you get now, as I know I've got crap street wiring as well as a crap in-home setup with multiple phone points, central splitter, mode3 dialler (in other words, it's a crap environment).
I currently get an ADSL sync of around 5Mb/s, my distance to the pillar/node will be around 650 metres.
So if I pay to clean up my in-home wiring I might get to 40Mb/s during the co-existence period. I'll have to toss that around to decide whether the extra cost of cleanup plus additional 50Mb/s plan cost is worth it or just suck it up and be on the 25Mb/s plan.

My next door neighbour and I share the same pit, and the other side of my neighbour is the boundary of a different DA, which is serviced by FTTC (FTTdp). It's frustrating that less than 60 metres pit-to-pit distance for me brings about a 60Mb/s difference in capability

[prl]

So you're on the old TransACT network then? It sounds like you should stay on what you have, and hope you don't get kicked off it.

Yes, that's the one. In an ideal world, iiNet would put VDSL2 line cards in all their nodes (where they used to have the VDSL1 line cards) rather than just in their "supernodes". They've done in in quite a number of places, but I suspect that it's only in areas where there's enough customer demand that they've run out of copper between the supernodes and the end users. I'm not holding my breath for that one. I check my line speed every few months to see if anything's happened, but I think I'm looking in vain.

An address check gives me "the nominal access line speed of your service is likely to be between 25 and 58Mbps." (I'm pretty sure those are post-co-existence figures).

Whose estimator did you use? Previously I've used Optus's, but yesterday I couldn't find it where I expected it, so I used ABB's. The width of the range in both your and my estimates says a lot about the variability of the copper quality.

I expect my best will be about what you get now, as I know I've got crap street wiring as well as a crap in-home setup with multiple phone points, central splitter, mode3 dialler (in other words, it's a crap environment).
I currently get an ADSL sync of around 5Mb/s, my distance to the pillar/node will be around 650 metres.
So if I pay to clean up my in-home wiring I might get to 40Mb/s during the co-existence period. I'll have to toss that around to decide whether the extra cost of cleanup plus additional 50Mb/s plan cost is worth it or just suck it up and be on the 25Mb/s plan.

My next door neighbour and I share the same pit, and the other side of my neighbour is the boundary of a different DA, which is serviced by FTTC (FTTdp). It's frustrating that less than 60 metres pit-to-pit distance for me brings about a 60Mb/s difference in capability

Yes, instead of them "wrecking your rose garden" to install FTTP, you get to re-wire your house. And participate in the great MTM node lottery.

[Grumpy_Geoff]

An address check gives me "the nominal access line speed of your service is likely to be between 25 and 58Mbps." (I'm pretty sure those are post-co-existence figures).

Whose estimator did you use? Previously I've used Optus's, but yesterday I couldn't find it where I expected it, so I used ABB's. The width of the range in both your and my estimates says a lot about the variability of the copper quality.

I think the Optus SQ checker went non-public a while back.
I got my between 25-58Mb/s range from the MyRepublic address checker - https://myrepublic.net/au/personal/nbn/
The ABB POI address checker doesn't give a speed estimate for me - https://www.aussiebroadband.com.au/nbn-poi/
The Future Broadband checker returns "NBN Speed Estimate: NBN advises the expected speed will be 58 Mbps." - https://futurebroadband.com.au/sq/

[peteru]

Also does anyone know how to disconnect/reconnect the ribbon cable between the front panel and the mainboard?

Pull it out with an even force and without wiggling / bending.

The T4 power supply design is different to the T3. The T4 PSU just supplies 12V. The T4 motherboard then does all the DC-DC conversions required. The most likely scenario is that one or more of the many power regulators has gone out of spec.

Inspect the motherboard for signs of thermal damage. Try to locate as many voltage regulators on the board and put a multimeter on them to see if the input and output are both in range.

[peteru]

It is odd to find devices that need some real down time before they come back to life.

There's a known problem with Helium (and possibly other light gasses) that can affect the performance of semiconductors for extended periods of time. It's entirely possible that a popped balloon from a children's party can take an iPhone out of action for about a week.

[prl]

It is odd to find devices that need some real down time before they come back to life.

There's a known problem with Helium (and possibly other light gasses) that can affect the performance of semiconductors for extended periods of time. It's entirely possible that a popped balloon from a children's party can take an iPhone out of action for about a week.

The reports of this appear to be restricted to MEMS (microelectromechanical systems) oscillators and other MEMS devices, and all the reports I found of instances of the problem were specific to the MEMS oscillators in Apple iPhone 6 and newer and Apple Watches. The main event that is talked about was a leak of ~120 litres of liquid helium into a relatively enclosed area (according to what looks like a first-hand report, that's about "90,000 m3 of gaseous He"). So, in that case, a lot more He than in a popped balloon.

In the Apple case, the particular device affected was an SiTime SiT512, "the world's smallest, lowest power 32 kHz oscillator" [my emphasis].

The mechanism is apparently helium diffusion into what should otherwise be an evacuated space affecting the ability of the oscillator to start. The fix is to put the device into an atmosphere with low helium partial pressure (like normal air) and allow the helium time to diffuse back out again.

There's a fairly good run-down of the issue in this article, including an electron micrograph of the SiT512.

Do Beyonwizes have system-critical MEMS oscillators (or any other system-critical MEMS devices)? Or is here an unrelated effect of helium on semiconductor devices?

[prl]

Also does anyone know how to disconnect/reconnect the ribbon cable between the front panel and the mainboard?

Pull it out with an even force and without wiggling / bending.

And putting it back in?

The T4 power supply design is different to the T3. The T4 PSU just supplies 12V. The T4 motherboard then does all the DC-DC conversions required. The most likely scenario is that one or more of the many power regulators has gone out of spec.

Inspect the motherboard for signs of thermal damage. Try to locate as many voltage regulators on the board and put a multimeter on them to see if the input and output are both in range.

Thanks. I'll have a look. I only really looked at the condition of the power supply.

Oh, and no helium anywhere the T4 as far as I know, other than the normal atmospheric amount.

[peteru]

And putting it back in?

Same deal. Insert carefully so that you don't kink the ribbon cable. If you are finding that the friction is excessive, a tiny amount of lubrication on the cable should help.

Oh, and no helium anywhere the T4 as far as I know, other than the normal atmospheric amount.

I was not suggesting that the T4 is likely to suffer from He poisoning. The comment was in the context of electronic devices needing a long recovery time in some cases. Much longer than a simple capacitor discharge. Apple's tech advice suggests leaving the device powered down for a week in a well ventilated place.

[peteru]

In terms of where to check...

U4 - voltage regulator. Pin 5 should have 12V, pin 6 should have 3.3V
U14 - MOSFET that turns on the 12V power supply. Pin 1 should have 12V at all times, pin 8 should have 12V when powered on.
U25 - MOSFET that turns on the 5V power supply. Pin 1 should have 5V, pin 8 should have 5V when powered on.
U32 - voltage regulator. Pin 8 should have 12V, pin 10 should have 5V
U23 - voltage regulator. Pin 8 should have 12V, pin 10 should have around 1V, but this will vary due to the AVS tuning done by the SoC.

Those are the most likely places to see a failure. If everything checks out there, I can either point you at a few more places to investigate, or you can decide to ship it off to WA.

[prl]

Thanks. I'll do some investigation tomorrow.

[prl]

...
U25 - MOSFET that turns on the 5V power supply. Pin 1 should have 5V, pin 8 should have 5V when powered on.
...
Those are the most likely places to see a failure. If everything checks out there, I can either point you at a few more places to investigate, or you can decide to ship it off to WA.

I haven't tested the voltages, but the top surface of U25 looks like it might be heat damaged to me, and there's some browning of the PCB near one end of U25. The top surface of Q11 (near U25) also looks heat damaged.

The goop that holds the heat sinks on U44/45/46/47 is also dark brown, so If I get Mark to look at the board, they could probably do with a look at, too.

Where's a good place to pick up earth on the mainboard (or power supply) to measure U25's voltages from?

I don't have any probes that could test the voltages on either U23 or U32 from the component side of the mainboard, and in any case, I can only see 7 pins on them on that side of the PCB.

[peteru]

the top surface of U25 looks like it might be heat damaged to me, and there's some browning of the PCB near one end of U25. The top surface of Q11 (near U25) also looks heat damaged.

That's probably the point of failure, although the failure could be due to something else drawing excessive current. Q11 and U25 are used to control STANDBY (the power-off kind of standby). I leave the T4s powered on all the time and I don't see any evidence of heat damage anywhere in that vicinity. Maybe regularly shutting down your T4 exercises these parts more and that's why they failed.

Where's a good place to pick up earth on the mainboard (or power supply) to measure U25's voltages from?

I usually strap one of the multimeter probes to the chassis or HDD bracket.

I don't have any probes that could test the voltages on either U23 or U32 from the component side of the mainboard, and in any case, I can only see 7 pins on them on that side of the PCB.

The output may be easier to measure on the inductor side. For U32 output / U25 input, measure at L28, looking for 5V. For U23 output, measure at L23, looking for something in the 0.95 - 1.0V range.

[prl]

OK. Thanks.

[stevebow]

the top surface of U25 looks like it might be heat damaged to me, and there's some browning of the PCB near one end of U25. The top surface of Q11 (near U25) also looks heat damaged.

That's probably the point of failure, although the failure could be due to something else drawing excessive current. Q11 and U25 are used to control STANDBY (the power-off kind of standby). I leave the T4s powered on all the time and I don't see any evidence of heat damage anywhere in that vicinity. Maybe regularly shutting down your T4 exercises these parts more and that's why they failed.

I boot up from deep standby several times a day on my T4. I don't see any heat damage on U25 or Q11 at all, and the PCB looks perfectly normal in that vicinity.

[prl]

U14 - MOSFET that turns on the 12V power supply. Pin 1 should have 12V at all times, pin 8 should have 12V when powered on.
U25 - MOSFET that turns on the 5V power supply. Pin 1 should have 5V, pin 8 should have 5V when powered on.

Measured when state should be "running", powered up from mains off. Actual state unknown:
U14 (possible visible heat damage)- Pin 1: 12.5V Pin 8: 12.5V
U25 (no visible heat damage) - Pin 1: 2.35V Pin 8: 0V

It looks as though the 5V switching regulator that I think Q11 (possible visible heat damage) and U32 (no visible heat damage) are part of is pretty sick.

[peteru]

It looks as though the 5V switching regulator that I think Q11 (possible visible heat damage) and U32 (no visible heat damage) are part of is pretty sick.

Maybe. It's also possible that there is a problem somewhere downstream and the regulator internal protection is being triggered. Given the visible damage to Q11 and U25, combined with the fact that they have no protection, I'd start with those. Then again, if you don't have the means to effect the repair yourself, you might as well pack up the motherboard and send it off. Whoever will repair it for you will appreciate the initial diagnostic, but at the end of the day they will do their own troubleshooting.

[prl]

Thanks.

I don't have the equipment or skills to do any repairs beyond old-fashioned through-board components like 0.1" DIL packaging.

I'll contact Mark about getting him to do the repairs.

[prl]

I'll contact Mark about getting him to do the repairs.

Done.