T3 crashes every few days

[togo]

The T3 2TB crashes and reboots for no apparent reason. Crashes are happening every day or two, and sometimes more often.

I thought the HDD was failing, as a crash would occur during disk activity, the titles of new recordings on disk were getting scrambled into non-alphabetical characters, some new recordings had random blank periods, and the video on some playbacks was distorted.

As a precaution I added a USB disk, copied stored movies from the HDD to it, and made it the default movie location.

I've attached the most recent crash log. A META file is mentioned in the log, and I tried to attach a copy of that too, but an 'invalid file extension' message showed. The same file is mentioned in each crash log over a few weeks. The file is probably corrupted: I could connect to the HDD and get a copy, but File Commander would not copy it between drives. I wasn't playing the .TS file for this before the unit crashed.

The T3 will be five years old in a few weeks, so perhaps it is just ageing.

Any ideas?

Togo

[Grumpy_Geoff]

I'd be using File Commander to delete the file '20170530 1450 - SBS VICELAND - Cyberwar.ts.meta' and see what then happens.

[IanB]

Note :- IOError: [Errno 5] Input/output error: '/media/usb/20170530 1450 - SBS VICELAND - Cyberwar.ts.meta'

Probably the USB disk/stick is dicked in some way.

However a disk error should not prang the system.

[prl]

On a T3 doing this, I'd be inclined to wonder whether the power supply hadn't gone out of spec.

[Paul_oz53]

On a T3 doing this, I'd be inclined to wonder whether the power supply hadn't gone out of spec.

+1. A job for Warkus (aka Mark) at Decisive computer solutions .

[Grumpy_Geoff]

On a T3 doing this, I'd be inclined to wonder whether the power supply hadn't gone out of spec.

+1. A job for Warkus (aka Mark) at Decisive computer solutions .

I'm thinking it was repaired two years ago - viewtopic.php?f=53&t=12212&p=161822#p161822

Perhaps the HDD has gone to crap, and some of that crap was copied to the USB drive?

[Paul_oz53]

On a T3 doing this, I'd be inclined to wonder whether the power supply hadn't gone out of spec.

+1. A job for Warkus (aka Mark) at Decisive computer solutions .

I'm thinking it was repaired two years ago - viewtopic.php?f=53&t=12212&p=161822#p161822

Perhaps the HDD has gone to crap, and some of that crap was copied to the USB drive?

Your mastery of the search function knows no bounds!

Deleting the .meta file is definitely worth a try but it is more likely a hardware problem given the recurring nature of the problem.

I'd swap out the HDD myself but I have spares lying around. Also, I'd try another USB drive but I have six to choose between. The OP may not have these options available.

[Grumpy_Geoff]

Deleting the .meta file is definitely worth a try but it is more likely a hardware problem given the recurring nature of the problem.

My thinking to initally try the file delete was from this -

The same file is mentioned in each crash log over a few weeks.

I'd swap out the HDD myself but I have spares lying around. Also, I'd try another USB drive but I have six to choose between. The OP may not have these options available.

Failing the "file delete" attempt, unless as you suggest another HDD is at hand, I'd try a file system check on the drive (stop timeshift and zap to a new service, stop debug logging, stop Samba, exit from any telnet sessions, then run the check).

[IanB]

Maybe I am missing something, I often do, but the crash stack trace in question pranged on a "IOError: [Errno 5] Input/output error:" on a file on the usb drive, while this is bad programming, because media errors are somewhat expected.

Yes maybe the 5 volt rail powering the usb is dodgy. More likely the file system on the usb is damaged, either logically or physically.

[togo]

It is certainly complex. Thanks for the contributions!

Grumpy_Geoff: I tried the .meta file with File Commander, and it caused an immediate crash, as did deleting the .eit file. But I think it removed the file. See the crash log - all the lines of key presses are to get to the end of the file list. Deleting other files on the USB disk caused no problems.

I mounted the disk on the Win10 machine with Ext2fsd, but it advised against deleting files. Does anyone see this as not to much of a risk?

Excellent searching - it was 2 years ago last week when the PSU was repaired. But perhaps it could still be a PSU issue. I'm inclined to send it to Warkus for a health check. At least I would know one way or the other about the PSU.

There is no question that crap on the HDD could have been copied to the USB disk. And the problem started with only an HDD in place.

I'll see if one of the HDDs around the place would be suitable. In the meantime a file system check sounds like a good idea. How do I stop debug logging?

Togo

[Grumpy_Geoff]

What format is the USB drive in? Did you format it on the Wiz?

It appears you can't delete that .meta file using File Commander as it gives the same error -

Code: Select all

{9537}<129064.393>   File "/usr/lib/enigma2/python/Plugins/Extensions/FileCommander/plugin.py", line 517, in doDelete
{9537}<129064.419> OSError: [Errno 5] Input/output error: '/media/usb/20170530 1450 - SBS VICELAND - Cyberwar.eit'
{9537}<129064.420> [ePyObject] (CallObject(<bound method Session.processDelay of <__main__.Session instance at 0x71f2dcb0>>,()) failed)

Perhaps the file system check may work on the USB drive?
Do you have debug logging enabled? It's not on by default and you have to take a few steps to turn it on. You have to enable/disable it via OpenWebif, which requires your Wiz to be network connected.
If your Wiz is not network connected then forget about this disk check pre-step.
If however your Wiz is network connected then open OpenWebif and have a look at the debug log settings - http://beyonwizt3 >> Settings >> Log settings. Is 'Enable debug logs' ticked? If not, then also forget about this disk check pre-step.

The file system check is in Menu>>Storage. Give it a run on the internal drive as well as the external one.

[prl]

The T3 2TB crashes and reboots for no apparent reason.

When you say "reboot", do you mean an actual reboot (restarts with the big blue Beyonwiz button), or a GUI restart (restarts with the blue "bubbles" screen)?

And the logs that you've posted look a bit incomplete. Did you post the whole log file on both cases, or is it trimmed down?

[Grumpy_Geoff]

In the first crash log, why is there an attempt at merging timeshift and recording files for a recording from May-2017?
A recording of Hard Quiz finishes, then 100 seconds later, bang!

Code: Select all

{30439}<365289.600> Traceback (most recent call last):
{30439}<365289.600>   File "/usr/lib/enigma2/python/Components/Timeshift.py", line 1130, in ptsMergeRecords
{30439}<365289.631>   File "/usr/lib/enigma2/python/Components/Timeshift.py", line 101, in readMetafile
{30439}<365289.635> IOError: [Errno 5] Input/output error: '/media/usb/20170530 1450 - SBS VICELAND - Cyberwar.ts.meta'
{30439}<365289.677> [ePyObject] (CallObject(<bound method InfoBar.ptsMergeRecords of <class 'Screens.InfoBar.InfoBar'>>,()) failed)

[prl]

In the first crash log, why is there an attempt at merging timeshift and recording files for a recording from May-2017?
A recording of Hard Quiz finishes, then 100 seconds later, bang!

I'd wondered a little about that, too. But the time difference is about 101 sec, and none of the timers that trigger calls of ptsMergeRecords() are that length.

The normal one that's called when timeshift is to be saved is 120 sec. Other timer delays are 10 and 15 sec. There is a timed trigger on the completion of a recording to resume the merge process, but that should be triggered 15 sec after a recording completes.

The code that does this is one of those nasty examples where a comment lies about what the code actually does:

Code: Select all

                # Restart Merge-Timer when all records stopped
                if timer.state == TimerEntry.StateEnded and self.pts_mergeRecords_timer.isActive():
                        self.pts_mergeRecords_timer.stop()
                        self.pts_mergeRecords_timer.start(15000, True)

But the problem seems to be directly associated with accessing the 20170530 1450 - SBS VICELAND - Cyberwar.ts.meta metadata file. Perkaps one of its file block pointers in the file system points off into the wide blue yonder (it should normally only have a single block pointer, because the file is small and blocks are 4kB).

It's a bit difficult to see exactly where in the code the error in readMetafile() is happening (open, read or close?), because the line numbers in the crash logs refer to older code. The OP is running 20190207.

I doubt that updating to newer firmware would help with the problem. There's nothing in the current firmware that would catch the exception thrown by readMetafile() in the first log.

It looks like something has curdled the disk, and I'm not sure why. I vaguely recall another user having HDD problems caused by a flaky USB power regulation. If there is some underlying problem like that, I'd guess that the HDD problems will continue.

A long time ago, I tried to create a state diagram for the timeshift merge process. It was a thing of nightmares.

[togo]

Thanks, Grumpy_Geoff.

The fsck on the internal drive completes with no errors, and no messages at all.

This is the error on the USB drive. It is an Ext4 drive, formatted by the Wiz:

Checking file system...
Error: T3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(ie., without -a or -p options)
has INLINE_DATA_FL flag on filesystem without inline data
support.

Prl, yes, it is a full-on reboot, starting with a half-screen of white text with crash details, and then showing the big blue button. I haven't trimmed any of the log files at all.

[prl]

The fsck on the internal drive completes with no errors, and no messages at all.

This is the error on the USB drive. It is an Ext4 drive, formatted by the Wiz:

Checking file system...
Error: T3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(ie., without -a or -p options)
has INLINE_DATA_FL flag on filesystem without inline data
support.

The INLINE_DATA_FL flag allows very small files to have their data stored directly in what's normally the block pointers field of the file inode (the place where all the file's metadata is stored apart from its name). Up to 60 bytes of data can be stored in the inode. Formatting the USB HDD from the Beyonwiz GUI will not set that flag.

The message is a bit ambiguous, but I suspect it means that an inode is flagged to have inline data but the file system is not flagged as allowing it. Most .ts.meta files would be too big to be stored as inline data, because they're usually a good bit larger than 60 bytes.

You'll need to do the fsck in a way that avoids setting the "-p" flag ("preen", or fix minor problems automatically, but fail on more complicated problems).

Prl, yes, it is a full-on reboot, starting with a half-screen of white text with crash details, and then showing the big blue button. I haven't trimmed any of the log files at all.

OK, thanks. The full reboot crash explains why the log files are truncated. So the enigma2 application throws an exception, that's caught by enigma2 and it starts printing the debug log, but before that finishes, the kernel crashes (possibly due to the same underlying problem). It would be nice to have the system messages log for that, but the system log is volatile and is cleared by the reboot.

[togo]

I can telnet to the Wiz, but fsck returns "command not found". Can anyone help with the file location, please?

[Grumpy_Geoff]

I can telnet to the Wiz, but fsck returns "command not found". Can anyone help with the file location, please?

See here - viewtopic.php?f=50&t=13567#p181367
Also, same topic - viewtopic.php?f=50&t=13567#p181374 for "'('a' enables 'yes' to all)' prompt hint"

[prl]

While they're useful, I don't think those links address the "command not found" problem. There is no fsck program in the firmware. Use fsck.ext4 instead.

[Grumpy_Geoff]

While they're useful, I don't think those links address the "command not found" problem. There is no fsck program in the firmware. Use fsck.ext4 instead.

The first link should, as it is to a post of yours showing fsck.ext4 in action

[prl]

I only read the text in the post, not the command output

[peteru]

The T3 has also been known to have an issue with one of the voltage regulators on the motherboard. It causes unexplained crashes and spontaneous reboots. I believe prl was bitten by this problem many, many years ago.

[togo]

The fsck results are attached. I ran them with '-n' option, so it is just a check, not a fix.

The HDD looks ok, but the USB file system does look curdled!

My thought is to copy some material from the USB disk, then reformat it, and see whether it behaves.

The original problem was the spontaneous reboots, and I'd like to focus on that.

by peteru » Fri Jan 31, 2020 10:27 pm

The T3 has also been known to have an issue with one of the voltage regulators on the motherboard. It causes unexplained crashes and spontaneous reboots. I believe prl was bitten by this problem many, many years ago.

This looks like a possible cause. The reboots have not occurred for a few weeks: this could be consistent with this, assuming high operating temp is a factor. There has been cooler weather recently, and lower power use, as I have had the USB as the default movie location.

[prl]

It looks like a run of 46 sequential inodes* has been clobbered on the USB drive. The contents of those files are probably toast. On the up-side, though, if those are recordings and are from the same set of recordings, it may only be as few as 8 recordings affected (6 files/recording).

However, if any of the inodes are directories, the naming information for the files in the affected directories will be lost. The files in the directories may be recoverable (fsck will move them into the USB drive's lost+found directory), but you'll have to work out what the files are and which files belong to which recordings. It's doable, but may not be easy. The files' last modification times will probably help you match the sets of files together. The .ts.meta file will help recover the recording names.

Of course, if something has been writing random rubbish into inode blocks, they may also have written rubbish into other parts of the disk. From the fsck errors, it doesn't look like any file indirect blocks (file system blocks containing pointers to blocks or index blocks) have been wrecked. Fsck can't tell whether file data blocks have been clobbered.

* An inode (index node) is a U*ix filesystem data structure element that contains all the information about a file except its name and its data blocks (though it contains pointers to its data blocks). A U*ix directory is a list of (name, inode number) pairs. Separating the inode data from the directory separates the file system structure (directed graph) from the structure defining the file metadata and contents (effectively a simple array).