Is there a way to schedule a reboot by T2 or U4

[IanL-S]

One of my T2's has recently developed the habit of becoming completely unresponsive - including failing to record scheduled timers. The only solution is to use the power switch on the back.

Is there a way to either have the T2 reboot when it becomes unresponsive (there is a TAP that does this for Topfield PVRs) or a way to schedule a reboot?

[Grumpy_Geoff]

or a way to schedule a reboot?

I'd do it as 2 x repeating Power Timers.

The first is a "go to deep standby" (shutdown) Power Timer.
It will fire at the nominated time, display a 3 min countdown popup, then shut down to deep standby. Don't leave the Wiz showing a menu/popup otherwise it won't be able to effect the shut down.

The second is a "wakeup to standby" Power Timer, timed for, say, 5 mins after the above shut down timer.
For a T2 though, since it doesn't have a smart front panel, the Power Timer "wakeup" and "wakeup to standby" bootup types are treated the same (as a wakeup to full running state) as the T2 doesn't know the specifics of the boot, so it can't actually do the direct-to-standby part of the action. As a work-around, you can set an end time for the ("wakeup" or "wakeup to standby") Power Timer for 3 mins after the start time and with an after event action of "go to standby" (don't make the end time any shorter as the UI startup needs to complete before the UI can get around to detecting a "mid-stream" timer and effect that after event action).

There is a "reboot" type Power Timer, but you can't have an after event action for it. You need a second Power Timer of type "goto standby" set for a few mins later.

[Paul_oz53]

...

Is there a way to either have the T2 reboot when it becomes unresponsive (there is a TAP that does this for Topfield PVRs) or a way to schedule a reboot?

Have you tried Menu/Power/Reboot to schedule a timer? As Grumpy_Geoff notes, you'll probably need two timers. Probably not going to solve your problem though if you mean to use it it to escape a lock up when it occurs.

At one stage I'd get lockups on my T4 and U4 if I used the remote to access menus a lot. I use a daily reboot on them which seems to have cured that problem.

However, if the problem occurs when watching you can try the Openwebif interface on a browser to trigger a restart provided your T2 is internet connected. Type http://beyonwizt2 or http://(IP-address of the T2) into the browser address bar to access the T2.

If the lock up is severe, cycling the power switch may still be required.

If this is chronic I'd firstly reflash the firmware to eliminate instability as a cause. If that doesn't work I'd investigate whether a hardware problem was at play.

[MrQuade]

A scheduled reboot isn't going to help you if the GUI is completely locked up and unresponsive.
A cron job that is scheduled at the OS level would have more luck, as long as the underlying operating system is still working.

But, as the other guys hinted at here, there should be zero reason for the Wiz to be locking up if the hardware is in good working order.

It would be more useful to try to track down the source of the lockups. Turn on logging. Wait for it to happen, and then see if the logs can be retrieved (preferably prior to rebooting).

If not. Then USB flash to the latest beta, and set up from scratch to avoid restoring potentially troublesome config files.

[Grumpy_Geoff]

One of my T2's has recently developed the habit of becoming completely unresponsive - including failing to record scheduled timers.

This is just a guess, but it's possible the periods of unresponsiveness may coincide with Linux waiting 5 mins between batches of attempted restarts caused by the GUI crashing/or being terminated.
Since you use IceTV and therefore have a networked T2, use Windows File Explorer to grab the file "enigma2_execution.log" from the "Logs" Samba share (\\BEYONWIZT2\Logs), and attach it here so we can have a "nosey".

[peteru]

Maybe turn off IceTV?

Apparently the IceTV servers are sending out data that causes the IceTV plugin to consume large amounts of memory. On machines with low free memory, such as the T2, this results in the machine becoming unresponsive or crashing. A classic case of denial of service, but probably not worth raising a CVE.

[IanL-S]

To you all, thank you for taking time to provide the many useful suggestions.

It would be best to attack the cause rather than deal with the symptoms.

The issue could be with IceTV data overload - I have had similar problems with my Toppys. Another possibility is that there is something amiss with the firmware. I am running the latest beta on the T2 (but also doing this on other T2s and they do not seem to be having this problem). With Toppys corrupt settings can cause many issues, and I assume the same is true for the T2.

Ian

Update: In my haste I missed the suggestion that I download the log file. Had slight problem in making sure that it was for the correct T2 (I have 3 of them - I am a big believer in abundant caution; necessary when you are away from home for extended periods). But given the time and nature of the log entries I think I have the correct one.

[Grumpy_Geoff]

Update: In my haste I missed the suggestion that I download the log file. Had slight problem in making sure that it was for the correct T2 (I have 3 of them - I am a big believer in abundant caution; necessary when you are away from home for extended periods). But given the time and nature of the log entries I think I have the correct one.

Based on that log, the first sign of an issue was Sunday 25th at 05:53 when the GUI was terminated, likely due to an OOM (out-of-memory) issue. The GUI was subsequently auto-restarted. Rinse, repeat for 10 more times.
There were no further GUI restarts after 17:12 Sunday.

[prl]

The times between the crashes doesn't seem to align with the default times (15 min intervals) for an IceTV fetch:

Code: Select all

mmm:ss
115:55
 89:39
 60:14
 66:53
 72:51
 58:03
 52:54
 52:56
 55:37
 53:56

How often do you fetch IceTV?

[Grumpy_Geoff]

The times between the crashes doesn't seem to align with the default times (15 min intervals) for an IceTV fetch:

It was the accumulative effect, I believe. Refer - viewtopic.php?f=16&t=14485#p190639
It appears to take a number of (failed) fetches before...bang!

[prl]

Based on that log, the first sign of an issue was Sunday 25th at 05:53 when the GUI was terminated, likely due to an OOM (out-of-memory) issue. The GUI was subsequently auto-restarted. Rinse, repeat for 10 more times.
There were no further GUI restarts after 17:12 Sunday.

The IceTV server was restarted at about 5pm on Sunday 25. I don't have anything that was running at about that time to know from logging when the restart happened.

I was on the phone to Daniel Hall when he restarted the IceTV server, and that call was 16:48-17:10, and he would have done the server restart a bit before the end of the call. That's a bit before IanL-S's last OOM crash at 17:12.

It's curious that Ian's logs show that relationship with the known IceTV "Python int too large" issues, but Judy's (csutak40) don't.

[prl]

The times between the crashes doesn't seem to align with the default times (15 min intervals) for an IceTV fetch:

It was the accumulative effect, I believe. Refer - viewtopic.php?f=16&t=14485#p190639
It appears to take a number of (failed) fetches before...bang!

Even taking that into account, if Ian's IceTV fetches were at the default 15 min intervals, then if the crash was directly associated with a fetch, the crashes would be expected after 15, 30, 45, 60, 75, etc minutes. Of course, if the IceTV fetch simply puts the system near the OOM limit, that might account for the crash not being aligned, though there's a (not consistent) tendency for the crash intervals to be a bit shorted than the expected IceTV fetch interval multiples (115min, 89min, 72min, 58min, 53min x 2, 54 min).

I'm not saying that it wasn't IceTV, but I think that there's some real doubt about whether it was.

[prl]

The times in your crash log also don't align with when IceTV fetches would happen:

Code: Select all

 64:48
 38:18
 38:55
 39:11

[IanL-S]

The times between the crashes doesn't seem to align with the default times (15 min intervals) for an IceTV fetch:

Code: Select all

mmm:ss
115:55
 89:39
 60:14
 66:53
 72:51
 58:03
 52:54
 52:56
 55:37
 53:56

How often do you fetch IceTV?

Good question; I do not recall changing from the default, but it appears that I have not done so - just looked at the IceTV log file and updates are 15 minutes apart.

[prl]

Thanks.

[Grumpy_Geoff]

Based on that log, the first sign of an issue was Sunday 25th at 05:53 when the GUI was terminated, likely due to an OOM (out-of-memory) issue. The GUI was subsequently auto-restarted. Rinse, repeat for 10 more times.
There were no further GUI restarts after 17:12 Sunday.

The IceTV server was restarted at about 5pm on Sunday 25. I don't have anything that was running at about that time to know from logging when the restart happened.

I know from my T4 that it was successful a few mins before that time -

Code: Select all

{607}< 15002.060> [IceTV] 2020-10-25 13:45:43: EPG download OK

I was on the phone to Daniel Hall when he restarted the IceTV server, and that call was 16:48-17:10, and he would have done the server restart a bit before the end of the call. That's a bit before IanL-S's last OOM crash at 17:12.

Daniel Hall posted on the IceTV forum at 13:41:28 (AWST) that it was "now fixed".
It's possible that Ian's T2 may have had low free memory at that stage, and even a fetch of "good" guide data (which may have been a full fetch) may have pushed it over the edge.

It's curious that Ian's logs show that relationship with the known IceTV "Python int too large" issues, but Judy's (csutak40) don't.

I think they do, the first being circa 5am, the last circa 5pm (AET).

[Grumpy_Geoff]

The times in your crash log also don't align with when IceTV fetches would happen:

Code: Select all

 64:48
 38:18
 38:55
 39:11

My first T2 kill occurred after 31 mins elapsed since the enigma2 start -

Code: Select all

Sun Oct 25 08:47:58 AWST 2020: executing enigma2
Sun Oct 25 09:19:33 AWST 2020: enigma2 terminated with code 137

That would have been two fetches.
I know it was the OOM killer nuking enigma2, because I saw it reported in the messages log.
Whether it was the IceTV plugin or not that was consuming the memory, I disabled IceTV and had no further issues. Later I re-enabled IceTV.
There were reports of other users disabling IceTV too, with success.

[IanL-S]

So, it is possible that there is an explanation for the most recent lock-up.

Clearly there can be several issues at play. First off, the T2 in question was a refurbished unit so it could be more prone to things going wrong. Unlike the other T2 and the U4 it does not have a direct ethernet link to the router; there is a section that uses a TPLink homeplug.

Is there an easy way for me to access the log files on the on the other T2s?

[Grumpy_Geoff]

I think another thing is, we don't know if IceTV was being consistent with whatever it was sending to the plugin, it could have been a varying pile of junk.

[Grumpy_Geoff]

Is there an easy way for me to access the log files on the on the other T2s?

If they're networked, then enter \\{name or IP-address of each T2}\Logs in a Windows File Explorer address bar.

Or in a web browser, enter "file://{name or IP-address of each T2}/logs" (e.g. for my T4 I can use file://beyonwizt4/logs/).

Or use File Commander (Source / Files main menu entry). Navigate through Internal Flash, then home, then root, then logs.

[prl]

I think another thing is, we don't know if IceTV was being consistent with whatever it was sending to the plugin, it could have been a varying pile of junk.

There's also the strange thing that on some Beyonwizes, they got the "int too large" exception, which aborted the fetch but otherwise didn't cause problems.

[Grumpy_Geoff]

I think another thing is, we don't know if IceTV was being consistent with whatever it was sending to the plugin, it could have been a varying pile of junk.

There's also the strange thing that on some Beyonwizes, they got the "int too large" exception, which aborted the fetch but otherwise didn't cause problems.

Do you mean models? The T2 has only 226MiB of memory. Whereas the T4 has 1127MiB. My T2 doesn't serve up any DLNA sources, my guess is that not too many T2 owners will've gone to that extent.

I saw our T2 was down to 25MiB at one point before I disabled the IceTV plugin. Possibly one more fetch of the IceTV garbage and it would've been nuked.
Normally it's in the 80-90 range.

[IanL-S]

Is there an easy way for me to access the log files on the on the other T2s?

If they're networked, then enter \\{name or IP-address of each T2}\Logs in a Windows File Explorer address bar.

Or in a web browser, enter "file://{name or IP-address of each T2}/logs" (e.g. for my T4 I can use file://beyonwizt4/logs/).

Or use File Commander (Source / Files main menu entry). Navigate through Internal Flash, then home, then root, then logs.

Thanks for the information - I have added bookmarks in my web browser for each unit. Now have log files for all of the T2s which are included in the attached zip file. All exhibited the same behaviour but only T2 03 froze.

Is it possible to access the IceTV logs in a similar manner?

Thanks for all the help it is greatly appreciated - I am learning a lot.

Ian

[prl]

Is it possible to access the IceTV logs in a similar manner?

Everything that's in the IceTV logs in MENU>IceTV>Show logs is in the debug logs, if they are enabled. If debug logging is not enabled, then the only IceTV logs available are the most recent 40 log lines in the onscreen IceTV log, which is cleared on reboot or GUI restart.

The only access to the IceTV server-side logs is by asking the IceTV folk (like Daniel Hall).

[Grumpy_Geoff]

Now have log files for all of the T2s which are included in the attached zip file. All exhibited the same behaviour but only T2 03 froze.

"T2 03"
After the GUI's final restart at 16:22 on Sun 25th, there was a further instance on Monday evening 21:27 of the GUI being terminated (likely memory exhaustion), then it appears to be fine until you turned off the rear switch on Thursday morning.
Without any other logs (e.g. enigma2 debug log or from /var/log/messages) then we'll likely never know why it became unresponsive.

Straw clutching, but I wonder if there was anything special in the Melbourne guide data for that Mon night polling period as Judy had similar on Monday evening.


"T2 02"
The log shows the IceTV issues for it started around 3:30am - I believe that's the earliest we've known about.

The U4's GUI didn't get killed because it has more main memory than the T2.

[prl]

"T2 02"
The log shows the IceTV issues for it started around 3:30am - I believe that's the earliest we've known about.

I have a "Python int too large" exception logged at 2020-10-25 00:06:11 (AEST) on our in-use T4.

[Grumpy_Geoff]

"T2 02"
The log shows the IceTV issues for it started around 3:30am - I believe that's the earliest we've known about.

I have a "Python int too large" exception logged at 2020-10-25 00:06:11 (AEST) on our in-use T4.

Yeah, I should've said "the earliest we've seen a T2's GUI get nuked"!
Now when I take a look back at my T4, I see I got one on the 24th 21:11:11 (AWST).
Of course, being T4's we didn't see GUI restarts.

There were 304 genre look errors listed for that time, so I'm guessing that was the new day's guide data.

Code: Select all

{780}< 46870.902> [IceTV] 2020-10-24 20:56:11: Start update
{780}< 46871.718> [IceTV] 2020-10-24 20:56:11: EPG download OK
{780}< 46871.718> [IceTV] 2020-10-24 20:56:11: Timers updated OK
{780}< 46871.719> [IceTV] 2020-10-24 20:56:11: End update
...
{780}< 47770.899> [IceTV] 2020-10-24 21:11:11: Start update
{780}< 47771.889> [EPGFetcher] ERROR: lookup of 0x20 "Current Affairs" returned "News"
...
{780}< 47773.579> [EPGFetcher] ERROR: lookup of 0x01 "Renovation" returned "Cult"
{780}< 47773.629> /usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py:517: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
{780}< 47773.649> Unhandled Error
Traceback (most recent call last):
{snipped}
  File "/usr/lib/enigma2/python/Plugins/SystemPlugins/IceTV/plugin.py", line 517, in doWork
    
exceptions.OverflowError: Python int too large to convert to C long

[Grumpy_Geoff]

...but only T2 03 froze.

Getting back to "T2 03", at the time when it "froze" -
Did it show the spinner?
Was it recording at the time?
Was it showing live TV, or was it in standby?

[IanL-S]

...but only T2 03 froze.

Getting back to "T2 03", at the time when it "froze" -
Did it show the spinner?
Was it recording at the time?
Was it showing live TV, or was it in standby?

I was not around when it froze. The TV was off. When I turned the TV on there was no signal coming from the T2; tried the power button on the RCU several times, no response, tried the power button on the front (if I recall correctly).

Not sure if the listing of completed timers means anything, but I have created such a listing for the T2 03.

Ian

[Grumpy_Geoff]

Not sure if the listing of completed timers means anything, but I have created such a listing for the T2 03.

That list is up until 26/10 21:20. We know that at 21:27:04 enigma2 was killed, and it restarted 1 second later.
Were there any timers set to record between Mon 26/10 21:20, and Thu Oct 29 07:39 when you turned off the power? If so, did any of them record?

[IanL-S]

Sorry, losing the plot here. Possibly another seniors moment. I have listed the timers that are in the completed timers during that period. I have listed those that definitely recorded or did not record. One with * are ones that I would have subsequently deleted if they recorded. But I cannot say if they recorded or not.

There may be some hint in the file.

Ian

[Grumpy_Geoff]

One with * are ones that I would have subsequently deleted if they recorded. But I cannot say if they recorded or not.

The best way to find out what happened to a timer, is to locate it in the "completed" timers section of the timers list, then press INFO on it to show the timer log entries. Then you'll know if it recorded or not.

I'm just trying to see if enigma2 was still working even though the front panel wasn't responding.

I think these last timers from your list are the ones to check -

Code: Select all

Bones *	27/10/2020	23:06	28/10/2020	1:15		
						
NCIS *	28/10/2020	1:51	28/10/2020	4:00		
NCIS *	28/10/2020	11:36	28/10/2020	13:45		
NCIS *	28/10/2020	19:06	28/10/2020	21:15		
Midsomer Murders *	28/10/2020	20:26	28/10/2020	23:45		
Blindspot	28/10/2020	23:06	29/10/2020	1:15		no
						
NCIS *	29/10/2020	1:46	29/10/2020	3:55		

I think you're saying Blindspot didn't record.

[IanL-S]

I checked the info for each of the scheduled recordings and the following indicates the time frame within which the T2 apparently froze:
NCIS 28/10/2020 1:51 28/10/2020 4:00 recorded
NCIS 28/10/2020 11:36 28/10/2020 13:45 recorded
NCIS 28/10/2020 19:06 28/10/2020 21:15 recorded
Midsomer Murders 28/10/2020 20:26 28/10/2020 23:45 did not record
Blindspot 28/10/2020 23:06 29/10/2020 1:15 did not record
NCIS 29/10/2020 1:46 29/10/2020 3:55 did not record
So it looks like the freeze occurred between 19:06 and 20:26 on 28 October. It seems odd that the NCIS episode that started recording at 19:06 completed recording on schedule about an hour after the freeze happened.

[Grumpy_Geoff]

So it looks like the freeze occurred between 19:06 and 20:26 on 28 October. It seems odd that the NCIS episode that started recording at 19:06 completed recording on schedule about an hour after the freeze happened.

Does "T2 03" spend its time in standby or in full running state (showing and timeshifting live TV)?

Perhaps there was enough system resources to complete the successful NCIS 28-Oct 19:06-21:15 timer recording, but not enough to commence a new one for the overlapping Midsomer Murders 20:26-23:45 timer.

[IanL-S]

The T2 03 is (normally) in full running state, not standby. Not sure what gets shut down when the unit is in standby - clearly no video signal is sent the the TV.

[MrQuade]

The T2 03 is (normally) in full running state, not standby. Not sure what gets shut down when the unit is in standby - clearly no video signal is sent the the TV.

Video signal, and none of the tuners are engaged (unless called for remotely), which also means no timeshifting is taking place. If there is no disk activity for a while, then the harddisk can also spin down.
Other than that, the T2 is fully operational.

[IanL-S]

Thanks, I have timeshift disabled on my T2s - it is enabled on my other PVRs.

[prl]

I've been able to reproduce the OOM errors and a lockup on my T2, though in somewhat different circumstances. I'm currently trying again with something more like a normal setup (but with added memory use code in the IceTV plugin).

[prl]

Ianl-S, when was the most recent time that you had the T2s seize up and go non-responsive to the remote? Was it on Wed 28 Oct?

I.e, is it still a problem?

[IanL-S]

It was working the last time I tried (would have been late Sunday). Everything that should have recorded has recorded; it is responding to ftp and Openwebif. There have been no further crashes of enigma2 according to the log. Last entry is from when I restarted on Thursday morning.

Ian

[prl]

OK, thanks Ian.