Ten HD guide empty 5pm to midnight

[Paul_oz53]

Odd but the guide for Ten HD is missing on the T4 for this evening All the rest of the channels are ok. Gap is from 5pm to 12:30 am.

The expected info is present in the app. Switching to FTA the Ten guide is correct. Resending all timers didn't fix it but the Ten HD timers are still listing as expected.

Any idea why the gap? How to fix?

Paul

[grampus]

On mine as well
+1

[Paul_oz53]

On mine as well
+1

So not just us. Rebooting didn't fix it either. Odd thing is our T3 and P2 are normal. WTF?

[prl]

I'm surprised that the T3 is OK when the T4 is not. The cause is common to both.

This is a known problem: Bug #603: Gaps in IceTV EPG.

The good news is that the IceTV side of the issue was fixed while I was taking a break on Norfolk Island. There are corresponding changes needed in the IceTV plugin, and I'll get those done ASAP. They aren't difficult, but need to be well tested.

When the new version is available, you'll probably need to disable IceTV, disable AutoTimer, restart the GUI, delete all timers (not just ones marked with the IceTV icon in the timer list), do a Resend all timers for this device on the IceTV Web page, then re-enable IceTV and do a Fetch EPG & timers, and get AutoTimer to re-scan (the fetch and re-scan may be easiest to do by just restarting the GUI again). Finally, add any manual timers. I'll have a bit more of a think about how to do the changeover and see if it can be made a bit simpler, but that may be as simple as it gets.

Before I submit the changes to the source code repository, I'll make patches available for testing by IceTV and anyone who wants to give it a try. If you want to try it and then go back to the current version, you'll need to go through all those changeover steps all over again. If you keep the changes installed, you'll need to uninstall them and then install the official version, but the changeover then will be a bit simpler. I will also test those steps.

The reason why it's so messy is that a new show id field will be being used in the IceTV EPG and in timers (IceTV, manual and AutoTimer-generated), so the EPG needs to be cleared and re-loaded, and all the timers need to be reconstructed.

If you remember looking at the Birthday Problem in high school probability and wondering why the probability of two people in a group having the same birthday (and how surprisingly high the probability is even for relatively small groups) is in any way relevant to anything, this issue is an example of the statistics of the Birthday Problem (or the Birthday Attack) in action.

There is more detailed description of the problem (including most of what's in the issue tracker and discussion of different approaches to fix it) in the forum in Gaps in IceTV EPG in the Developers Community section.

[Paul_oz53]

Thanks for the detailed description of the problem Peter. As always, very educational. I rather expected it to be bug 603 but, like you, was puzzled why the T3 is ok.

Since I only have 3 autotimers your recipe for a fix is quite ok.

I'd be happy to test a patch on one machine. The bug has only occurred on the secondary channels until today so I didn't immediately suspect it as the cause.

Cheers, Paul

[prl]

...
I'd be happy to test a patch on one machine. The bug has only occurred on the secondary channels until today so I didn't immediately suspect it as the cause.
...

Thanks.

The underlying process that causes the problem isn't (or largely isn't) random, but its effects can appear to be random. It can affect any channel, at any time, and in various different patterns, though because of the underlying process, runs of missing EPG entries are a common outcome. The problem can affect any region, but the particular errors (or absence of them) at any time are particular to a region. So one region may have one group (or set of groups) of EPG data missing, another region may have another group (or set of groups) missing, and another have no problem at all.

[prl]

One thing that just occurred to me is that the setting of MENU>Setup>TV>EPG settings>Keep old EPG for minutes can affect the outcome of this bug.

If the T3 has a smaller value for that setting than the T4, some colliding event ids may have been cleared out of the T3, but are still being retained in the T4, and that might explain why the data isn't missing on the T3. However, the scope of the missing data on the T4 inclines me to think that that isn't the reason for the difference, and at the moment I can't think of anything else that might cause it..

[Paul_oz53]

One thing that just occurred to me is that the setting of MENU>Setup>TV>EPG settings>Keep old EPG for minutes can affect the outcome of this bug.

If the T3 has a smaller value for that setting than the T4, some colliding event ids may have been cleared out of the T3, but are still being retained in the T4, and that might explain why the data isn't missing on the T3. However, the scope of the missing data on the T4 inclines me to think that that isn't the reason for the difference, and at the moment I can't think of anything else that might cause it..

No idea why they are different at my end because both have identical settings for the EPG. In truth, I set up one machine then copy the settings files and the network and autotimer files to the other - except for machine specific parameters like the machine id which I edit manually.

Just checked and the setting for "Keep old EPG for minutes" is 30 min in both. About the only difference I can see is they login to the IceTV server at different times but both are set to 15 mins cycle time.

[prl]

The code to fix this has now been written for both the IceTV and Beyonwiz sides, and testing has started. The new data fields needed for the fix now appear in the IceTV data sent from the server.

I'm running the fix on our in-use T4 (running the latest beta, 20170828), and I've just sent a copy of the Beyonwiz side of the fix to Daniel Hall at IceTV for testing by them.

If we don't see any problems in the next few days I'll then make an alpha version available in the Gaps in IceTV EPG topic in the Developers Community section.

If you don't use AutoTimers with IceTV, applying the fix is fairly simple:
Disable IceTV
Install patch
Restart GUI
Enable IceTV
Fetch EPG and update timers now

If you use AutoTimers with IceTV, it's a bit more complicated, but not excessively so. The worst bit is that all AutoTimer-generated timers need to be deleted by hand before applying the patch.

I've tested the fix on both the latest beta, 20170828 and on the official version, 20170310, but I'm most interested in how it works when applied to the latest beta. I'm interested in testing both the fix itself and the procedures for updating the timers, in preparation for the instructions to go with a proper beta, and then an official, release.

Distribution of the fix may need to wait until the initial public firmware for the new U4 has been finalised. But that may not be excessively long after testing of the fix has been done.

[Paul_oz53]

I look forward to seeing the patch prl.

Only have three autotimers so that's no great loss and I'll probably convert them to IceTV anyway. I'll update to the latest beta too as part of the process.

Cheers, Paul

[prl]

You might wast to upgrade to the beta a bit in advance of the patch, so there's not so many things changing at once.

[Paul_oz53]

You might wast to upgrade to the beta a bit in advance of the patch, so there's not so many things changing at once.

Hi prl,

I think i'm ready to load the IceTV patch when it becomes available. This will be the last post in this thread - I'll use the Developer section for any commentary on the patch effectiveness.

The T4 is now running 20170828 + online update. Setup from scratch. Also, have swapped the internal drive over as discussed elsewhere. Quite painless in fact. Plugged it in and it was working straight away, as you guys said it would.

Might be worth noting that I had some initial trouble getting the IceTV plugin running. Same effect as Maindee mentioned in his recent post - i.e. IceTV error message about empty email or password. Ran the setup wizard and reentered my credentials and got the error message but it spontaneously came good about 5 minutes later.

Cheers, Paul

[prl]

There's still a bit of a glitch in the new code. If you enter a manual timer (TIMER, GREEN Add Timer) for anything but the show currently showing in live TV, the show appears on the IceTV side to be a recording of the show currently being viewed, rather than the show that the timer is for.

To fix this requires (more or less) the rejected Proposal #23: Fill in timer name from EPG to be implemented.

It's slightly messy, but I'm looking at it now.

[Grumpy_Geoff]

There's still a bit of a glitch in the new code. If you enter a manual timer (TIMER, GREEN Add Timer) for anything but the show currently showing in live TV, the show appears on the IceTV side to be a recording of the show currently being viewed, rather than the show that the timer is for.
...

I realise you're looking at fixing this, but just for curiosity sake - what happens on the IceTV side if one adds a manual timer whilst viewing a recording, as there'd be no current event in that case?

[prl]

Good question. But my aim is to find the show id of the program to be recorded by the timer anyway.

[Grumpy_Geoff]

That timer's times might then be gazumped by the IceTV server if it decides it knows best
Experience has taught me that if I want to create a manual timer that had a 'large' (loose definition) overlap on surrounding events, to firstly disable IceTV, create the manual timer, then lastly re-enable IceTV.

[prl]

That timer's times might then be gazumped by the IceTV server if it decides it knows best
...

No, in this case the problem is solidly on the Beyonwiz size. The Beyonwiz is sending IceTV times and show ids that aren't consistent. What IceTV is doing is inconvenient, but perfectly logical.

[prl]

...
I realise you're looking at fixing this, but just for curiosity sake - what happens on the IceTV side if one adds a manual timer whilst viewing a recording, as there'd be no current event in that case?

What happens on the IceTV side is pretty much the same as what hapens if you do the same from live TV, but the fine details are somewhat different.

When you create a manual timer while playing a recording, the event details apart from the name and description) are extracted from the recording's .eit file. That includes the start time, end time and event id of the recording. The serviceref for the timer is also the recording's serviceref.

Those details are used to create a record timer. If the start and end time are less than two weeks old, they're used as-is, otherwise the start and end time are set to the current time. The recording's event id is used as the timer's event id.

The RecordTimerEntry __init__() code then tries to fill in the name and description by looking up the EPG. This will fail, because the recording's serviceref isn't a valid EPG serviceref. Even if it was correct, you'd only get a match in the EPG if the midpoint of the recording begin - end times was a time that matched an event in the EPG.

If you're currently playing a media file rather than a recording, the event id won't be found, and so won't be sent to IceTV, and it would probably work as it used to.

It's likely to be the same as it is for media if you're playing a stream (though I haven't looked at the code).

The good news is that I've implemented proposal #23 above. I've tested the basics, but I'm still to test the case for repeating timers, where getting the timer's start and end times isn't exactly straightforward.

If anyone feels like giving themselves a headache, I'm pretty sure that the implementation of this in timer.TimerEntry.processRepeated() is wrong (or at least doesn't match the helpful comments). The code has a few bits of strangeness, like separating out the bit flags that represent the repeat pattern into an array, but inverting them in the conversion (and setting them to ints rather than bools), and then the comments describe the loop termination condition, but the code, of course, implements the negation of that - the loop continuation condition.

I'm pretty sure it's the code that's wrong, not the comment, though Norm Shryer's adage "When code and comments disagree, both are probably wrong" may well apply.

And why anyone would write (even when retaining the inversion):

Code: Select all

			flags = self.repeated
			for x in (0, 1, 2, 3, 4, 5, 6):
				if flags & 1 == 1:
					day.append(0)
				else:
					day.append(1)
				flags >>= 1

rather than

Code: Select all

			flags = self.repeated
			for x in range(7):
				day.append(not (flags & 1))
				flags >>= 1

is really beyond me.

But this week's prize for "most egregious misuse of time module functions" goes to this bit of code from Screens/TimerEntry.py:

Code: Select all

weekday = int(strftime("%u", localtime(self.timer.begin))) - 1

[Grumpy_Geoff]

...
I realise you're looking at fixing this, but just for curiosity sake - what happens on the IceTV side if one adds a manual timer whilst viewing a recording, as there'd be no current event in that case?

What happens on the IceTV side is pretty much the same as what hapens if you do the same from live TV, but the fine details are somewhat different.

When you create a manual timer while playing a recording, the event details apart from the name and description) are extracted from the recording's .eit file. That includes the start time, end time and event id of the recording. The serviceref for the timer is also the recording's serviceref.

Those details are used to create a record timer. If the start and end time are less than two weeks old, they're used as-is, otherwise the start and end time are set to the current time. The recording's event id is used as the timer's event id.

The RecordTimerEntry __init__() code then tries to fill in the name and description by looking up the EPG. This will fail, because the recording's serviceref isn't a valid EPG serviceref. Even if it was correct, you'd only get a match in the EPG if the midpoint of the recording begin - end times was a time that matched an event in the EPG.

If you're currently playing a media file rather than a recording, the event id won't be found, and so won't be sent to IceTV, and it would probably work as it used to.

It's likely to be the same as it is for media if you're playing a stream (though I haven't looked at the code).

The good news is that I've implemented proposal #23 above. I've tested the basics, but I'm still to test the case for repeating timers, where getting the timer's start and end times isn't exactly straightforward.
...

Thanks. that reply carries a fair bit more info than the one I attempted to reply to that you removed
I note a manual timer created whilst in recording playback uses the event name as the 'Channel' which gives rise to a neat serviceref in the timer

Code: Select all

serviceref="1:0:0:0:0:0:0:0:0:0:/media/hdd/movie/20171022 0850 - TEN HD - Motor Sport_ V8 Supercars.ts"

Of course, Ice says I'm just being silly (and I was) -

Code: Select all

[IceTV] 406 Not Acceptable

Does your code now grab the correct Service from the .eit file?

In my opinion, manual timers don't need to be sent to IceTV, If they do, then they risk gretting altered by Ice too.

[MrQuade]

In my opinion, manual timers don't need to be sent to IceTV, If they do, then they risk gretting altered by Ice too.

Ideally, you'd want to be able to distinguish between timers that were created to record an event vs ones that are set with arbitrary start and end times. If you were to simply say that all manual timers (ie. "A timer") are not sent to Ice that'd mean that the plugin would simply not have the ability to send any timers to Ice.

If you at recording an event, then it would be advantageous for Ice to assume control, whereas a freeform start/end timer would not want to be interfered with by Ice. Or is that what you mean by a "manual" timer?

Part of getting something like that to work is the often spoken about ability to store the padding duration with each timer so that the Wiz can make smarter decisions about what event is actually being recorded. Prl.... Is that code still percolating around in your head? I know you spoke about making dinner progress on that a while back.

Personally I still think that all repeating timers should be ignored by Ice. That'd solve some big problems quite simply.

[prl]

...
Does your code now grab the correct Service from the .eit file?

No. I'm not even sure if it's in there. It's normally extracted from the .ts.meta file. I haven't tested it, but if a timer's service isn't recordable, you should get a warning and be sent off to set a proper service.

In my opinion, manual timers don't need to be sent to IceTV, If they do, then they risk gretting altered by Ice too.

That's the reason they're sent to IceTV.

Anyway, updating the event id in repeated timers when they repeat should fix Bug #174: Graphical EPG highlights active timers, but does not offer to allow you to change them, and be a step along the way to fix the bug of repeat timers set on the Beyonwiz disappearing from IceTV after their first run.

The code that I've written for the gaps fix should be able to be reused to help fix both of them, but both of them are out of scope for what I'm doing now.

There may be an argument for a flag on a timer to stop it being sent to IceTV.

[Grumpy_Geoff]

In my opinion, manual timers don't need to be sent to IceTV, If they do, then they risk gretting altered by Ice too.

Ideally, you'd want to be able to distinguish between timers that were created to record an event vs ones that are set with arbitrary start and end times. If you were to simply say that all manual timers (ie. "A timer") are not sent to Ice that'd mean that the plugin would simply not have the ability to send any timers to Ice.

If you at recording an event, then it would be advantageous for Ice to assume control, whereas a freeform start/end timer would not want to be interfered with by Ice. Or is that what you mean by a "manual" timer?
...

Yes, this discussion is about TIMER, GREEN/Add - not about timer creation via the EPG.
In an Ice world, I view manual timers as recording by time block and that block should not change. Timers created via the EPG are recording by event and thus should follow any Ice (EPG) change.

[Grumpy_Geoff]

...
Does your code now grab the correct Service from the .eit file?

No. I'm not even sure if it's in there. It's normally extracted from the .ts.meta file. I haven't tested it, but if a timer's service isn't recordable, you should get a warning and be sent off to set a proper service.

Nup, it leaves the serviceref set as the (Enigma2 file?) serviceref as I previously posted. No warning, nothing. Ice gets sent - "channel_id": null.

In my opinion, manual timers don't need to be sent to IceTV, If they do, then they risk gretting altered by Ice too.

That's the reason they're sent to IceTV.

As I posted to MrQuade, I view a free-form manual timer as recording by block, and IceTV doesn't need to know about.

[prl]

...
Yes, this discussion is about TIMER, GREEN/Add - not about timer creation via the EPG.
In an Ice world, I view manual timers as recording by time block and that block should not change. Timers created via the EPG are recording by event and thus should follow any Ice (EPG) change.

How, precisely, would you define the difference between a "timer created via the EPG" and a "manual timer"? How would you distinguish a timer created by TIMER, GREEN Add from one created by EPG, GREEN Add Timer, especially if they contained exactly the same start and end times?

How would you distinguish them from a timer created by EPG, REC, GREEN Add Timer, if they all had the same start and end times.

How would that definition be affected if the user changed their padding settings so that when the timer's start and end times are adjusted for the padding change, they no longer correspond to the start and end of a program? What if the EPG changes and that changes whether the timer corresponds to an EPG entry after stripping padding?

I have a certain amount of sympathy for the idea of a "don't send timer to IceTV" setting on a timer. Less enthusiasm for trying to work out the user's intention automatically.

[Paul_oz53]

I have a certain amount of sympathy for the idea of a "don't send timer to IceTV" setting on a timer. Less enthusiasm for trying to work out the user's intention automatically.

FWIW - I am comfortable with this as an option, even if it means the IceTV app will not show the manual timer has been set. At worst, the unit might record the program twice because the user also adds it to IceTV. A better outcome than not recorded at all.

If making it work as intended is difficult then I think this is the best alternative.

Paul

[prl]

...
If making it work as intended is difficult then I think this is the best alternative.
...

In this case, my problem is that IMO the intention is very difficult to divine.

[Grumpy_Geoff]

...
Yes, this discussion is about TIMER, GREEN/Add - not about timer creation via the EPG.
In an Ice world, I view manual timers as recording by time block and that block should not change. Timers created via the EPG are recording by event and thus should follow any Ice (EPG) change.

How, precisely, would you define the difference between a "timer created via the EPG" and a "manual timer"? How would you distinguish a timer created by TIMER, GREEN Add from one created by EPG, GREEN Add Timer, especially if they contained exactly the same start and end times?

How would you distinguish them from a timer created by EPG, REC, GREEN Add Timer, if they all had the same start and end times.

How would that definition be affected if the user changed their padding settings so that when the timer's start and end times are adjusted for the padding change, they no longer correspond to the start and end of a program? What if the EPG changes and that changes whether the timer corresponds to an EPG entry after stripping padding?

I have a certain amount of sympathy for the idea of a "don't send timer to IceTV" setting on a timer. Less enthusiasm for trying to work out the user's intention automatically.

Differentiate by their method of creation, from the screen they were created from.
If a timer is then subsequently changed, if it contains an 'ice_timer_id' value then send the update to IceTV, if no 'ice_timer_id' value then don't send to IceTV.

My view is that if one is in the EPG and creates a timer (GREEN/Add Timer) then that is a program/event timer, and EPG changes should be followed (by sending the timer to IceTV). If one is in the Timer List and creates a timer (GREEN/Add) then that is a time block timer and EPG changes don't need to be followed.

I've been burnt a number of times by IceTV altering my time block timers, so I now work around it by disabling IceTV, create my time block timers, then re-enable IceTV. An example of why I create block timers - frequently ABC ME will show 5 hours worth of programs from the same series as 13 back-to-back programs. I don't want to end up with 13 recordings with their pre/post padding, transferring each to the PC for editing the in-between promo fluff et. al., before creating three mp4/mkv files with those episode split across them. So I create three block timers covering those back-to-back programs and proceed from there

If what I'm suggesting is not feasible/possible then that's no biggie for me as I'll just continue using my work-around. If you're able to implement a 'don't tell Ice' setting then that'd work for me.

[Paul_oz53]

...
If making it work as intended is difficult then I think this is the best alternative.
...

In this case, my problem is that IMO the intention is very difficult to divine.

Touche! Divine is probably the right word too.

Like Grumpy_G, I sometimes record adjoining episodes of various programs or a block of programs on one channel, esp live sports which can run overtime and have suffered a disappearing manual timer. Haven't tried the disable/enable trick but may just do so if no other solution is viable.

[prl]

...
Differentiate by their method of creation, from the screen they were created from.
If a timer is then subsequently changed, if it contains an 'ice_timer_id' value then send the update to IceTV, if no 'ice_timer_id' value then don't send to IceTV.

Unfortunately, that method of flagging doesn't fit well with how ice_timer_id is set. When a new timer is created on the Beyonwiz, a callout from a RecordTimer onTimerAdded or onTimerChanged event invokes an IceTV plugin method. The onTimerAdded action sends the timer details to IceTV, which then sends back a response containing the ice_timer_id, which is then added to the timer. The code in the IceTV plugin has no idea how the timer was created (and neither does RecordTimer). The onTimerChanged action does the same as onTimerAdded action if the timer has no ice_timer_id and treats it as an update otherwise.

My view is that if one is in the EPG and creates a timer (GREEN/Add Timer) then that is a program/event timer, and EPG changes should be followed (by sending the timer to IceTV). If one is in the Timer List and creates a timer (GREEN/Add) then that is a time block timer and EPG changes don't need to be followed.

That oversimplifies the situation. There are two ways of creating a timer in the EPG: REC>Add Timer and GREEN Add Timer. REC>Add Timer adds a new timer with default padding for the highlighted event (and that would be the nearest thing to a definite program/event timer action available, rather than GREEN Add Timer). GREEN Add Timer opens the Timer entry screen with the defaults for being and end being the padded times for the highlighted event. If the user immediately presses OK, that's equivalent to REC>Add Timer. But the user is free to modify the start and end times for the timer before pressing OK.

The only difference between EPG, GREEN Add Timer and TIMER, GREEN Add is the default values that the Timer entry screens get; the highlighted EPG event in one case, and the currently playing service's event in the other.

I appreciate the problem you're trying to fix, but I think it's a better solution for the user to simply be able say "this timer shouldn't be sent to IceTV," rather than try to have the code try to guess it. The setting should probably default to True for timers created by from the EPG (both methods) and False otherwise.

This is really getting a long way from what I'm trying to do with this topic, which is to prevent the gaps in the ceTV EPG (and as a side-effect of those changes, be able to fix the problem of the wrong event id getting attached to IceTV-generated timers and so fix Bug #174: Graphical EPG highlights active timers, but does not offer to allow you to change them.