EPG

[sonicblue]

The non-overlap rule in the EPG means that if the broadcaster data has overlaps (which it shouldn't, since they can't broadcast two different programs at the same time), then the EPG loaded into the Beyonwiz will have gaps.

I'd be disinclined to put any sort of "bodge" into the EPG code. It's complicated enough already. Of course, anyone who thinks that doing it is a good idea is welcome to code it and submit it for acceptance into the code repository. That part of the code doesn't require much knowledge of the rest of the workings on enigma2.

Does this mean there are currently no plans to rework the EPG cache to trim overlaps and prevent this issue?

If so, then I'm interested in taking a look at it myself. The relevant source code file appears to be epgcache.cpp. Unfortunately I've never written anything in cpp, so it will take a very long time for me to get up to speed. Also I'm not sure how to even compile cpp files for Enigma or how to deploy them (does the whole image need to be rebuilt?).

In the meantime I've put some debug lines in EpgList.py and EpgSelection.py to see what it's actually pulling out of the EPG cache when event data is missing. In this case I observe the EPG cache query in EpgList.fillSingleEPG() is not returning any event data when the issue occurs, and then correctly populates the missing event with a None entry in self.list. So the problem seems to be an issue with eEPGCache randomly not providing event data. As I'm unable to add debug lines to epgcache.cpp I've no way of telling if it's due to mishandling of overlaps inside epgcache.cpp or just simply no data being broadcast at the time the cache is updated.

One idea I had was to keep a second cache inside EpgList.py, and then if an event is missing at the next EpgList.fillSingleEPG(), then retrieve it from the second cache.

[MrQuade]

If so, then I'm interested in taking a look at it myself. The relevant source code file appears to be epgcache.cpp. Unfortunately I've never written anything in cpp, so it will take a very long time for me to get up to speed. Also I'm not sure how to even compile cpp files for Enigma or how to deploy them (does the whole image need to be rebuilt?).

I wrote a guide for compiling the easy-ui-4 program source under Windows, using the WSL2 feature.
The same basic approach can be taken for a vanilla Ubuntu Linux installation though.
viewtopic.php?f=70&t=14161

It's up to you to work out how to manage git and the source control, because as far as that goes, I have zero idea .

At the moment, there is no generally available SDK for the T and U series boxesfor version 19.3 firmware, but you can happily build for the V2 with its SDK.

[prl]

Does this mean there are currently no plans to rework the EPG cache to trim overlaps and prevent this issue?

If so, then I'm interested in taking a look at it myself.

Detecting the difference between overlaps where the old overlapping program must be deleted and where one of them should be adjusted may prove more difficult than you imagine.

In order to keep cache updates efficient, the events in the cache must be kept non-overlapping (this is an O(n) vs O(log N) efficiency difference, so it can be important). Time-based searches on the cache are also complicated if overlaps are permitted.

The relevant source code file appears to be epgcache.cpp.

That's the relevant source code.

Unfortunately I've never written anything in cpp, so it will take a very long time for me to get up to speed. Also I'm not sure how to even compile cpp files for Enigma or how to deploy them (does the whole image need to be rebuilt?).

While epgcache.cpp isn't wildly complicated, it's not something that I'd recommend as a place for someone new to C++ to start.

You'll need to wrap your head around the C++ Stdlib classes for maps, too.

I've no way of telling if it's due to mishandling of overlaps inside epgcache.cpp or just simply no data being broadcast at the time the cache is updated.

Under which conditions do you think that overlaps are being mishandled? IMO, the most likely cause of the gaps is something that you haven't listed in your alternatives.

[sonicblue]

Under which conditions do you think that overlaps are being mishandled? IMO, the most likely cause of the gaps is something that you haven't listed in your alternatives.

I was under the impression that overlapping events were the issue based on the discussion about that on the previous page. It seems that is not the case and I'm interested to hear your ideas on what may be causing it.

It seems there is more than one python interface to the guide data, so I am interested in trying others to see if they can be used to fill in the empty events. It seems we can already do that for now and next events by using the same interface used by Infobar, since I've never observed Infobar fail to retrieve now and next. The way infobar interfaces with EPG is still a bit of a mystery to me as it seems to define it only in the skin XML, which I can't link to any python code examples. Similar thing with channel selection EPG which simply displays a ServiceList object, and I can't see any explicit EPG queries inside the ServiceList object.

If you could help me locate all the EPG interfaces in the various CPP files I could then iterate through all of them and try to fall back to whichever one can provide accurate data, similar to how on the V2 we can fall back to proc/vfmw/pts_info to get the video frame rate when iServiceinformation fails to provide accurate info.

[peteru]

There are two sources of EIT events. One is the present/following (aka. now and next) event signalling of the currently running service (which has provisions for marking ad breaks), the other source is the EIT schedule (aka 7 day EPG) data. More detailed information is in the DVB specification. See ETSI EN 300 468 and ETSI TR 101 211. The DVB specification allows for overlap of present/following events, but only under very, specific circumstances that require certain flags to be set and the two events must follow strict transition rules.

The semantics of the EPG cache are such that overlapping events are not permitted. EIT events are placed into the cache as they arrive. Any event that causes an overlap should cause all existing overlapping events to be deleted from the cache. If the broadcaster is sending an invalid EPG stream (i.e. events that overlap), there will be cache thrashing and events will appear and disappear continuously.

[sonicblue]

The semantics of the EPG cache are such that overlapping events are not permitted. EIT events are placed into the cache as they arrive. Any event that causes an overlap should cause all existing overlapping events to be deleted from the cache. If the broadcaster is sending an invalid EPG stream (i.e. events that overlap), there will be cache thrashing and events will appear and disappear continuously.

So the issue is caused by overlapping events? In that case would it be possible to trim the overlap?

For example in EpgList.fillSingleEpg() the EPG data looks like this

Code: Select all

list = [
('1:0:2:258:251:1010:EEEE0000:0:0:0:', 5039L, 1597890600L, 5400L, 'Quiet Time with Rachel Coopes'),
('1:0:2:258:251:1010:EEEE0000:0:0:0:', 5040L, 1597896000L, 3600L, 'Good Afternoon with Various Presenters'),
('1:0:2:258:251:1010:EEEE0000:0:0:0:', 5041L, 1597899600L, 3600L, 'Shake and Move with Jimmy Giggle'),
('1:0:2:258:251:1010:EEEE0000:0:0:0:', 5042L, 1597903200L, 3600L, "Let's Pretend: All about our body"),
('1:0:2:258:251:1010:EEEE0000:0:0:0:', 5043L, 1597906800L, 7200L, 'Home Time with Matthew Backer')
]
Format: (service, event index, event start time in seconds since epoch, event duration in seconds, event title)

Trimming the overlap might be something like:

Code: Select all

for i in range(len(list)-1, 0, -1):
	this_beg = list[i][2] 
	prev_end = list[i-1][2] + list[i-1][3] 
	if prev_end > this_beg:	
		list[i][2] = prev_end
		list[i][3] = list[i][3] - (prev_end - this_beg)

This is a very small list so performance isn't an issue, but doing it for the entire epg cache every time it refreshes will be significantly more expensive. Although cpp is much more efficient than Python and it's only the occasional event which overlaps, so most of the time it should only be 3 extra lines of code per event (calculating this_beg, prev_end, and comparing the two). edit: or just make it a one liner conditional since the vast majority of the time the conditional won't be true.

[prl]

The semantics of the EPG cache are such that overlapping events are not permitted. EIT events are placed into the cache as they arrive. Any event that causes an overlap should cause all existing overlapping events to be deleted from the cache. If the broadcaster is sending an invalid EPG stream (i.e. events that overlap), there will be cache thrashing and events will appear and disappear continuously.

So the issue is caused by overlapping events? In that case would it be possible to trim the overlap?

The cache code requires that the cache is always non-overlapping, so you don't have the ability to load the cache as-is and then patch it to be non-overlapping. You'd need a separate list per service containing at least the start time, duration/end time and the event id.

The problem is that the EPG arrives as a continuous stream. A cache update may start part way through it and end part-way through it (user channel changes are not synchronised with when the broadcast EPG update starts and ends.

This complicates knowing when a full EPG has been transmitted.

If this were simple, it probably would have already been done It's not the first time it has been raised on the forum.

You may find these two references from the FreeTV Australia Operational Practices useful (along with the DVB standards):
OP-44 Implementation Guide for DVB EIT Present/Following Information (EITp/f)
OP-58 Implementation Guide for DVB EITSchedule Information (EITscheduleactual)

[peteru]

So the issue is caused by overlapping events?

I don't recall seeing enough evidence to confirm that assumption. For all we know, it could be a one-off bug in the EPG drawing code somewhere or genuine lack of valid EPG data.

Some unit tests would help. The Python API could be used for a number of tests. The test code would probably do something similar to what IceTV plugin does: disable the parsing of the broadcast EPG, then upload it's own EPG data. The unit test code could simply generate it's own EPG data on the fly to specifically create all sorts of corner cases to test for specific behaviours.

It has been a long time since I touched the EPG code. It's still an ugly mess, but nowhere near as buggy as the code that came from upstream. If I was looking for bugs in the existing epgcache C++ code, I'd be inclined to start with examining the use (and invalidation) of iterators when changing the EPG followed by an audit of the locking logic. But, before looking at any of that, I'd try to find out more about the root cause of the problem. There isn't much point in trying to get your head around the convoluted epgcache code if it isn't the source of the problem.

[sonicblue]

There isn't much point in trying to get your head around the convoluted epgcache code if it isn't the source of the problem.

I suspect it is the source of the problem, because when the issue occurs, querying the epg from Python via eEPGCache::lookupEvent (such as when printing self.list in EpgList.fillSingleEPG()) shows events are in fact missing. Then, querying again later can cause those events to reappear.

Here are the relevant sections of code which appear to be handling overlapping events:

eEPGCache::sectionRead
if ((old_start < new_end) && (old_end > new_start))
      {
            eDebug("[EPGC] removing old overlapping event %04x\n"
                        "       old %ld ~ %ld\n"
                        "       new %ld ~ %ld",
                        getEventID(it), old_start, old_end, new_start, new_end);
            if (eventmap.erase(getEventID(it)) == 0)
            {
                  eDebug("[EPGC] Event %04x not found in eventMap at %ld", getEventID(it), getStartTime(it));
            }
            delete getEventData(it);
            timemap.erase(it++);
      }

bool overlap = false; // Actually overlaps, zero-length event or not time ordered
time_t last_end = 0;

while(size--)
{
      time_t this_start = event->getStartTime();
      time_t this_end = this_start + event->getDuration();
      if (this_start < last_end || this_start == this_end)
            
      else
            last_end = this_end;
      ++cnt;
}
if (overlap)
      overlaps.insert(key);

for (std::unordered_set<uniqueEPGKey, hash_uniqueEPGKey >::iterator it = overlaps.begin();
      it != overlaps.end();
      it++)
      {
            EventCacheItem &servicemap = eventDB[*it];
            eventMap &eventmap = servicemap.byEvent;
            timeMap &timemap = servicemap.byTime;
            time_t last_end = 0;
            for (timeMap::iterator It = timemap.begin(); It != timemap.end(); )
            {
                  time_t start_time = getStartTime(It);
                  time_t end_time = start_time + getDuration(It);
                  if (start_time < last_end || start_time == end_time)
                  {
                        eDebug("[EPGC] load: svc(%04x:%04x:%04x) delete overlapping/zero-length event %04x at time %ld",
                              it->onid, it->tsid, it->sid,
                              getEventID(It), (long)start_time);
                              
                        if (eventmap.erase(getEventID(It)) == 0)
                        {
                              eDebug("[EPGC] Event %04x not found in timeMap at %ld", getEventID(It), start_time);
                        }
                        delete getEventData(It);
                        timemap.erase(It++);
                  }
      

Without understanding how this ties in with everything else, it would be impossible to determine the root cause.

[peteru]

There isn't much point in trying to get your head around the convoluted epgcache code if it isn't the source of the problem.

I suspect it is the source of the problem, because when the issue occurs, querying the epg from Python via eEPGCache::lookupEvent (such as when printing self.list in EpgList.fillSingleEPG()) shows events are in fact missing. Then, querying again later can cause those events to reappear.

While this diagnosis narrows down the field, it still does not mean that there is a bug in the epgcache code. Since you are able to see the relevant events come and go, it should be possible to gather a bit more information to determine the exact combination of values that cause the entries to be only available intermittently. See if you can capture the event_id, start_time and duration for the problematic event and any adjacent events. It could be that the broadcaster is actually sending invalid EPG that has duplicate event_id's rather than overlapping times.

The more specific the information about the data is, the easier it will be to identify the code that handles the problematic events. It may also be the case that the broadcast is non-compliant and the broadcaster needs to be notified. If the broadcaster does not rectify the issue then a complaint to the regulator may be in order.

[peteru]

In terms of potential bugs in the epgcache code, timemap.erase(it++) should be replaced by it=timemap.erase(it) to ensure that we don't try to advance an iterator that has just been invalidated by the erase.

I'm not sure if that is an actual problem in practice though - one would expect a reference to an invalid iterator to have fairly severe consequences, although it is possible that the runtime just substitutes the end iterator when an invalidated iterator is used.

[peteru]

I fixed the iterator code as per above and pushed the code change.

I'll see if I can get a new beta built so that you can experiment and see if there is any change in behaviour. It would be a good idea to see if you can trigger the problem with the existing epgcache code, then retest after applying the update and see if the problem persists or whether this change fixes the issue.

No estimate as to when I can actually produce a new beta build. If I'm lucky, it could be later today, but it will be untested.

[prl]

The overlap elimination code in eEPGCache::load() should not be relevant to the creation of gaps. It should be a no-op on epg.dat files saved by eEPGCache::save().

It's there because of the "EPG cache reload" feature in the EPG settings page, which is intended to import EPG data from other sources via the epg.dat file, and that data may have overlaps.

So you should just need to concentrate on eEPGCache::sectionRead(). NB, IceTV EPG data is also imported via eEPGCache::sectionRead(), even thoug you might not expect that

It's the overlap elimination in eEPGCache::sectionRead() that requires the EPG data to be non-overlapping: it allows O(log N) detection of overlaps with new EPG events. To do fast overlap detection on data that contains overlaps, you need a fancier data structure, like an interval tree.

NB: once the EPG has been filled, it is normal for a new event to overlap events in the current EPG.

[sonicblue]

See if you can capture the event_id, start_time and duration for the problematic event and any adjacent events. It could be that the broadcaster is actually sending invalid EPG that has duplicate event_id's rather than overlapping times.

I observe the EPG cache query in EpgList.fillSingleEPG() sees that there is 5 minutes or more of no guide data in between events and explicitly adds None entries (otherwise we wouldn't even see a blank entry in Single EPG view).

Here is some data retrieved from eEPGCache just now:

LCN31
('1:0:19:347:340: 3202:EEEE0000:0:0:0:', 5087L, 1598115296L, 6928L, 'The Wackness'),
('1:0:19:347:340: 3202:EEEE0000:0:0:0:', 5088L, 1598122224L, 5386L, 'Moonrise Kingdom'),
('1:0:19:347:340: 3202:EEEE0000:0:0:0:', None, 1598127610L, 900L, None),
('1:0:19:347:340: 3202:EEEE0000:0:0:0:', 5089L, 1598128510L, 6055L, 'And God Created Woman'),
('1:0:19:347:340: 3202:EEEE0000:0:0:0:', 5090L, 1598134565L, 8112L, 'Loving'),

LCN32
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44842L, 1598115435L, 1972L, 'France 24 News In English From Paris'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44843L, 1598117407L, 1796L, 'Thai News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', None, 1598119203L, 5420L, None),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 45190L, 1598124623L, 1837L, 'Korean News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 45191L, 1598126460L, 2273L, 'Indonesian News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44846L, 1598128733L, 1270L, 'Hong Kong News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44844L, 1598130003L, 7220L, 'Live WNBA: Indiana V Chicago'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44851L, 1598137223L, 1800L, 'Macedonian News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44852L, 1598139023L, 1733L, 'Croatian News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', None, 1598140756L, 1823L, None),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44854L, 1598142579L, 3063L, 'Portuguese News'),
('1:0:19:346:340: 3202:EEEE0000:0:0:0:', 44855L, 1598145642L, 2603L, 'Urdu News')

Format: (service, event ID, event start time in seconds since epoch, event duration in seconds, event title)


EpgList.py

Code: Select all

def fillSingleEPG(self, service):

		t = time()
		epg_time = t - config.epg.histminutes.value * 60
		test = ['RIBDT', (service.ref.toString(), 0, epg_time, -1)]
		self.list = self.queryEPG(test)
		
		# Add explicit gaps if data isn't available.
		for i in range(len(self.list) - 1, 0, -1):
			this_beg = self.list[i][2]
			prev_end = self.list[i-1][2] + self.list[i-1][3]
			if prev_end + 5 * 60 < this_beg:
				self.list.insert(i, (self.list[i][0], None, prev_end, this_beg - prev_end, None))

[sonicblue]

I fixed the iterator code as per above and pushed the code change.
I'll see if I can get a new beta built so that you can experiment and see if there is any change in behaviour. It would be a good idea to see if you can trigger the problem with the existing epgcache code, then retest after applying the update and see if the problem persists or whether this change fixes the issue.
No estimate as to when I can actually produce a new beta build. If I'm lucky, it could be later today, but it will be untested.

Cheers, no hurry. Whereabouts can I download your latest beta? I can't seem to find the closed beta area.

[MrQuade]

It'll get pushed out as an online update for all of us to test I'd say. You're registered to the beta feed yes?

[sonicblue]

It'll get pushed out as an online update for all of us to test I'd say. You're registered to the beta feed yes?

Not currently (I'm running 19.3.20200328 from the Public Beta forum).

[MrQuade]

It'll get pushed out as an online update for all of us to test I'd say. You're registered to the beta feed yes?

Not currently (I'm running 19.3.20200328 from the Public Beta forum).

The"public" beta is the only beta.

If you are running the beta image, you should be switching over to the beta feed. You can run into problems if you try to install packages from the public feed.

[peteru]

Instructions for switching to the beta feed are here: viewtopic.php?f=56&t=9085

The script isn't very complicated, it just changes the conf files in /etc/opkg/ to use beta.beyonwiz.org instead of feeds.beyonwiz.org

I also tend to upload full USB images for some beta releases. When I do that I post an announcement in the Public Beta section of the forums.

Earlier you mentioned that sometimes you see missing entries in the EPG, then later these entries will appear in the EPG. It would be interesting to see the details for these particular events (and any adjacent events) and compare the two sets. I'd be looking to see if the missing entry that comes back causes the eviction of other entries, either via a duplicate event_id or because of overlap.

[sonicblue]

So I've just checked my TVs internal tuner EPG, and guess what, no event data for those exact same events

I'll need to do more testing to make sure; but for now it's looking like the tv networks fault

[prl]

I'll need to do more testing to make sure; but for now it's looking like the tv networks fault

I've never thought otherwise.

BTW, while the whole broadcast EPG is repeated every 10 sec (or should be, according to OP58), the IceTV EPG is loaded in full at GUI startup, and after that it only loads the EPG data that has changed since the last update, unless the user does a Disable IceTV/Enable IceTV.

[sonicblue]

I'm still curious to know if it can be worked around though

If it's due to the networks illegally broadcasting overlapping start and end times, then in theory this could be worked around. If I can manage to build according to MrQuade's guide, this opens the possibility of putting some debug lines in epgcache.cpp to see what's actually being broadcast.

Honestly though I probably won't get around to doing that as I'm quite burnt out from spending the last 3 months learning from scratch Python, Enigma and writing my autosharpness plugin and V2 patch. Perhaps you are also burnt out from my neverending questions in the Hello World thread

[sonicblue]

I'm currently observing gaps in the guide data for ABC, but not on my TVs internal tuner EPG. This is contrary to the previous observation where both showed gaps on SBS channels. Perhaps there is more than one issue at play here.

[sonicblue]

The latest beta 20200901 firmware still has the issue unfortunately (I presume this latest version contains Peter's iterator fix?).

It seems the issue only affects the ABC channels, since they appear to be the only ones with missing events exclusively on the Beyonwiz.

It appears to only be an issue when tuned to the ABC multiplex. It seems possible to partially workaround the issue by simply not refreshing from the EPG cache when tuned to ABC multiplex, and then only refreshing ABC EPG when the box is not in use (eg. overnight). I say this as I've never observed any missing ABC events as long as I'm not tuned to ABC, and the missing events seem to always be the ones neighbouring the current event.

[MrQuade]

So an issue with the EPG data conflicting now/next data?

[peteru]

Sounds very much like that.

[prl]

I'm sure that it will be a surprise to few, but the broadcasters don't even follow their own agreed standards on what times are shown in the EPG.

The accuracy of the timing information in the EITschedule_actual shall reflect the timing in the published station schedule.

(my emphasis)

But if the Beyonwiz code is also using now/next to update the EPG (and from memory, it does), and the stations follow that rule, then the now/next will cause conflicts with the main EPG data:

The timing of events may differ from timing information in the EITp/f_actual as this may be amended closer to the air time of the event.

(EITp/f_actual is the now/next information).

Freeview Operational Practices OP 58 Implementation Guide for DVB EIT Schedule Information (Section 2.5 Schedule Accuracy)