CRIDs

[adoxa]

I think the CRID data, in itself, is essentially useless. As mentioned earlier, I'll extract the information I can, adding it to my Single EPG info and the EPG search results. I'll push the code, but don't intend to release it.

[IanSav]

Hi Adoxa,

I think the CRID data, in itself, is essentially useless.

I agree. The initial value in the CRID data is as a point of comparison between events for the purpose of trying to more reliably identify separate programs by a method other than the broadcast title (which often contains obfuscating data). Also to try and identify new or repeat episodes of an individual program.

I'll push the code, but don't intend to release it.

Understood. I think I am beginning to understand your development style.

Regards,
Ian.

[prl]

If you are proposing that the C++ layer simply returns the raw CRID data and the converter can/will assemble the parts under Python / Converter control then I think this is a better way to implement the CRIDs.

What would that be? Should it, for example, include returning the Content Identifier Table so that CRIDs with their location specified as "Carried in Content Identifier Table" can be constructed? Tables of the network-level and transport stream-level and sevice-level default authorities for abbreviated CRIDs?

The current implementation of the Python interface simply returns a list of tuples representing eCridData structures and containing fully-specified CRIDs for an event as a list of text strings. I'm not sure whether there's any real need for the CRID components to be displayed. I'm not even convinced there's a need for a Converter for CRIDs at all. If the Python code wants to split the CRID into its logical component parts, there's always:

Code: Select all

>>> import urlparse
>>> urlparse.urlsplit("crid://canberra.nitv.sbs.au/211351#IM")
SplitResult(scheme='crid', netloc='canberra.nitv.sbs.au', path='/211351', query='', fragment='IM')
>>> 

The Crid-dev code has simple examples of how to use the interface in Components.Converter.EventName.

[prl]

I just have member strings for the authority (without scheme)

Is Trevor at work there?

[adoxa]

I just have member strings for the authority (without scheme)

Is Trevor at work there?

No, I have std::string m_authority, which does not include crid://.

[IanSav]

Hi Prl,

I didn't know there was a built in library to easily pull the URL format apart.

Regards,
Ian.

[prl]

I just have member strings for the authority (without scheme)

Is Trevor at work there?

No, I have std::string m_authority, which does not include crid://.

Why not use the existing API and reformat the CRID into the form you want:

Code: Select all

	def getCrid(self, event, types):
		crids = event.getCridData(types)
		if not crids or not crids[0][2]:
			return ""
		try:
			import urlparse
			fields = urlparse.urlsplit(crids[0][2])
			ret = fields.netloc + fields.path
			print "[EventName] getCrid", ret
			return ret
		except:
			return "unparseable CRID"

Code: Select all

{1178}<   442.444> [EventName] getCrid canberra2.abc.net.au/NC1823C
{1178}<   442.445> [EventName] getCrid canberra2.abc.net.au/NC1823C040S00

[adoxa]

Here's my CRID code (and its Full-Metal-Wizard skin).

Here's an EPG search showing the CRID for each channel, as well as in the info (if there's no CRID the icons are flush on the right):

Event info adds it to the name:

crid-eventview.png (6.2 KiB) Viewed 3489 times

The movie list places it to the right of the genre:

crid-movieselection.png (8.34 KiB) Viewed 3489 times

IanSav, if you'd like to point your Seven contact here, maybe they can tell me what I missed.

[IanSav]

Hi Adoxa,

I will reach out to him tomorrow. I doubt he will look at your code (he is an engineer and not a programmer). I am also not sure how much information he will be allowed to give me. I'll see what I can find out.

If I have enough time I will try to reach out to my SBS contact to see why they have all the different authorities. It would be good if SBS joined Seven and Nine with a single authority across all their LCNs. I have a friend who works at the ABC. He was in TV but is now in radio. He may have an old contact that can help me with the same question for the ABC.

I don't think I have any contacts within Ten.

Regards,
Ian.

[IanSav]

Hi,

I contacted my ABC friend. He is not involved with areas that could be of assistance to this matter. He is going to see if he can find some appropriate contacts for me. No promises.

By the way, has anyone looked at the CRIDs for DVB-T radio? ABC has no CRID data at all. SBS has very strange looking Episode CRID data only.

Should the DVB-T radio stations have CRID data?

Regards,
Ian.

[prl]

SBS has very strange looking Episode CRID data only.

SBS has completely non-conforming CRIDs where what looks like it might be Episode CRID path components have been put into the authority part of the CRID, and the path part of the CRID has been left empty.

From Grumpy_Geoff's data: SBS Radio 1 14.08, 21:00 crid://8931f97735014d348c16eb619aee552a Korean

Code: Select all

>>> from urlparse import urlsplit
>>> urlsplit("crid://8931f97735014d348c16eb619aee552a")
SplitResult(scheme='crid', netloc='8931f97735014d348c16eb619aee552a', path='', query='', fragment='')

The result field names use the URL conventions, but the parsing shows how broken the CRID is.

Also, according to Freeview OP-72, the maximum permitted length of the data part of the CRID is 29 characters, including the leading "/". If the SBS Radio CRIDs were to be re-written to have a sensible authority and use the existing "authority" as the data part, the data part is longer than permitted in OP-72 - it's 33 characters including the "/".

Should the DVB-T radio stations have CRID data?

Good question, but no matter what the answer to that question is, they shouldn't be including non-conforming CRIDs in any service.

On the original question, as far as I can see, OP-72 doesn't mention radio services either way.

More naughty words that are censored by Sucuri: data:

[IanSav]

Hi,

I had a long chat with my contact at Seven. The CRID encoding is not done directly by Seven but is an output of a black box system that Seven uses to generate their operational data. Thus, the encoding is the intellectual property of an organisation outside of the Network. I was reminded that we don't need to know what the CRIDs are but simply use them as tokens for comparison to see if one event relates in any way to another even. There is one issue that needs to be noted. When these systems are changed by the broadcasters the CRID encodings *can* change! This happened to Seven when they upgraded equipment from one manufacturer to that of another.

My contact also indicated that Seven intends their CRIDs to be applicable across all their LCNs. While the aim was to have the CRIDs unique across ALL broadcasters this has not happened. I was cautioned that while the intention for Seven is for the CRIDs to be uniform across all the Seven LCNs he had not specifically verified that this is the case.

I raised the issue of the differing authorities by some broadcasters. The suggestion was that this should be a difference that can be ignored. This is not an issue for Seven as they use a single authority. It was suggested that when we code the series recording capability into Enigma2 we add a configuration option to allow for minor differences in the authority to be ignored. I will take this up with SBS when I contact them.

My contact has no involvement with Radio LCNs so had no opinion as to whether or not they should have CRIDs. He asked if anyone would record a radio program. I said it was possible and that the Radio LCNs do have an EPG and there are events that can be separately identifiable that can also repeat. This is what CRIDs are used to identify. I will also ask this question of SBS when I contact them.

Is there anything else you want me to find out?

Regards,
Ian.

[prl]

Does your contact have any information about Prime CRIDs? Prime Canberra used to carry CRIDs (e.g. in Nov 2016), but no longer appears to. When Prime Canberra carried CRIDs, they appear to have been in an obfuscated form similar to that used by Seven's CRIDs now. E.g.:

Code: Select all

crid 0946:099e:3281 episode 2016-11-22T12:01 <crid://can.6.primetv.com.au/qufquwemda4> <PRIME7 Canberra> <Jack Reed: Death And Vengeance>

Also. when Prime Canberra carried CRIDs, they weren't under a single authority name:
can.6.primetv.com.au PRIME7 Canberra
can.62.primetv.com.au 7TWO Canberra
can.63.primetv.com.au 7mate Canberra
can.65.primetv.com.au ishoptv
can.68.primetv.com.au RACING.COM

[adoxa]

The CRID encoding is not done directly by Seven but is an output of a black box system that Seven uses to generate their operational data.

That would explain why RACING.COM is different.

My contact also indicated that Seven intends their CRIDs to be applicable across all their LCNs. [...] I was cautioned that while the intention for Seven is for the CRIDs to be uniform across all the Seven LCNs he had not specifically verified that this is the case.

It does look like the Seven series CRID is consistent across their LCNs. But as a series identifier, it's not directly comparable as it usually includes a season identifier. Some ids are only four characters, but that's not enough to uniquely identify a series (e.g. Life Off Road and The Lion Guard both have the same first four characters). Harry's Practice is one instance where even the first six characters are different.

I raised the issue of the differing authorities by some broadcasters. The suggestion was that this should be a difference that can be ignored.

For my purposes I just test the suffix (e.g. .sbs.au), but that might not work nation-wide.

Is there anything else you want me to find out?

I'm good, thanks.

[IanSav]

Hi Prl,

Does your contact have any information about Prime CRIDs? Prime Canberra used to carry CRIDs (e.g. in Nov 2016), but no longer appears to. When Prime Canberra carried CRIDs, they appear to have been in an obfuscated form similar to that used by Seven's CRIDs now. E.g.:

Code: Select all

crid 0946:099e:3281 episode 2016-11-22T12:01 <crid://can.6.primetv.com.au/qufquwemda4> <PRIME7 Canberra> <Jack Reed: Death And Vengeance>

Also. when Prime Canberra carried CRIDs, they weren't under a single authority name:
can.6.primetv.com.au PRIME7 Canberra
can.62.primetv.com.au 7TWO Canberra
can.63.primetv.com.au 7mate Canberra
can.65.primetv.com.au ishoptv
can.68.primetv.com.au RACING.COM

I will EMail this as a follow up question.

I also received an SMS from my friend in the ABC this afternoon. He may have a contact for someone who can help with CRID information. If he is permitted to talk to people outside of the ABC I have arranged for them to get my telephone number. This may get us some information about the variable authorities and the applicability of CRIDs to radio broadcasts.

My contact at SBS wasn't answering their phone this afternoon. I will try again next week.

Regards,
Ian.

[IanSav]

Hi,

The next question will become who wants to tackle changing AutoTimers to add the option of using CRID data to create the series link functionality with repeat event filtering. I will call SHOTGUN and say NOT ME! To be sure I will also take a preemptive step back.

By the way, if anyone looks at AutoTimers can they please consider looking at all timers with a view to capturing the start and end padding so that event details can be better calculated.

Regards,
Ian.

[prl]

By the way, if anyone looks at AutoTimers can they please consider looking at all timers with a view to capturing the start and end padding so that event details can be better calculated.

That's probably an even bigger can of worms than using CRIDs in AutoTimer.

I've had no problems getting some Beyonwiz-specific code past the gatekeepers of the enigma2-plugins repository (specifically adding IceTV timer icons to the EPGSearch search results). Of course, with suitable protections so that the code worked on other enigma2 systems.

[prl]

The next question will become who wants to tackle changing AutoTimers to add the option of using CRID data to create the series link functionality with repeat event filtering. I will call SHOTGUN and say NOT ME! To be sure I will also take a preemptive step back.

I'm not convinced that it would be all that much better than using AutoTimers as they are now. Things like the fact that the ABC encodes the season number into their series CRIDs will make CRID-based AutoTimers fail in ways that are at least as confusing as the ways that current AutoTimers fail.

In the ACT, only 3 of the 5 broadcasters have CRID coverage, anyway (ABC, SBS & SCA).

OP-72 just doesn't specify well enough what the CRIDs should represent. And, of course, some broadcasters just ignore bits of it anyway, or can't be bothered at all. The fact that Prime Canberra once carried CRIDs and no longer does, and the various other problems inconsistencies and lack of data don't give me the impression of an industry determined to make CRIDs work.

[prl]

This continues new discussion of the use of CRIDs to improve the reliability of AutoTimers that was started here.

IIRC, last time I looked, CRIDs weren't particularly useful for series recording.

Not at all, see here (in general, the series id of a show changes with each season, possibly even each channel).

I was fairly sure that I'd remembered correctly, but thanks for the link to your summary.

Properly implemented CRIDs would probably be a good resource for AutoTimers. What we have, much less so.

If there's anyone who's still enthusiastic about using CRIDs to make AutoTimers better, I'm happy to get my CRID extraction code working again and give it to them. NB: the main works of it are C++, with a few Python hooks so that CRIDs can be displayed in the GUI. I don't think that there's any call for displaying them, but IIRC, IanSav thinks there is.

But here's a run-down of the reasons why I don't think that CRIDs are useful for AutoTimers. Last time I looked at CRIDs:

• As adoxa details in the post he linked to, show CRIDs don't stay the same for different seasons of the same series.
• There are multiple naming authorities for CRID data, and series and episode CRIDs are not consistent between different broadcasters.
• For some broadcasters, there are multiple naming authorities for the same broadcaster, though it's possible that the naming is consistent between different naming authorities for the same broadcaster, though there is no reason to expect that they would be the same. E.g, for ABC Perth: crid://perth2.abc.net.au, crid://perth20.abc.net.au, crid://perth21.abc.net.au, crid://perth22.abc.net.au, crid://perth23.abc.net.au, crid://perthabcnews24.abc.net.au.
• According to FreeTV OP 72, broadcasters are supposed to use a private data specifier in the range 0x3200-0x320f to flag that the same loop contains CRID data. Seven Perth uses 0x233a and Prime7 Canberra uses 0x0029. Those values appear to already be allocated to other broadcasters.
• In Canberra, WIN carries almost no CRID data.
• Some SBS radio services use CRIDs that are in a non-compliant form (e.g. crid://9cc19ee1-53a4-473d-ba02-21a7b5d82ed8). They are not syntactically valid CRIDs, and even if they were, the part that is apparently supposed to be the data part of the CRID (the /9cc19ee1-53a4-473d-ba02-21a7b5d82ed8) is longer (37 characters) than OP 72 allows (29 characters).
• The FreeTV OP 72 spec allows for the CRID default authority (DA) (used for compact CRIDs that don't have an authority) to be in the SDT, NIT and the Transport Stream loop of the NIT. I have coded all of those, but as far as I can see, only the SDT and NIT TS used for DAs, so the NIT code remains untested. The DVB standard also allows DAs in the BAT. The Beyonwiz enigma2 code has a hook for processing the BAT, but it's unused in Australian broadcasts, so I have no way of testing DA extraction code for the BAT.
• It would be preferable to have CRIDs used in the oe-alliance enigma2-plugins AutoTimers, but that would require the CRID capability to be in all enigma2 images that continue to cherry-pick from the oe-alliance repository. However, I have no way to test anything to do with CRIDs and their use outside Australia.

Really, to actually be useful for AutoTimers, CRIDs ideally should have:

• A single naming authority for the whole scope of the EPG data.
• Consistent series CRID naming for different seasons of the same series.
• Historically consistent naming of both series and episode CRIDs, so that long-term series AutoTimers can properly follow seasons of the same series (also needs the point above), and so that CRIDs can potentially be saved to allow repeat detection in the absence of any consistent repeat metadata in the rest of the EPG. Even with that implementation might be a bit tricky (e.g. handling cancelled shows).
• All services to carry full CRID data.
• Have all the other inconsistencies and errors cleaned up.

For Australian FTA TV, I think that's "tell 'em they're dreamin'" territory. Similarly for the European market where a user's EPG may have FTA, Cable and Satellite services included.

[prl]

I still have my most recent implementation of my CRID extraction code. If anyone is seriously interested in implementing CRID matching in AutoTimers, I'm willing to get that code running again and give it to them as a set of diffs that can be applied using patch.

Implementing that would require knowledge of C++ and Python, and the willingness to grapple with the morass that is the AutoTimer code.

Getting it all accepted into other enigma2 images would require quite a bit of research into how CRIDs are used (or not) in other markets, and possibly a not inconsiderable diplomatic effort.

Personally, I'm with peteru's comment up-thread that it simply isn't worth the effort (he worded it rather more strongly than I did )

[adoxa]

[...] Python hooks so that CRIDs can be displayed in the GUI. I don't think that there's any call for displaying them, but IIRC, IanSav thinks there is.

I don't see a need to display raw CRIDs, but go back up a bit to this post and showing decoded CRIDs is somewhat useful.

[prl]

[...] Python hooks so that CRIDs can be displayed in the GUI. I don't think that there's any call for displaying them, but IIRC, IanSav thinks there is.

I don't see a need to display raw CRIDs, but go back up a bit to this post and showing decoded CRIDs is somewhat useful.

True, for CRIDs from some broadcasters in the AU market. All bets are off for any other market.

There's no specification in the relevant DVB standard (ETSI TS 102 323) as to the internal structure of the data part of a CRID, other than the syntax to mark the IMI part. SBS chose random UUIDs for what seems to be intended to be the data part of the CRIDs its radio services, for example.

[prl]

I have my experimental code to handle CRIDs working in firmware 20191106.

If anyone is really interested in trying to implement the use of CRIDs in AutoTimers (or for anything else), I'm happy to post the diffs here.

The diffs include code and changes the easy-skin-aus-hd skin to display CRIDs in the EPG and in recordings in the media selection screen, provided the EPG entry for the program has CRID entries. The code changes for EventName converter used to display the CRIDs in the UI show an example of how to access an event's CRID data.