CRIDs

[IanSav]

Hi Prl,

According to my understanding of CRIDs, supported by my reading of WikiPedia, CRIDs are meant to be unique. There can be different CRIDs that point to the same content but no single CRID can be used to point to different content. ( WikiPedia - CRID )

Edit: This isn't to say that our TV stations aren't using the facility correctly.

Regards,
Ian.

[IanSav]

Hi Prl,

It appears that CRIDs may also be available in parts of the UK. I am trying to get more information for you.

Regards,
Ian.

[prl]

... It appears that CRIDs may also be available in parts of the UK. I am trying to get more information for you. ...

CRIDs can appear in a number of different contexts in DVB-T. Only one of them is used in Australia (though a number of them are allowed). I'm not sure how applicable code that I can test here would be in the UK.

And then there's the problem of the Content Identifier Table, which provides indirect CRIDs. Their use is allowed in OP 72, but direct CRIDs are noted as preferred. They're not used in Canberra broadcasts, and I'd guess not used elsewhere in Australia, either.

No idea whether they're used in the UK.

[IanSav]

Hi All,

Sorry to bump and old thread but I have some potentially good news on the CRID front...

A friend at Network Seven contacted me today to inform me of some changes coming to OP 61 (This is the old link that will soon be updated to Issue 4.)

The changes are being brought about due to the looming demise of Freeview EPG (MHEG) at the end of next week (Freeview Plus will be the only EPG service operating via HbbTV).

The interesting item to note is paragraph 6.5:

6.5 Carriage of CRIDs in EIT actual schedule It is required that Australian television broadcasters carry content reference information (CRIDs) for all content referenced in the broadcasters schedule (8 day) event information table. This information shall align with Section 4 of AS4599.1 - 2013. It is expected that Hbb TV compliant PVR manufacturers shall be able to utilise this information to manage scheduled recordings.

I believe that the changes to OP 61 should be a good sign that Prl's experimental work on CRIDs should resume / continue. Now that they will be required this should make for a far more useful AutoTimer system (assuming the spaghetti code of AutoTimer can be understood and support changes to use CRIDs.)

Regards,
Ian.

[prl]

Is that quote from the "new" OP61? The quote looks exactly the same as what is in the current (Dec 2014) one:

6.5 It is required that Australian television broadcasters carry content reference information (CRIDs) for all content referenced in the broadcasters schedule (8 day) event information table. This information shall align with Section 4 of AS4599.1 – 2013. It is expected that HbbTV compliant PVR manufacturers shall be able to utilise this information to manage scheduled recordings.

I may have another look at CRIDS when I have some spare time. The last time I looked at them, in November last year, some of the data seemed to be pretty much unusable for what people want to use it for (e.g. at that time the Canberra EPG data for WIN and PRIME carried no usable CRID data).

To even get CRIDs into the Beyonwiz EPG data and have access to them from the EPG requires changes in 12 of the C++ files in lib/dvb and lib/service. That's without providing any actual user-level improvements. I'm unable to easily offer the changes upstream, because the standard allows for more places where CRID data can be carried than are used in Australian broadcasts. That means I'd be unable to offer tested code for those cases.

And then there's the whole issue of getting the necessary changes to allow for any user benefit accepted into plugins like AutoTimer.

[IanSav]

Hi Prl,

That quote came from my contact at Seven. I didn't have time to check how the quote matches or differs from the Issue 3 version. I don't have access to the Issue 4 version.

My interpretation of the message sent to me was that with MHEG going all broadcasters *should* be adding or using CRID data to enable PVRs and users to manager recordings. I take this to mean that the providers you checked that don't have CRID data should be adding it. If they aren't then their attention should be directed to OP 61 and requested to add the information as directed therein.

I appreciate that this is not an easy task but I would gladly assist, to the best of my abilities, to get this happening for our users.

I had a chat with Huevos and it appears that CRID processing would be of limited appeal. The data is simply not commonly available in most other markets. (That is why XML imports are so common and important in other countries.)

While Beyonwiz draws on OpenViX for much of its upstream code the reality is that OpenViX draws a lot of code from OpenPLi. For something like CRID processing, and all the deep rooted changes required to support it, to be acceptable for OpenViX it would also need to be acceptable to OpenPLi. Further it is most likely that the code changes will need to be injected at the OpenPLi level. This is usually *VERY* difficult to achieve.

I feel it could be worth the effort for Australian users. If the code causes no harm for the rest of the world then it could be submitted to go into the core code. The advantage for them could be that if/when CRIDs become available in a market then the core code would be available to support their use. What do you think?

Regards,
Ian.

[prl]

...
That quote came from my contact at Seven. I didn't have time to check how the quote matches or differs from the Issue 3 version. I don't have access to the Issue 4 version.

My interpretation of the message sent to me was that with MHEG going all broadcasters *should* be adding or using CRID data to enable PVRs and users to manager recordings. I take this to mean that the providers you checked that don't have CRID data should be adding it. If they aren't then their attention should be directed to OP 61 and requested to add the information as directed therein.

As the quote says, they should have been doing it since at least December 2014, but as of November 2016, they still weren't, at least not uniformly.

I appreciate that this is not an easy task but I would gladly assist, to the best of my abilities, to get this happening for our users.

Most of the supporting code is written and tested, though because it's been a while since I last worked on it, it may need a bit of rework to fit into the current code and will certainly need retesting.

What sort of assistance would you be able to lend? The code to put the CRID data into the EPG and lamedb (for the default authority names) is already written and tested, as is the ability to fetch CRIDs from an event and use it in the Python part of the UI code. An exemplar of the use in the Python code is in the added CridSeries, CridEpisode and CridRecommendation types for the EventName converter.

What's left is what I think is the hard bit: trying to integrate that into AutoTimer in a sensible way and so that AutoTimer still works in systems that don't support CRIDs. See below for some of the "interesting" properties of CRIDs, at least as they were in November 2016.

I had a chat with Huevos and it appears that CRID processing would be of limited appeal. The data is simply not commonly available in most other markets. (That is why XML imports are so common and important in other countries.)

While Beyonwiz draws on OpenViX for much of its upstream code the reality is that OpenViX draws a lot of code from OpenPLi. For something like CRID processing, and all the deep rooted changes required to support it, to be acceptable for OpenViX it would also need to be acceptable to OpenPLi. Further it is most likely that the code changes will need to be injected at the OpenPLi level. This is usually *VERY* difficult to achieve.

II feel it could be worth the effort for Australian users. If the code causes no harm for the rest of the world then it could be submitted to go into the core code. The advantage for them could be that if/when CRIDs become available in a market then the core code would be available to support their use. What do you think?
...

I really don't see how your expectation in the third paragraph that the code would be accepted into the upstream code follows, given what's in the two previous paragraphs, especially if, as is quite possible, the code to extract the CRIDs into the EPG doesn't work at all in the European scene, even for those markets that carry CRIDs.

There's also the possibility that DVB-T implementations can use a Content Identifier Table to hold the CRIDs themselves, and index this table by a 16-bit reference in the event's content_identifier_descriptor field.

One specific issue is that CRIDs in the EPG can refer to a default authority: a CRID can be specified as simply "/cridSpecificData". The authority for that CRID is supplied by service-specific data. That data can appear in any of 5 different locations in the SI data, but Australian broadcasts (as of Nov last year) allow only three of them, and only use one of them, service descriptor loop of the Service Description Table (SDT). I can't test code to extract them from any of the other 4 possible locations (though I've written code to do it).

There are a number of issues with CRIDs that are unclear and affect how they can be used even just in the Australian context.

One is that with the CRID data as it was last year, it's really unclear exactly what the scope of an authority is. For example, ABC Canberra had no fewer than five authorities:
'ABC' 'ABC News 24' <CRID://canberraabcnews24.abc.net.au>
'ABC' 'ABC' <CRID://canberra2.abc.net.au>
'ABC' 'ABC2/KIDS' <CRID://canberra22.abc.net.au>
'ABC' 'ABC' <CRID://canberra21.abc.net.au>
'ABC' 'ABC ME' <CRID://canberra23.abc.net.au>

That means that, strictly speaking, Canberra ABC CRIDs have 5 independent name spaces, and the same entity need not have the same data part in different authorities. That doesn't seem to be the case for the ABC. They seem, sensibly, to use common reference data for the same entities across all the authority names. But how does the code get to know that, and what is the extent of the commonality? E.g, if someone can receive both ABC Canberra and ABC Wollongong, just what part of the authority name is actually significant? And just what are the rules for other networks?

There isn't any sensible commonality of form for the authority names across broadcasters, either:
canberra.1.sca.au
canberra21.abc.net.au
can.sr1.win.au
can.6.primetv.com.au
canberra.sbshd.sbs.au

There isn't even any commonality about which position the network name appears in the authority name, nor in what way (or position) the channel name appears (not even within the same broadcaster, e.g. compare canberra2.abc.net.au and canberraabcnews24.abc.net.au).

One thing that is pretty certain, though, is that it won't be possible to use the data part of CRIDs to record a show no matter which network it appears on (<series CRID data>, <episode CRID data part>:
ABC: <The Jungle Book> </ZX9477A> </ZX9477A011S00>
SBS: <Spanish News> </199869> </S61561520161118>
SCA: <Hyde & Seek> 32 </16462> </16462D1_8>
Prime: <The Neighbors> </TkVJRw> </TkVJRwEMDA0>
WIN: <Shopping, Travel, Entertainment, Lifestyle> </8278EE1339104373A8D9C48A148> </2E6F8E49812D4A17AAAFBB49CC6>

[IanSav]

Hi Prl,

As the quote says, they should have been doing it since at least December 2014, but as of November 2016, they still weren't, at least not uniformly.

The interesting thing is why did I get the unsolicited message from my contact at this point in time? Perhaps there are changes coming into place that may improve the potential for actively using CRID data.

What sort of assistance would you be able to lend?

I could help with any skin changes. I could help with testing. I would try and help with anything you asked.

I really don't see how your expectation in the third paragraph that the code would be accepted into the upstream code follows, given what's in the two previous paragraphs, especially if, as is quite possible, the code to extract the CRIDs into the EPG doesn't work at all in the European scene, even for those markets that carry CRIDs. ...

If the code is well written and does no harm then why shouldn't it be included?

As I understand the situation CRID data does exist in some other markets but the data content isn't obvious as the data is exploited for commercial benefit.

Just because there are more options than you can test doesn't mean the the options you can test should not happen. The code could be used as a structure or platform for others to extend should the CRID option become available elsewhere. I would compare the existence of the Genre and Ratings code that exists in Enigma2 but is rarely used as the underlying data is mostly unavailable.

The choice is your's to make. I would simply like to encourage you to consider progressing the code to an implementation for the Beyonwiz.

As for the inconsistency of the CRID data. That is an issue. However, my understanding is that the data is fully consistent for each LCN and possibly consistent for a broadcaster. For me that means that recording a program that is unique to an LCN should be fine using CRIDs. Thus recording something like the ABC News or Gruen etc on ABC HD should be easy to achieve. Recording something like Mythbusters may be problematic given that this program is often screened on both SBS and Network Seven. Similarly The Big Bang Theory screens on both the Seven and Nine Networks. In these cases the CRIDs for the different channels is probably not comparable. However, even with this restriction it would still be very useful to be able set a timer to record every unique episode of Mythbusters on SBS and every unique episode of Mythbusters on Network Seven. Yes it could mean the use of two AutoTimers but the results should be significantly more reliable than current title and description matching rules.

Regards,
Ian.

[IanSav]

Hi Prl,

I have been in discussion with some OpenViX developers and they are very interested in the CRID code.

I have been given some .TS file captures of British TV for your analysis to see if that helps you identify the CRID data and if your code works or can be made compatible. I have been trying to contact you offline to arrange for the data transfer.

I would like to propose that you continue with your CRID code development. I would ask that the first phase be to create a display converter (or update an existing one) to make the CRID data available in the skin. This could be used by developers and testers to see what CRID data is available and get a feel for if or how the CRID data could be useful. If the testing goes well we could look at a second phase of implementing the CRID data into AutoTimers.

What do you think?

Regards,
Ian.

[prl]

Hi Prl,

I have been in discussion with some OpenViX developers and they are very interested in the CRID code.

I have been given some .TS file captures of British TV for your analysis to see if that helps you identify the CRID data and if your code works or can be made compatible. I have been trying to contact you offline to arrange for the data transfer.

That would potentially be useful, assuming that it's a whole stream capture, but I'm not sure how easy it would be to feed the relevant bits of it into the EIT decoder. There's also the problem that the area in which there is the most diversity in permitted ways of encoding it is in the tables that give the default authority names, and that is only extracted at scan time.

I would like to propose that you continue with your CRID code development. I would ask that the first phase be to create a display converter (or update an existing one) to make the CRID data available in the skin.

That is already done. I have already added EventName converter types CridSeries, CridEpisode and CridRecommendation to the EventName Converter. This could be used by developers and testers to see what CRID data is available and get a feel for if or how the CRID data could be useful. It was discussed at the time I was last working on the code (and even demonstrated, initial discussion following that post), and I have mentioned it in this recent discussion (in the post you are replying to).

Personally, I think that lists of CRIDs against program name and service name are more useful for getting a feel as to what CRID data looks like. I posted some of that information last year when I was last working on CRIDs. I see little utility (apart from for developers) in having the CRIDs available in the EventName Converter, though I'm inclined to leave them in.

If the testing goes well we could look at a second phase of implementing the CRID data into AutoTimers.

What do you think?
...

The testing of the existing changes has already gone well, when I did it a year ago. It works quite nicely. There may be some fixups needed to adapt the code to any changes to the base code since I last worked on it, but I don't expect that to be a big problem.

My main questions in all this are about things that are not stated in the FreeTV Operational Practices, like just what the actual scope is of an authority name (as opposed to its apparently intended scope in the standards). For example:

Code: Select all

<ABC> <Hard Quiz> 2016-11-23T20:01 <crid://canberra2.abc.net.au/le1531v006s00>
<ABC> <Hard Quiz> 2016-11-23T20:01 <crid://canberra21.abc.net.au/le1531v006s00>
<ABC2/KIDS> <Hard Quiz> 2016-11-24T20:30 <crid://canberra22.abc.net.au/le1531v006s00>
<ABC> <Hard Quiz> 2016-11-25T22:20 <crid://canberra2.abc.net.au/le1531v006s00>
<ABC> <Hard Quiz> 2016-11-25T22:20 <crid://canberra21.abc.net.au/le1531v006s00>

Where they are:

simulcast of Hard Quiz on LCNs 2 & 21 (ABC) 2016-11-23T20:01,
"encore" of Hard Quiz on LCN 22 (ABC2/KIDS) 2016-11-24T20:30 and
"encore" simulcast of Hard Quiz LCNs 2 & 21 (ABC) 2016-11-25T22:20.

Here, at least the authorities canberra2.abc.net.au, canberra21.abc.net.au and canberra22.abc.net.au are all using the same data part of the CRID to denote the same episode of Hard Quiz, even though as independent naming authorities, there is no a priori reason to think that they would all use the same names for the same entities.. To what extent is that true across the ABC's probably hundred or more naming authorities? There are good practical reasons why they might use the same names, but then why aren't aren't the naming authorities structured to reflect that? Why structure the CRIDs such that the same episode of Hard Quiz might be permitted to be named differently in the CRID on ABC and on ABC2?

As far as I can tell, all networks use similar region/channel naming in their CRID authority names. I could understand that network affiliates like Prime or WIN might want different naming authority from their "parent" networks, but why does, say, SBS One Perth need a different namespace from SBS One Canberra, and those two need different namespaces from VICELAND Perth and Canberra?

The argument may seem academic, but it goes to the function of being able to set a CRID-based AutoTimer that is allowed to (and expected to) track the same program between channels, at least of the same network, if not across networks. And if the authority name is ignored, is there any risk that a CRID from another naming authority may have the same value, but represent a different entity, between networks (my experience so far is that that is extremely unlikely) or even within networks? However, assuming that the data part alone is unique to a network and always represents the same entity within that network breaks what I understand the role of naming authorities to be.

[prl]

....
That quote came from my contact at Seven. I didn't have time to check how the quote matches or differs from the Issue 3 version. I don't have access to the Issue 4 version.

My interpretation of the message sent to me was that with MHEG going all broadcasters *should* be adding or using CRID data to enable PVRs and users to manager recordings. I take this to mean that the providers you checked that don't have CRID data should be adding it. If they aren't then their attention should be directed to OP 61 and requested to add the information as directed therein.

OP61 Issue 3 Dec 2014 says "It is expected that HbbTV compliant PVR manufacturers shall be able to utilise this information [CRIDs] to manage scheduled recordings." A year ago, WIN and Prime had incomplete CRID data and WIN had incomplete authority name data, so they certainly hadn't bothered to do what they should for at least two years.

The issue I see are more about naming scopes and dealt with at tedious length in my previous post.

A year ago, SBS was even sending CRIDs for its radio stations that had no authority name (ie, crid://data_part)

So yes, broadcasters should indeed now be sending CRID information. But they should have been doing that for nearly the last 3 years (and possibly for longer). It's very unclear to me just what's supposed to have changed from what you've posted from your discussion with your Seven contact. I guess time will tell when the new OP61 is published.

So I remain sceptical. To understand my scepticism:

[My Seven contact] was please to report that at least all stations now populate the CRID data fields. (If I remember the conversation correctly Seven was the last channel to give in and publish the data in the open.)
...

...
To me, it certainly doesn't seem to be the the case, no matter what your Seven contact says, that "all stations" have populated CRID data. Prime Canberra, a Seven affiliate, doesn't appear to, to any usable extent. Neither does WIN.

Note the dates.

[IanSav]

Hi Prl,

I am probably missing the point of your post regarding authorities of the CRIDs but my immediate expectation is that the authorities don't really matter for a single broadcaster. I suspect that all the ABC LCNs, as per the example, will use compatible data. The fact that the data is not 'clear' text is annoying but not a show stopper. For the moment we can use the CridSeries and CridEpisode data as simple comparison values the see if a program and/or episode matches another program and/or episode on the same network.

The limitation of authorities and the obfuscation of the CridSeries and CridEpisode data will mean that we can't match shows across broadcasters. This is not optimal but should still result in more accurate AutoTimer recordings from any single broadcaster.

I agree with your concern that I am assuming that the CRID data will be consistent across a broadcaster. If the different regions or affiliates change the data will it *really* matter for a user and their AutoTimer recordings. As long as the data from the broadcaster they are using is internally consistent then all should be fine.

When you get time to go back to the CRID exploration I will ask my contact if more information about the authorities and the reason for obfuscation of the CRID data. I feel that if I wait until you are ready to return the CRID effort there may be a better opportunity to get more/better information. That is, the new OP 61 may be out and CRID data may be more readily available and open for discussion.

As for broadcasters still not broadcasting CRID data, I will ask my contact for more information about the real expectation of this data being broadcast by *ALL* broadcasters across Australia. That said, I still think that adding CRID functionality to Enigma2 would be helpful for anyone who *can* receive the data. For those who can't there is noting gained *and* nothing lost.

Regards,
Ian.

[prl]

...
I am probably missing the point of your post regarding authorities of the CRIDs but my immediate expectation is that the authorities don't really matter for a single broadcaster. I suspect that all the ABC LCNs, as per the example, will use compatible data.

One of the issues I have is precisely about having to "suspect" things about the scope of the actual naming authorites rather than knowing. It's hard to make a reliable system based on data whose rules are not fully known.

The fact that the data is not 'clear' text is annoying but not a show stopper. ...

The limitation of authorities and the obfuscation of the CridSeries and CridEpisode data will mean that we can't match shows across broadcasters. This is not optimal but should still result in more accurate AutoTimer recordings from any single broadcaster.

For the use of CRIDs in AutoTimer, it 's not just not a show stopper, but IMO entirely irrelevant. I would have much preferred it if there was a single naming authority for DTV in Australia, freetv.au, or dtv.au or whatever, so that there was a one-to-one mapping of series and episodes to the sets of entities that they represent, but I doubt that that will ever happen. It would have been nice, too, to be able to extract usable series/episode numbers from the CRIDs, but it just can't be done across all broadcasters (or at least, that was the situation a year ago).

I agree with your concern that I am assuming that the CRID data will be consistent across a broadcaster. If the different regions or affiliates change the data will it *really* matter for a user and their AutoTimer recordings. As long as the data from the broadcaster they are using is internally consistent then all should be fine.

That's part of the issue with the CRID data that I was alluding to in my post. Unless the authority naming scheme has been drastically revised in the past year, then there is simply no straight-forward way to tell whether two CRIDs with the same data part are in fact CRIDs from the same broadcaster without a whole heap of market-dependent and rather fragile special-casing. Even if the naming is consistent across a broadcaster, or evrn a broadcaster in a single broadcast area, there's no way to simply and reliably tell that two CRIDs are from the same broadcaster (with the naming scheme in place a year ago).

Another issue with the actual CRID authority naming scope is, if, for example, I move to another city. It would be nice for all the CRID-based AutoTimers to "just work", in the way that most of the text-matching based ones will. Similarly for folk who take their Beyonwiz "on the road."

When you get time to go back to the CRID exploration I will ask my contact if more information about the authorities and the reason for obfuscation of the CRID data. I feel that if I wait until you are ready to return the CRID effort there may be a better opportunity to get more/better information. That is, the new OP 61 may be out and CRID data may be more readily available and open for discussion.

I wish it wasn't so, but I have little expectation from their past efforts that the networks will actually get this right, new OP 61 or not. I suspect that they actively do not want CRIDs to be compatible across networks (and there are some good technical reasons why, as well as the obvious "we'll make it hard for you to watch this anywhere but with us").

As for broadcasters still not broadcasting CRID data, I will ask my contact for more information about the real expectation of this data being broadcast by *ALL* broadcasters across Australia. That said, I still think that adding CRID functionality to Enigma2 would be helpful for anyone who *can* receive the data. For those who can't there is noting gained *and* nothing lost.
...

That expectation has been there for nearly 3 years, possibly longer (I don't know what was in issues 1 & 2 of OP 61), and a year ago, they weren't all broadcasting full CRID data on "all stations," despite the assurances of your Seven contact. Certainly WIN Canberra and Prime Canberra (a Seven affiliate, even) weren't.

I disagree about the "all gain and no loss" in having CRIDs used in AutoTimer if there isn't at least CRID data for all programs in all normal broadcast channels in all broadcast areas where people use Beyonwizes. There is definitely a downside to something that works inconsistently. Remember that it isn't (or at least wasn't, a year ago) that CRIDs were either across all channels and all programs or completely absent. In Canberra, both WIN and Prime carried some CRID data, it was just very patchy.

Anyway, I've just merged my stashed CRID code changes into the current code (which didn't happen automatically, but wasn't too bad), and got it to compile.

I won't be able to test it until late next week, at best.

[prl]

Hi All,

Sorry to bump and old thread but I have some potentially good news on the CRID front...

A friend at Network Seven contacted me today to inform me of some changes coming to OP 61 (This is the old link that will soon be updated to Issue 4.)

The changes are being brought about due to the looming demise of Freeview EPG (MHEG) at the end of next week (Freeview Plus will be the only EPG service operating via HbbTV).

The interesting item to note is paragraph 6.5:

6.5 Carriage of CRIDs in EIT actual schedule It is required that Australian television broadcasters carry content reference information (CRIDs) for all content referenced in the broadcasters schedule (8 day) event information table. This information shall align with Section 4 of AS4599.1 - 2013. It is expected that Hbb TV compliant PVR manufacturers shall be able to utilise this information to manage scheduled recordings.

I believe that the changes to OP 61 should be a good sign that Prl's experimental work on CRIDs should resume / continue. Now that they will be required this should make for a far more useful AutoTimer system (assuming the spaghetti code of AutoTimer can be understood and support changes to use CRIDs.)
...

OP61 so far remains unchanged (and it's in fact OP72 that's the one that really defines how CRIDs are to be transmitted, and that's also so far unchanged). The state of CRIDs in Canberra is still as much of a mess as it was a year ago.

As for the three-year-old injunction that "It is required that Australian television broadcasters carry content reference information (CRIDs) for all content referenced in the broadcasters schedule (8 day) event information table" (which is as it is in the December 2014 version of OP61), that seems to only be true for a relatively restricted definition of "all": one that certainly doesn't include Canberra.

• Seven regional affiliate Prime Canberra transmits no CRID information at all.
• Out of ~1300 events in WIN's EPG, only 20 had CRID information.
• In the SCA EPG, all EPG entries have CRID information except for SBN and Aspire, which have no CRID information at all.
• The ABC has CRIDs for all TV services, but none for radio services.
• SBS has CRIDs for all services, TV and radio, but the radio CRIDs are in an illegal form that has no name authority (e.g. crid://b6911263-2e94-430a-b337-1769c338ed9c). There are no series CRIDs for SBS radio.
• If the part of the SBS radio CRID following the "crid://" is intended to be the data part of a CRID, it's longer (36 bytes) than the length allowed for the data part in OP 72 (29 bytes, including the leading "/").
• Prime Canberra transmits an incorrect (and probably non-permitted) value (0x233a) for the "private_data_specifier_id" specified in section 2.2.1 of OP 72 (the allowed values for Australia must lie in the range 0x3200-0x320f, with 0x3206-0x3209 "reserved for future use").
• SBS doesn't seem to transmit a "private_data_specifier_id" at all, or at least not in the SDT (where everyone else puts it) or in the NIT.
• The ABC seems to be transmitting a non-permitted "private_data_specifier_id", 0x3201, since section 2.1.1 says: "should
  also include a private_data_specifier_descriptor (tax [sic] 0x5F) with a private_data_specfier [sic] value of 0x000032000", but the Table 10 lists 0x00003201 as the value for ABC. As I've quite often found, there's ambiguity in the OP.

So I'm not sure how Prime and WIN think they they "carry content reference information (CRIDs) for all content referenced in the broadcasters schedule (8 day) event information table". The other data is also a bit deficient in a number of ways.

In Canberra, apart from Prime, for which there now seems to be no naming authority information transmitted at all, for the other Canberra broadcasters, there is still the bewildering plethora of what seem to me to be far too "local" naming authorities, where the scope of the authority is a single service in a region, which surely doesn't reflect reality:
canberra.nitv.sbs.au
canberra.sbshd.sbs.au
canberra.sbsone.sbs.au
canberra.sbsthree.sbs.au
canberra.sbstwo.sbs.au
canberra.sbstwohd.sbs.au
can.sr1.win.au
can.sr2.win.au
can.sr3.win.au
can.sr4.win.au
can.sr5.win.au
can.sr6.win.au
can.sr7.win.au
canberra.1.sca.au
canberra.2.sca.au
canberra.3.sca.au
canberra.4.sca.au
canberra.5.sca.au
canberra.6.sca.au
canberra.7.sca.au
canberra.8.sca.au
canberra2.abc.net.au
canberra20.abc.net.au
canberra21.abc.net.au
canberra22.abc.net.au
canberra23.abc.net.au
canberraabcnews24.abc.net.au

Note that canberra2.abc.net.au canberra20.abc.net.au are the separate naming authorities for ABC LCN 2 and ABC LCN 20, which are simulcasts. The SBS naming authority names refer to four channels that no longer exist: sbshd, sbsthree, sbstwo and sbstwohd.

[prl]

Since the state of CRIDs in Canberra hasn't changed in any useful way since I last mothballed the test CRID code, I intend to mothball it again, with just the changes needed for it to run in the current firmware.

[IanSav]

Hi Prl,

Since the state of CRIDs in Canberra hasn't changed in any useful way since I last mothballed the test CRID code, I intend to mothball it again, with just the changes needed for it to run in the current firmware.

Before you go ahead with the mothball can you make your code available for others to test / investigate.

I am going to copy your posts to my friend at Channel 7 to see what insights he may be able to offer. Hopefully, at least, he can get the 7 affiliates to fall into line.

Regards,
Ian.

[prl]

... Hopefully, at least, he can get the 7 affiliates to fall into line. ...

And WIN?

[prl]

... can you make your code available for others to test / investigate. ...

In what form would people like it: most of the changes are to the C++ code, and so the source changes are only useful to people who can build the enigma2 binary? Alternative, I can make the binary available (too).

[IanSav]

Hi Prl,

... Hopefully, at least, he can get the 7 affiliates to fall into line. ...

And WIN?

I will ask how far he can go in getting stations to comply, agree on formats etc.

As you said, the multiple authorities for a broadcaster seems to be a complete waste of effort. Surely the stations can save themselves lots of effort by condensing the authority lists down to fewer entries.

Regards,
Ian.

[IanSav]

Hi Prl,

... can you make your code available for others to test / investigate. ...

In what form would people like it: most of the changes are to the C++ code, and so the source changes are only useful to people who can build the enigma2 binary? Alternative, I can make the binary available (too).

I would need binaries to make any use of the code. I am happy to simply have skin messages to get a feel for the data that the Beyonwiz can see.

The changes may also be useful for the OpenViX people to start exploring their CRID opportunities.

Regards,
Ian.

[prl]

...
As you said, the multiple authorities for a broadcaster seems to be a complete waste of effort. Surely the stations can save themselves lots of effort by condensing the authority lists down to fewer entries.
...

I don't think that the multiple authority names per broadcaster represent any effort. I very strongly suspect that the set of authority names in a region actually represent a single authority, and probably a single authority with a scope much larger than even the region part of the name.

[prl]

... can you make your code available for others to test / investigate. ...

Can anyone who wants to test the code say just want it is that they want to test? That will determine how much of the debug prints I'll leave in.

[IanSav]

Hi Prl,

I don't mind what level of debug printing is left operating, though I suspect that very little debug printing would be better (less serial port clutter). I think that a single line of a fully qualified CRID URI on every program change would be helpful.

I intend to add skin elements to a number of EPG related screens to try and get a handle on what the CRID data looks like during regular use. What is present, what is missing, is the data relateable, etc.

Regards,
Ian.

[prl]

...
I don't mind what level of debug printing is left operating, though I suspect that very little debug printing would be better (less serial port clutter). I think that a single line of a fully qualified CRID URI on every program change would be helpful.

It's more difficult to pull data out of the EPG at the point where it's entered into the EPG on the Beyonwiz.

At the moment, I'm displaying it at the point where the data is extracted from the EPG for display in the UI. To get a full dump of the EPG CRID data, I set the visible timeline in the graphical EPG to 300 min, and then step through the graphical EPG using 6 Page Right. That puts a complete set of the EPG information into the log (with some duplication at the edges of the pages)'. I then use "grep" in conjunction with tools like "wc" (word count) to get estimates of the CRID coverage by broadcasters. If the Last Scanned bouquet is used for this process, the CRID data extracted also includes radio services.

I wouldn't recommend the patched version for normal use.

This is an example of the output:

Code: Select all

{558}<    98.428> [Event] 0212:0211:1010 15.12, 13:03 <ABCComedy/Kids> <Daniel Tiger's Neighbourhood> crids: episode<crid://canberra22.abc.net.au/zx8997a010s00> series<crid://canberra22.abc.net.au/zx8997a>

So: SID:TSID:ONID start_time <channel_name> <title> crids: list_of_crids

I intend to add skin elements to a number of EPG related screens to try and get a handle on what the CRID data looks like during regular use. What is present, what is missing, is the data relateable, etc.
...

IMO it's more productive to examine the logged data. It's especially useful where a broadcaster's CRID coverage is partial. However, I'll add the series and episode CRIDs to the "event in focus" data for the graphical EPG in at least the easy-skin-aus-hd skin. I've never used that capability as more than a curiosity.

There's little to say about a general form of the CRID data, at least in Canberra, because the form is chosen by each individual broadcaster. In some, the episode and series CRID data are relatable, and in some of those, there appears to be series/episode information used to construct the episode CRID, and in others both the episode and series data are arbitrary, unrelatable (in themselves) strings. And, of course, for SBS radio, they seem to me to simply be wrong.

# Arbitrary series id, relatable series & episode, possibly identifiable episode information
series<crid://canberra2.abc.net.au/zx6247a> episode<crid://canberra2.abc.net.au/zx6247a005s00>
series<crid://canberra.7.sca.au/19879> episode<crid://canberra.7.sca.au/19879d17_5>

# Arbitrary series id, relatable series & episode, no identifiable episode information
series<crid://can.sr7.win.au/367634> episode<crid://can.sr7.win.au/367634e8086893>

# Arbitrary, unrelatable series and episode
series<crid://canberra.sbstwohd.sbs.au/228703> episode<crid://canberra.sbstwohd.sbs.au/883507>

# Just wrong:
episode<crid://ecbda175-dd7e-48e5-9275-a5970850e102> (SBS radio, no series CRID)

The SBS radio form CRID is wrong in both not having an authority name, and by its data part being longer (36 chars) than allowed in OP72 (29 chars, including the '/').

There are no recommendation CRIDs in the Canberra EPG data.

I don't think that there is any potential to make use of relationships encoded within the CRIDs. Use of CRIDs for, for example, series recording would be by identifying an episode of a series wanted to series record in the EPG and then using the series CRID of that episode to identify future episodes to record.

Recommendation CRIDs might also be useful, if they were carried by the broadcaster and if they served some more useful purpose than mere crosspromotion. E.g. a recommendation of the broadcaster's current affairs shows in their news EPG entry might be useful, but a recommendation CRID for some arbitrary new show for the network that's currently being plugged as "news" might be less useful.

It may also be possible to retrofit IceTV series and show ids into the EPGs in CRID form, something like: crid://beyonwiz.icetv.com.au/series_id and crid://beyonwiz.icetv.com.au/series_id.episode_id. That would make AutoTimer series recording using the IceTV "CRID" information possible, though I'm not sure why you'd want to forego the control over recording repeats that normal IceTV-generated timers give you.

[IanSav]

Hi Prl,

Rather than try to relay the information in this thread to my friend at Seven I have contacted him and provided a link to this thread. We shall see if he chooses to reply directly or through me. Hopefully we can get some insight on the future for CRIDs in Australia from a broadcasting perspective.

Regards,
Ian.

[prl]

OK, thanks.

[IanSav]

Hi Prl,

My friend at Seven has reviewed this thread and will explore the issues raised.

Regards,
Ian.

[prl]

Thanks, Ian. I've just added a new dot point about the length of what looks like the data part of the malformed SBS Radio CRID to my summary post.

[IanSav]

Hi Prl,

Just to add some spice to this conversation, I rechecked the CRIDs broadcast in Melbourne. Seven has a single authority "crid://seven.net.au". Nine uses a CRID authority for each LCN but they are all the same "crid://nine.com.au". (Nine's series and episode CRIDs are in plain text - not obfuscated.) SBS has no CRID authority at all. ABC and Ten use a different authorities for each LCN, even if the LCN is a simulcast!

I wonder if it is worth reaching out to a contact at SBS about the issues with their CRIDs. They did eventually address the error in the broadcast flags that I brought to their attention.

The situation in Melbourne appears better than what you see in Canberra but is still not "perfect". I wonder how the data looks in other capital cities and other locations around the country.

Regards,
Ian.

[prl]

...
Just to add some spice to this conversation, I rechecked the CRIDs broadcast in Melbourne. Seven has a single authority "crid://seven.net.au". Nine uses a CRID authority for each LCN but they are all the same "crid://nine.com.au". (Nine's series and episode CRIDs are in plain text - not obfuscated.) SBS has no CRID authority at all. ABC and Ten use a different authorities for each LCN, even if the LCN is a simulcast!

I'm not sure what you're looking at. Are you talking about the default authority names, and if you are, where are they being found (if they are being found)? It sounds like Seven is sending its default authority name in the first descriptor loop of the NIT or in the transport stream descriptor loop of the NIT, and the others apart from SBS are sending it in the service descriptor loop of the SDT. I don't currently have code for extracting the default authority name from anywhere other than the SDT.

In Canberra, SBS doesn't use the default authority name mechanism, and has a complete CRID, including the authority name, in each event entry in the EPG, other than in their strange radio CRIDs. Is that the same in Melbourne? If it is, that's a supported mechanism for specifying CRIDs.

In Canberra, all channels that use default authority names send them in the service descriptor loop of the SDT.

I've also noticed that in Canberra, SBS uses two different styles of episode CRID, a long one and a short one:

crid://canberra.sbstwohd.sbs.au/s61600520171216
crid://canberra.sbstwohd.sbs.au/890310

There doesn't seem to be any consistency in where they are used (those examples are both from VICELAND HD, whose authority name, somewhat confusingly, is canberra.sbstwohd.sbs.au). Neither seems to bear any relationship to its corresponding series CRID. I guess it's just another indication that there's little point in trying to extract any useful additional information from the CRID data parts.

That's also consistent with what ETSI TS 102 323 V1.3.1 (section 12.1.1 Note 2) says: "A CRID is simply an identifier."

I wonder if it is worth reaching out to a contact at SBS about the issues with their CRIDs. They did eventually address the error in the broadcast flags that I brought to their attention.

The only issue I've seen with SBS CRIDs in Canberra is that their radio CRIDs are, IMO, completely broken, and not well-formed CRIDs at all. The SBS TV CRIDs appear to be fine otherwise, just always transmitted in the EPG in their fully qualified form, not abbreviated, which is a bit wasteful, but just fine according to OP72 and ETSI TS 102 323.

The situation in Melbourne appears better than what you see in Canberra but is still not "perfect". I wonder how the data looks in other capital cities and other locations around the country.
...

My guess would be that in broadcast areas where the 5 national networks all broadcast, the situation would be similar to that in Melbourne and anywhere that regional affiliates are involved, expect the unexpected.

[IanSav]

Hi Prl,

Does this layout provide the information in which you are interested:

For Seven - Default Authority:

Code: Select all

  NIT PID = 0x0010 - Actual Network
  -> TransportStreamID = 0x0503 - 177500 kHz
    -> Descriptor = 0x73 - Default Authority
      -> DefaultAuthority = crid://seven.net.au

For Seven - Series CRID:

Code: Select all

  EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0534 - 7HD Melbourne
    -> EventID = 0xBF19 - House Of Wellness
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x32 -  User Private
          -> CRID = /seCxtdaO

For Seven - Episode CRID:

Code: Select all

  EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0534 - 7HD Melbourne
    -> EventID = 0xBF19 - House Of Wellness
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x31 -  User Private
          -> CRID = /seCxtdaOltaPmdeJ

I can do this for all the Melbourne broadcasters but it takes a little manual effort. I first wanted to check if this actually helps.

Regards,
Ian.

[prl]

Hi, Ian.

Yes, thanks. That is useful.

From what you've posted, the default authority name for Seven seems to be in the transport stream loop of the NIT. I coded the extraction of that that earlier today. I also coded the extraction of the default authority name from the first loop of the NIT.

The Seven CRID data is otherwise exactly what I'd expect it to look like when a default authority name is set.

Does SBS Melbourne have fully specified CRID data in its EPG entries?

[IanSav]

Hi Prl,

Here is the comparable data for the other Melbourne broadcasters.

For SBS - Default Authority: Not available!

For SBS - Series CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0315 - SBS ONE HD
    -> EventID = 0x8F5E - CGTN English News
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x32 -  User Private
          -> CRID = crid://melbourne.sbshd.sbs.au/219177

For SBS - Episode CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0315 - SBS ONE HD
    -> EventID = 0x8F5E - CGTN English News
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x31 -  User Private
          -> CRID = crid://melbourne.sbshd.sbs.au/S77039120171217

For Nine - Default Authority:

Code: Select all

SDT PID = 0x0011 - Actual TS
  -> ServiceID = 0x0431 - 9HD Melbourne
    -> Descriptor = 0x73 - Default Authority
      -> DefaultAuthority = crid://nine.com.au

For Nine - Series CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0431 - 9HD Melbourne
    -> EventID = 0x006E - Extra
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x32 -  User Private
          -> CRID = /EXTR

For Nine - Episode CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0431 - 9HD Melbourne
    -> EventID = 0x006E - Extra
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x31 -  User Private
          -> CRID = /EXTR_24_69

For Ten - Default Authority:

Code: Select all

SDT PID = 0x0011 - Actual TS
  -> ServiceID = 0x0634 - TEN HD
    -> Descriptor = 0x73 - Default Authority
      -> DefaultAuthority = crid://melbourne.ten.ten.au

For Ten - Series CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0634 - TEN HD
    -> EventID = 0x9C9D - Sammy And Bella's Kitchen Rescue
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x32 -  User Private
          -> CRID = /59837

For Ten - Episode CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0634 - TEN HD
    -> EventID = 0x9C9D - Sammy And Bella's Kitchen Rescue
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x31 -  User Private
          -> CRID = /59837E1502263

For ABC - Default Authority:

Code: Select all

SDT PID = 0x0011 - Actual TS
  -> ServiceID = 0x0235 - ABC HD
    -> Descriptor = 0x73 - Default Authority
      -> DefaultAuthority = CRID://melbourne20.abc.net.au

For ABC - Series CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0235 - ABC HD
    -> EventID = 0x91F2 - Classic Countdown
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x32 -  User Private
          -> CRID = /LE1601V

For ABC - Episode CRID:

Code: Select all

EIT PID = 0x0012 - Actual TS, Schedule
  -> ServiceID = 0x0235 - ABC HD
    -> EventID = 0x91F2 - Classic Countdown
      -> Descriptor = 0x76 - Content Identifier
        -> CRIDType = 0x31 -  User Private
          -> CRID = /LE1601V013S00

For C31 - Default Authority:

Code: Select all

SDT PID = 0x0011 - Actual TS
  -> ServiceID = 0x0E01 - C31
    -> Descriptor = 0x73 - Default Authority
      -> DefaultAuthority = crid://C31.community.com.au

For C31 - Series CRID: Not available!

For C31 - Episode CRID: Not available!

Is there any other information that I can extract and provide for you?

Regards,
Ian.

[prl]

Thanks. All of those were as I'd expect them to be.

[prl]

Here's a alpha patch of the that collects CRID information and inserts it in the EPG (broadcast EPG only, not IceTV).

It also includes patches to the two built-in skins that show the CRID information for the in-focus program in the Graphical EPG PIG, for the "now" program for the in-focus channel in the Channel Selection screen and for the in-focus recording in the media selection screen. Of course, CRIDs can only be shown for recordings if the recording has been made after CRID information has been collected into the EPG, and only for recordings where the program being recorded has CRIDs in the EPG.

The CRIDs are labelled Series and Episode in the EPG screen, and are unlabelled in the other two screens, but the series CRID is always the upper one and the episode always the lower.

The patch has been tested with firmware version 20171204 (beta).

This patch was built for the T4 firmware. The CRID functionality has been tested to work on the T2 and T3, but there are some small oddities in running the firmware on the wrong PVR. In particular, the front panel display on the T3 won't be updated when using this firmware.

This alpha firmware is experimental and is not intended for normal use.

To apply the patches, download the linked .ZIP file, and extract it. It will create a new directory/folder called crid-installer, which contains two files, installer.sh and uninstaller.sh.

Copy the two files somewhere convenient on a T series box (like /home/root), then log into the box using telnet or ssh, change directory to the place you put the installer.sh/uninstaller.sh files. If you put the files in /home/root you'll be in the right place as soon as you log in.

To install the patches run:

sh installer.sh

and restart the GUI (or reboot).

After installing, you need to rescan, otherwise the CRIDs won't have their correct form on most channels. Part of the scan process needed for getting correct CRID forms on Seven Melbourne are untested, and the scan could crash if there's a bug in that untested code. The conditions for testing it don't exist in Canberra broadcasts.

When the scan completes, I'l like to get some information to verify that it has correctly extracted the right CRID authority information.

The results of the following commands would be useful:
grep '\[eDVBScan\].*private' debug_log_file
grep '\[eDVBScan\].*da.au' debug_log_file
grep '\[eDVBScan\].*authority' debug_log_file

Once the scan completes, the EPG needs to be cleared and refilled. A simple way to clear it is to do MENU>IceTV>Enable IceTV, MENU>IceTV>Disable IceTV.

Then simply step through viewing a channel from each broadcaster, staying on each channel for long enough for the EPG for the broadcaster to fill, about a minute or so.

Then CRID data should be visible in the modified EPG, channel selection and media selection screens as noted above.

CRID data for programs is only printed to the debug log when event information is fetched from the EPG, e.g. by the EPG screen.

You can get a full dump to the log of all CRID information in the EPG by setting your current bouquet to Last Scanned and opening the Graphical EPG, and then repeatedly pressing 6 to skip forward until you display the last page of the EPG data. This way of harvesting CRID data will give full coverage, but CRID entries that cross a page edge will be logged more than once. Using the Last Scanned bouquet will log EPG CRID data from both radio services and TV services.

CRID information in the log looks like this:

Code: Select all

{2153}<   869.203> [Event] 0356:0350:3202 17.12, 13:50 <SBS VICELAND HD> <Morgan Spurlock: Inside Man> crids: episode<crid://canberra.sbstwohd.sbs.au/794087> series<crid://canberra.sbstwohd.sbs.au/221337>

The useful data part is: SID:TSID:ONID dd.mm, HH.MM <channel_name> <program_title> crids: crid_info

You can extract all the CRID data with:
grep '\[Event\]' debug_log_file

You can make it specific to a network by adding the ONID to the search, so for all Canberra SBS CRID data:
grep '\[Event\].*:3202 ' debug_log_file

To count the number of matches use the "-c" option to grep.

You can search for only entries that have CRID data by adding ".*crids:." to the end of the search pattern. You can search for all the entries that don't have CRID data by adding ".*crids:$" to the each pattern.

E.g. to count the total number of logged CRID items for Canberra SBS:
grep -c '\[Event\].*:3202 ' debug_log_file

and to count all the Canberra SBS entries that have no CRID data:
grep -c '\[Event\].*:3202 .*crids:$' debug_log_file

I'm interested in any broadcasters that have a significant number of EPG items with missing CRID information.

I'm also interested in any CRIDs that aren't in the correct crid://authority_name/crid_data form, e.g. the CRID data for Canberra SBS radio EPG entries: crid://2e146653-74bd-405a-8b79-b2e2311e6b1e, or where the authority_name is longer than 32 characters or the crid_data is longer than 28 characters (the part of the SBS Radio CRID after the '//' is too long to be valid for either, and in any case the whole thing has the wrong syntax).

To uninstall the patches, log in and go to the directory as you did to install, and run

sh uninstaller.sh

And restart the GUI.

After uninstalling, do another service scan to remove the additional data that the alpha firmware puts in /etc/enigma2/lamedb.

Make sure you uninstall before doing an upgrade. Don't run the installer if you've already installed & don't run the uninstaller if you've already uninstalled.

You can check whether the patches are installed by logging in and running this on the box:

find /usr/lib/enigma2 /usr/bin /usr/share/enigma2 -name \*.bak

It should print nothing if the patch isn't installed, and it should print

/usr/lib/enigma2/python/Components/Converter/EventName.pyo.bak
/usr/lib/enigma2/python/enigma.pyo.bak
/usr/bin/enigma2.bak
/usr/share/enigma2/Full-Metal-Wizard/skin_videos.xml.bak
/usr/share/enigma2/Full-Metal-Wizard/skin_epg.xml.bak
/usr/share/enigma2/Full-Metal-Wizard/skin.xml.bak
/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml.bak
/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml.bak
/usr/share/enigma2/easy-skin-aus-hd/skin.xml.bak

if the patch is installed.

Comments welcome!

[IanSav]

Hi Prl,

Unfortunately my T4 is dead so I can't try the code at this time.

To further your investigation and development I have uploaded the analysis of all the UK TV samples I previously provided for you. There is a series of .XML files containing the analysed data in https://bitbucket.org/IanSav/easy-ui-4/downloads/. (UPDATE: Files now removed to free up space.) Please let me know when you have downloaded the samples so I can free up the space.

If you can support the CRID data in these samples then the OpenViX team would love to have your code so that it can be integrated into an OpenViX test.

NOTE: Files removed from repository to free up space. Data is still available if required.

Regards,
Ian.

[prl]

... Unfortunately my T4 is dead so I can't try the code at this time. ...

Is there a particular reason why you can't try the T4 code on your T3? I have tested it on a T3, and apart from it not writing to the front panel display, it works well enough for me to to all the testing that I think would be useful.

I wouldn't recommend using this code on an ongoing basis anyway, on a T4 or other device, because it's test code that doesn't add any useful user function.

To further your investigation and development I have uploaded the analysis of all the UK TV samples I previously provided for you. There is a series of .XML files containing the analysed data in https://bitbucket.org/IanSav/easy-ui-4/downloads/. Please let me know when you have downloaded the samples so I can free up the space.

I've downloaded the files. Thanks for doing the extraction of the SI information for me.

If you can support the CRID data in these samples then the OpenViX team would love to have your code so that it can be integrated into an OpenViX test.
...

There's no way for me to test my CRID code on UK DVB data, so any changes I make to OpenViX would only be tested on Australian broadcasts. I haven't looked at the samples you provided yet, so I don't know yet whether there's anything more I'd need to do to allow the code to work on UK DVB.

[IanSav]

Hi Prl,

Is there a particular reason why you can't try the T4 code on your T3? I have tested it on a T3, and apart from it not writing to the front panel display, it works well enough for me to to all the testing that I think would be useful.

I wouldn't recommend using this code on an ongoing basis anyway, on a T4 or other device, because it's test code that doesn't add any useful user function.

While my T4 is dead, my T3 is very sick. I am trying to contact Warkus to book in the repair but I haven't been able to contact him lately. I feel it would be unwise to try out the new code on a box that is known to crash / freeze many times a day.

There's no way for me to test my CRID code on UK DVB data, so any changes I make to OpenViX would only be tested on Australian broadcasts. I haven't looked at the samples you provided yet, so I don't know yet whether there's anything more I'd need to do to allow the code to work on UK DVB.

Some members of the OpenViX team would like the code so they can build a test branch of OpenViX and explore the possibilities that accessing the CRID data may provide. If you can add the code that you think is required the OpenViX team can test it out for you.

Regards,
Ian.

[IanSav]

Hi Prl,

I have alerted some of the OpenViX developers to this thread.

One developer has a box that should be compatible with the Beyonwiz. He may be able to test the code you are generating directly. Alternatively, access to the source code may also be helpful.

Regards,
Ian.

[prl]

Hi, Ian.

I've been looking at the SI information you made available. So far there don't seem to be any big differences between the relevant bits of the UK SI data and the Australian data, but the UK, like the Melbourne broadcasters, does send the default authority names in both the SDT table and in the transport stream loop of the NIT.

That means that if someone in Melbourne can try out my test version, I can test all of what I've found so far in the UK data.

[prl]

Hi Prl,

Is there a particular reason why you can't try the T4 code on your T3? I have tested it on a T3, and apart from it not writing to the front panel display, it works well enough for me to to all the testing that I think would be useful.

I wouldn't recommend using this code on an ongoing basis anyway, on a T4 or other device, because it's test code that doesn't add any useful user function.

While my T4 is dead, my T3 is very sick. I am trying to contact Warkus to book in the repair but I haven't been able to contact him lately. I feel it would be unwise to try out the new code on a box that is known to crash / freeze many times a day.
...

Ah, OK. I thought your reluctance was just because it's the wrong software for the T3. Perhaps someone else in Melbourne could give it a go. I'd also be interested in seeing some results from other broadcast areas where network affiliates broadcast the commercial networks, or at least some of them.

[IanSav]

Hi Prl,

I have a friend with a T4 that is fully up to date (beta wise). I will see if he can help me out to run some tests.

Regards,
Ian.

[Grumpy_Geoff]

...
I'd also be interested in seeing some results from other broadcast areas where network affiliates broadcast the commercial networks, or at least some of them.

No affiliates here in Perth, but I can give your CRID alpha patch a go if you'd like the data. I can rescan the T4 and disable Ice to get the FTA EPG to gather the data.


Cheers
Geoff

[prl]

Thanks to both of you.

[Grumpy_Geoff]

Perth results attached. It appears Seven Network (ONID 1013) isn't returning much. Nor West TV (ONID 323E).


Cheers
Geoff

[prl]

Thanks, Geoff. I'll have a look tomorrow.

[prl]

Hi, Ian.

There's an odd inconsistency between how the tool you've used to display the SI info from the UK DVB broadcasts lays out service_description_section - actual_transport_stream (table id 0x42) and service_description_section - other_transport_stream (table id 0x46).

The actual transport stream SDT (services in the current transport stream) is displayed as:

Code: Select all

      <SDT PID="0x0011" Name="Actual TS">
        <OriginalNetworkID HValue="0x233A"/>
        <TransportStreamID HValue="0x1044"/>
        <ServiceID HValue="0x1044" Name="BBC ONE Lon">
          <RunningStatus Value="4" Name="running"/>
          <EITScheduleFlag Value="1"/>
          <EITPresentFollowingFlag Value="1"/>
          <FreeCAMode Value="0" Name="not encrypted"/>
          <Descriptor HValue="0x48" Name="Service">
            <DescriptorLength Value="14"/>
            <ServiceType Value="1" Name="Digital Television"/>
            <ServiceName String="BBC ONE Lon"/>
          </Descriptor>
          <Descriptor HValue="0x73" Name="Default Authority">
            <DescriptorLength Value="12"/>
            <DefaultAuthority String="fp.bbc.co.uk"/>
          </Descriptor>
        </ServiceID>
        ... more service information
    </SDT>

and the other transport stream SDT (services in other transport streams) is displayed as:

Code: Select all

      <SDT PID="0x0011" Name="Other TS">
        <OriginalNetworkID HValue="0x233A">
          <TransportStreamID HValue="0x104B">
            <ServiceID HValue="0x104B" Name="BBC ONE Oxford">
              <RunningStatus Value="4" Name="running"/>
              <EITScheduleFlag Value="1"/>
              <EITPresentFollowingFlag Value="1"/>
              <FreeCAMode Value="0" Name="not encrypted"/>
              <Descriptor HValue="0x48" Name="Service">
                <DescriptorLength Value="17"/>
                <ServiceType Value="1" Name="Digital Television"/>
                <ServiceName String="BBC ONE Oxford"/>
              </Descriptor>
              <Descriptor HValue="0x73" Name="Default Authority">
                <DescriptorLength Value="12"/>
                <DefaultAuthority String="fp.bbc.co.uk"/>
              </Descriptor>
            </ServiceID>
        ... more service information
          </TransportStreamID>
        </OriginalNetworkID>
      </SDT>

There doesn't seem to be any basis in the DVB standard for this difference, since both use the same syntax (described in 5.2.3 Service Description Table (SDT) of ETSI EN 300 468 V1.14.1 (2014-05)), and the form used for the actual transport stream seems to better reflect what's in the standard. I don't think this difference reflects real differences in the DVB SI data between the two tables, but it's a bit confusing.

Any ideas?

[prl]

Perth results attached. It appears Seven Network (ONID 1013) isn't returning much. Nor West TV (ONID 323E).
...

The Seven CRIDs are particularly bizarre: events all contain CRID entries, but the CRID data in most of them is empty!

West TV, on the other hand, just doesn't seem to bother at all.

[Grumpy_Geoff]

The Seven CRIDs are particularly bizarre: events all contain CRID entries, but the CRID data in most of them is empty!

West TV, on the other hand, just doesn't seem to bother at all.

The odd Seven Network event has them for a repeat entry:

Code: Select all

[Event] 0560:0506:1013 18.12, 12:01 <7 Perth> <My Christmas Love> crids: episode<>
[Event] 0560:0506:1013 18.12, 12:01 <7 Perth> <My Christmas Love> crids: episode<crid://seven.net.au/tufsv3anltapmdej>
[Event] 0560:0506:1013 18.12, 13:56 <7 Perth> <The Daily Edition> crids: episode<> series<>
[Event] 0560:0506:1013 18.12, 13:56 <7 Perth> <The Daily Edition> crids: episode<crid://seven.net.au/verfrdeiltapm3mp> series<crid://seven.net.au/verfrdei>

Seven's events with CRIDs -

Code: Select all

[Event] 0560:0506:1013 18.12, 12:01 <7 Perth> <My Christmas Love> crids: episode<crid://seven.net.au/tufsv3anltapmdej>
[Event] 0560:0506:1013 18.12, 13:56 <7 Perth> <The Daily Edition> crids: episode<crid://seven.net.au/verfrdeiltapm3mp> series<crid://seven.net.au/verfrdei>
[Event] 0561:0506:1013 18.12, 13:56 <7 Perth> <The Daily Edition> crids: episode<crid://seven.net.au/verfrdeiltapm3mp> series<crid://seven.net.au/verfrdei>
[Event] 0562:0506:1013 18.12, 14:00 <7TWO Perth> <Million Dollar Minute> crids: episode<crid://seven.net.au/tu1ettismdaontex> series<crid://seven.net.au/tu1ettjb>
[Event] 0564:0506:1013 18.12, 13:56 <7HD Perth> <The Daily Edition> crids: episode<crid://seven.net.au/verfrdeiltapm3mp> series<crid://seven.net.au/verfrdei>
[Event] 0565:0506:1013 18.12, 13:59 <7flix Perth> <Once Upon A Time In Wonderland> crids: episode<crid://seven.net.au/tlgdvnlpmdapn6xx> series<crid://seven.net.au/tlgdvpxx>