CRIDs

prl · Post by **prl** » Tue Dec 19, 2017 12:13

Another oddity of the Perth CRID data that Geoff posted is that Seven uses a different, but still apparently non-sanctioned, Private Data Specifier 0x00000029, which is different from the non-sanctioned Private Data Specifier that Prime Canberra uses, 0x0000233a.

According to OP 72, the Private Data Specifier to flag that CRIDs are available in Australian DVB broadcasts should be 0x00003200, or at least in the range 0x00003200-0x0000320f.

Perhaps the use of 0x0029 and 0x233a by Seven Perth and Seven's Canberra affiliate means Hic sunt dracones, instead.

Added: According to DVB Services' list of allocated Private Data Specifiers, 0x00000029 is allocated to NorDig (I'd guess, this NorDig) and 0x0000233a is allocated to the Independent Television Commission (probably this Independent Television Commission).

IanSav · Post by **IanSav** » Tue Dec 19, 2017 14:37

Hi Prl,

prl wrote: ↑
Tue Dec 19, 2017 12:05

IanSav wrote: ↑
Mon Dec 18, 2017 23:49
...
I am still trying to understand CRIDs. I can't explain the differences. I will see if someone at OpenViX can enlighten us.
...
If the differences you're referring to are the ones I asked about in the UK DVB SI data you made available, I don't think the differences have much to do with CRIDs at all. More to do with the fact that the software that created the .xml files has two different ways of describing two tables (SDT Actual TS and SDT Other TS) that should have exactly the same syntax in the DVB stream.

I corrected the CRID mis-speak before my question to OpenViX.

Abu suggested that the difference may be due to his inability to access some of the channels in the lists.

Regards,
Ian.

prl · Post by **prl** » Tue Dec 19, 2017 21:23

IanSav wrote: ↑
Tue Dec 19, 2017 14:37
Hi Prl,
prl wrote: ↑
Tue Dec 19, 2017 12:05

IanSav wrote: ↑
Mon Dec 18, 2017 23:49
...
I am still trying to understand CRIDs. I can't explain the differences. I will see if someone at OpenViX can enlighten us.
...
If the differences you're referring to are the ones I asked about in the UK DVB SI data you made available, I don't think the differences have much to do with CRIDs at all. More to do with the fact that the software that created the .xml files has two different ways of describing two tables (SDT Actual TS and SDT Other TS) that should have exactly the same syntax in the DVB stream.
I corrected the CRID mis-speak before my question to OpenViX.

Abu suggested that the difference may be due to his inability to access some of the channels in the lists.
...

No, I don't think it has anything to do with that. It's just that the software that's producing the XML is using different XML structures for two SDT tables in the same MUX stream when the two tables should have identical syntax in the DVB stream.

IanSav · Post by **IanSav** » Tue Dec 19, 2017 21:48

Hi Prl,

Sorry, I misunderstood what you were trying to highlight.

I used TransEdit version 4.2.1.1 from DVBViewer to analyse the files. I am using the latest version available. I can rerun the analysis on an older version of TransEdit if you are interested in comparing the results.

Regards,
Ian.

IanSav · Post by **IanSav** » Wed Dec 20, 2017 10:04

Hi,

A few OpenViX developers may try to come online here to discuss and help with CRID development and testing.

Is it possible to relax/remove the geo-blocking for the UK, Spain and Germany for starters?

Regards,
Ian.

prl · Post by **prl** » Wed Dec 20, 2017 16:13

IanSav wrote: ↑
Tue Dec 19, 2017 21:48
...
Sorry, I misunderstood what you were trying to highlight.

I used TransEdit version 4.2.1.1 from DVBViewer to analyse the files. I am using the latest version available. I can rerun the analysis on an older version of TransEdit if you are interested in comparing the results.
...

The issue is that when TransEdit tries to represent something like:

Code: Select all

Syntax                                  Number of bits
some_section(){
    some_field                          8
    list_len                            8
    for (i = 0; i < list_len; i++){
        other_field                     8
    }
}

Sometimes it's done as:

Code: Select all

<SomeSection>
    <SomeField HValue="val1"/>
    <OtherField HValue="val2"/>
    <OtherField HValue="val3"/>
    <OtherField HValue="val4"/>
    ...
</SomeSection>

and sometimes it's done as:

Code: Select all

<SomeSection>
    <SomeField HValue="val1">
        <OtherField HValue="val2"/>
        <OtherField HValue="val3"/>
        <OtherField HValue="val4"/>
        ...
    </SomeField>
</SomeSection>

In the particular case I posted about, Actual TS is converted to XML in the first style and Other TS is converted to XML in the second style, even though they are both service_description_section instances and described by the same syntax. That's not the only place where there's this confusion about whether, when there are two fields in a section, the second one is a child or sibling of the first in the XML.

My inclination is that the sibling representation is a more natural representation of the DVB syntax.

Other than this oddity, there doesn't seem to be any reason why my CRID code shouldn't work on the UK satellite data that Abu provided and you transcribed into XML using TransEdit.

prl · Post by **prl** » Wed Dec 20, 2017 16:14

IanSav wrote: ↑
Wed Dec 20, 2017 10:04
...
A few OpenViX developers may try to come online here to discuss and help with CRID development and testing.

That would be great.

IanSav wrote: ↑
Wed Dec 20, 2017 10:04
Is it possible to relax/remove the geo-blocking for the UK, Spain and Germany for starters?
...

I'm not able to do that.

prl · Post by **prl** » Wed Dec 20, 2017 16:27

I'm currently tidying up my code a bit as a precursor to trying to make the same modifications to OpenViX.

IanSav · Post by **IanSav** » Wed Dec 20, 2017 22:09

Hi Prl,

prl wrote: ↑
Wed Dec 20, 2017 16:14

IanSav wrote: ↑
Wed Dec 20, 2017 10:04
...
A few OpenViX developers may try to come online here to discuss and help with CRID development and testing.
That would be great.

Let's see if the interest expressed becomes real testing an assistance.

prl wrote: ↑
Wed Dec 20, 2017 16:14

IanSav wrote: ↑
Wed Dec 20, 2017 10:04
Is it possible to relax/remove the geo-blocking for the UK, Spain and Germany for starters?
...
I'm not able to do that.

I had a chat to Gully about this and we agreed that we will deal with issues as they arise. If no-one offers to help then there is nothing to be done.

Regards,
Ian.

IanSav · Post by **IanSav** » Wed Dec 20, 2017 22:10

Hi Prl,

prl wrote: ↑
Wed Dec 20, 2017 16:27
I'm currently tidying up my code a bit as a precursor to trying to make the same modifications to OpenViX.

This would certainly help get more testers.

Regards,
Ian.

prl · Post by **prl** » Thu Dec 21, 2017 09:48

IanSav wrote: ↑
Wed Dec 20, 2017 22:10
Hi Prl,
prl wrote: ↑
Wed Dec 20, 2017 16:27
I'm currently tidying up my code a bit as a precursor to trying to make the same modifications to OpenViX.
This would certainly help get more testers.
...

I'd have thought that otherwise there'd be very few potential testers.

IanSav · Post by **IanSav** » Thu Dec 21, 2017 10:23

Hi Prl,

I have my first PM to get a developer/tester in Ireland online here.

Regards,
Ian.

prl · Post by **prl** » Thu Dec 21, 2017 18:03

Great to hear!

I guess we won't be hearing from your Seven contact until the new year?

IanSav · Post by **IanSav** » Thu Dec 21, 2017 20:47

Hi Prl,

prl wrote: ↑
Thu Dec 21, 2017 18:03
Great to hear!

I guess we won't be hearing from your Seven contact until the new year?

I suspect not but he does appear to also read his E-Mail away from the office.

Regards,
Ian.

adoxa · Post by **adoxa** » Mon Aug 06, 2018 15:53

Here's my observations for the CRIDs in the Rockhampton/Central Queensland region (Nine being SCA and WIN being TEN).

ABC
The series CRID is a prefix of the episode CRID.
The episode CRID can be used as the iview URL (https://iview.abc.net.au/programs/episode/CRID).
The series CRID is generally different for each season of a show.
The episode CRID could be used to get a possible episode number.

SBS
The series CRID is generally different for each season of a show.
Episode CRIDs generally increment with each episode, but don't directly relate to episode number.
Episode CRIDs for news programs embed the date.

Seven
The default authority is not apparently broadcast (according to the Freeview guide it's crid://seven.net.au, for all channels).
The series CRID is usually a prefix of the episode CRID, but there are exceptions.
The series CRID is generally different for each season of a show.
The CRIDs appear to be encoded, so it may be possible to extract more information if the encoding could be reversed.

Nine
The series CRID is a prefix of the episode CRID.
The series CRID is different for each channel.
The episode CRID includes season (or year) and episode number.

WIN
No CRID information is apparently broadcast (apart from Sky News and shopping).
According to the Freeview guide the CRIDs are mostly a series of hex digits, and not only is the series CRID different for each channel, so is the episode CRID, so there's not much useful, anyway.

Conclusions
Series CRID is useless to identify a show.
Nine's episode CRID can be used to extract season number or year and episode number.
ABC's episode CRID can be used to extract an episode number.
SBS's episode CRID could be used as a relative episode number.

prl · Post by **prl** » Mon Aug 06, 2018 18:41

Claytons CRIDs. I'm beginning to agree with peteru's WOFTAM comment.

prl · Post by **prl** » Mon Aug 06, 2018 18:44

BTW, if you see the CRID data part for a CRID in the logs, but not in the EPG or media selection list displays, that's a skin bug, not a code bug (and it took me far too long to work it out

). The fix is to add noWrap="1" to the relevant widgets in the skin.

IanSav · Post by **IanSav** » Mon Aug 06, 2018 19:41

Hi Prl,

prl wrote: ↑
Mon Aug 06, 2018 18:41
Claytons CRIDs. I'm beginning to agree with peteru's WOFTAM comment.

My aim was to get the code into Enigma2 so that the data can be seen and evaluated. I then wanted to lobby the broadcasters to make the data meaningful and correct so as to comply with / improve upon the operating standards.

There is no point trying to get the data fixed / improved if it can't be seen or used.

Regards,
Ian.

IanSav · Post by **IanSav** » Mon Aug 06, 2018 19:46

Hi Prl,

While I wait for your next build of the code for the T3 can I please have a pointer to what needs to be changed in a skin to see the CRID data.

Regards,
Ian.

prl · Post by **prl** » Mon Aug 06, 2018 19:52

Aren't the modified skin files in the ZIP that adoxa posted? If not have a look at the Crid-dev branch in my fork of easy-skin-aus-hd.

IanSav · Post by **IanSav** » Mon Aug 06, 2018 20:14

Hi Prl,

Sorry, I have been side tracked working on the new VirtualKeyBoard for OpenPli and OpenViX.

I will check out the resources you suggest. (There are no skins in the ZIP file.)

Regards,
Ian.

adoxa · Post by **adoxa** » Tue Aug 07, 2018 00:09

The Freeview guide has a program_id that does look to uniquely identify shows, even across networks. It also includes episode titles. It'd be nice if the broadcast EPG could have this info. But I guess that would be useful, which is anathema to the networks...

prl · Post by **prl** » Tue Aug 07, 2018 10:52

IanSav wrote: ↑
Mon Aug 06, 2018 20:14
Sorry, I have been side tracked working on the new VirtualKeyBoard for OpenPli and OpenViX.

I will check out the resources you suggest. (There are no skins in the ZIP file.)

Here are the CRID diffs in the easy-skin-aus-hd to display Series and Episode CRIDs in the GraphicalEPGPIG and movie slecetcuin screens. They include the noWrap fixes mentioned above.

Code: Select all

diff --git a/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml b/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml
index 50b980b..aa61d08 100644
--- a/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml
+++ b/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml
@@ -200,7 +200,15 @@
 		<widget position="25,105" size="845,125" source="Event" render="Label" zPosition="1" font="Regular;22" foregroundColor="grey" backgroundColor="background" transparent="1" valign="top">
 			<convert type="EventName">FullDescription</convert>
 		</widget>
-		<widget position="25,235" size="835,30" source="Title" render="Label" font="Regular;24" valign="bottom" transparent="1" foregroundColor="white" backgroundColor="background" zPosition="1" />
+		<eLabel text="Series" position="375,220" size="100,22" zPosition="1" font="Regular;20" foregroundColor="white" backgroundColor="background" transparent="1" valign="top"/>
+		<widget position="475,220" size="445,22" source="Event" render="Label" zPosition="1" font="Regular;20" foregroundColor="white" backgroundColor="background" transparent="1" valign="top" noWrap="1">
+			<convert type="EventName">CridSeries</convert>
+		</widget>
+		<eLabel text="Episode" position="375,243" size="100,22" zPosition="1" font="Regular;20" foregroundColor="white" backgroundColor="background" transparent="1" valign="top"/>
+		<widget position="475,243" size="445,22" source="Event" render="Label" zPosition="1" font="Regular;20" foregroundColor="white" backgroundColor="background" transparent="1" valign="top" noWrap="1">
+			<convert type="EventName">CridEpisode</convert>
+		</widget>
+		<widget position="25,220" size="350,45" source="Title" render="Label" font="Regular;24" valign="bottom" transparent="1" foregroundColor="white" backgroundColor="background" zPosition="1" noWrap="1" />
 		<widget position="880,240" size="40,18" source="Event" render="Pixmap" pixmap="epg/icetv_40x18.png" alphatest="blend" zPosition="1">
 			<convert type="ConfigEntryTest">config.plugins.icetv.enable_epg,True</convert>
 			<convert type="ConditionalShowHide" />
diff --git a/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml b/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml
index c21b713..e7418a0 100644
--- a/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml
+++ b/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml
@@ -219,6 +219,12 @@
 		</widget>
 		<widget name="movie_sort" position="1118,600" size="35,20" pixmaps="icons/msels_az_31-1.png,icons/msels_31-1_az.png,icons/msels_shuffle.png,icons/msels_za_1-31.png,icons/msels_1-31_za.png,icons/msels_faz.png,icons/msels_fza.png,icons/msels_rec.png,icons/msels_az_1-31.png,icons/msels_za_31-1.png,icons/msels_sl.png,icons/msels_ls.png,icons/msels_sb.png,icons/msels_bs.png" zPosition="10" transparent="1" alphatest="on"/>
 		<widget position="1158,600" size="35,20" name="movie_off" pixmaps="icons/ask.png,icons/movielist.png,icons/quit.png,icons/pause.png,icons/playlist.png,icons/playlistquit.png,icons/loop.png,icons/rep.png" zPosition="10" transparent="1" alphatest="on" />
+		<widget source="Service" render="Label" position="830,480" size="410,20" font="Regular;18" foregroundColor="grey" transparent="1" zPosition="1" noWrap="1">
+			<convert type="EventName">CridSeries</convert>
+		</widget>
+		<widget source="Service" render="Label" position="830,500" size="410,20" font="Regular;18" foregroundColor="grey" transparent="1" zPosition="1" noWrap="1">
+			<convert type="EventName">CridEpisode</convert>
+		</widget>
 		<widget source="Service" render="Label" position="830,549" size="410,20" font="Regular;18" foregroundColor="grey" transparent="1" zPosition="1">
 			<convert type="EventName">GenreList</convert>
 		</widget>

prl · Post by **prl** » Tue Aug 07, 2018 10:54

adoxa wrote: ↑
Fri Aug 03, 2018 18:32
keep a crid without a default authority (needed to see 7's crids).

Have you rescanned services since installing the CRID modifications? I think that some of the default authority data is only retrieved if you do a scan.

adoxa · Post by **adoxa** » Tue Aug 07, 2018 10:56

Not directly related to CRIDs, but I've wondered if the ABC/iview id prefix had any meaning. Here's what I came up with:

AC - arts & culture
CH - children
CK - children/kindergarten?
CO - comedy
DC - documentary/children? (Becoming Superhuman)
DO - documentary
DR - drama
FA - family?
IP - indigenous production? (8MMM)
LE - light entertainment
LR - ? (Dirty Laundry, Gruen XL, rage)
NC - news & current affairs
NE - news entertainment? (BtN)
NU - news update
RA - regional area? (Lateline and Heywire)
RN - religion
SC - science
TR - ? (rage)
ZW - ? (various)
ZX - ? (various)
ZY - ? (various)
ZZ - ? (The Black Adder)

adoxa · Post by **adoxa** » Tue Aug 07, 2018 10:58

prl wrote: ↑
Tue Aug 07, 2018 10:54

adoxa wrote: ↑
Fri Aug 03, 2018 18:32
keep a crid without a default authority (needed to see 7's crids).

Have you rescanned services since installing the CRID modifications? I think that some of the default authority data is only retrieved if you do a scan.

Yep, multiple times with extra debugging - I just don't think it's there (same with WIN's CRIDs).

adoxa · Post by **adoxa** » Tue Aug 07, 2018 11:03

prl wrote: ↑
Tue Aug 07, 2018 10:52
Here are the CRID diffs in the easy-skin-aus-hd to display Series and Episode CRIDs in the GraphicalEPGPIG and movie slecetcuin screens. They include the noWrap fixes mentioned above.

That's the diff for the noWrap fix, not for the CRIDs themselves (but it might be sufficient).

prl · Post by **prl** » Tue Aug 07, 2018 11:24

When people rescan, could they also look for the string "private" in the log after a scan. That will show the Private Data Specifiers (PDS) that the networks (if they are following the rules they set themselves) should be including to flag that they carry CRID data.

E.g for Canberra:

Code: Select all

root@beyonwizu4:/media/hdd/logs# grep private E*.log
{2061}<  2122.275> [eDVBScan]   NIT TS private 00003200
{2061}<  2157.720> [eDVBScan]   NIT TS private 00003201
{2061}<  2160.718> [eDVBScan]   NIT TS private 00003200
{2061}<  2169.365> [eDVBScan]   NIT TS private 0000233a
root@beyonwizu4:/media/hdd/logs#

They should all be in the range 0x3200-0x320f. SBS is not sending any PDS. Prime still sends 0x233a, which is just wrong, and out of the range reserved for Australian broadcasters. But then, they're not sending any CRID data at all in Canberra.

prl · Post by **prl** » Tue Aug 07, 2018 11:31

adoxa wrote: ↑
Tue Aug 07, 2018 11:03

prl wrote: ↑
Tue Aug 07, 2018 10:52
Here are the CRID diffs in the easy-skin-aus-hd to display Series and Episode CRIDs in the GraphicalEPGPIG and movie slecetcuin screens. They include the noWrap fixes mentioned above.
That's the diff for the noWrap fix, not for the CRIDs themselves (but it might be sufficient).

I don't know what happened there. I "just did the same as I did first time" and I got the right diffs. Anyway, I've fixed the post and it has the correct diffs now.

prl · Post by **prl** » Tue Aug 07, 2018 11:56

adoxa wrote: ↑
Tue Aug 07, 2018 10:58

prl wrote: ↑
Tue Aug 07, 2018 10:54

adoxa wrote: ↑
Fri Aug 03, 2018 18:32
keep a crid without a default authority (needed to see 7's crids).

Have you rescanned services since installing the CRID modifications? I think that some of the default authority data is only retrieved if you do a scan.

Yep, multiple times with extra debugging - I just don't think it's there (same with WIN's CRIDs).

The problem may be that the Freeview OP-72 Implementation of Content Reference IDs by Australian Television Broadcasters allows the default authority to be in one of 3 places (s2.2.4): First descriptor loop of NIT (scope: network), Transport stream descriptor loop of NIT (scope: transport stream) or Service descriptor loop of SDT (scope: service). The code only looks for it in the last one of those, because that's all I see in Canberra broadcasts.

ETSI TS 102 32 also allows the default authority to be sent in the Bouquet Association Table (BAT), but that doesn't seem to me allowed in OP-72.

I could try to code up extracting the default authority from the Network Information Table (NIT), but I don't think I have any way of testing it.

adoxa · Post by **adoxa** » Tue Aug 07, 2018 12:45

prl wrote: ↑
Tue Aug 07, 2018 11:56
I could try to code up extracting the default authority from the Network Information Table (NIT), but I don't think I have any way of testing it.

7 Rockhampton has "no valid NIT". ... Hm, with the two logs I still have, all channels have "no valid NIT", but I'm sure one log I looked at (with the extra debugging) had SBS with it, as I saw an unprocessed tag 0x73 (the default authority). 7 still didn't have it, but again, I thought I saw it one time, but it hasn't appeared since. It's probably not worth it. For my purposes, all I intend to do is extract info from ABC ("/ZW1234A001S00" -> "i1234 e1"), Nine ("/12345D1_2" -> "s1 e2" or "/12345D2018_123" -> "2018 e123") and SBS ("/900123" -> "n900123"). I might have a play with 7's and WIN is just useless.

IanSav · Post by **IanSav** » Tue Aug 07, 2018 13:12

Hi,

If you guys can identify any errors in the Network Seven broadcasts then I would like to know. I think my contact at Seven could help to get any issues resolved. (Adoxa: I did ask how to decode the CRID obfuscation for Seven but my questions went unanswered. Now that we have code to access the CRID data I can always ask again.)

Regards,
Ian.

prl · Post by **prl** » Tue Aug 07, 2018 13:13

adoxa wrote: ↑
Tue Aug 07, 2018 12:45

prl wrote: ↑
Tue Aug 07, 2018 11:56
I could try to code up extracting the default authority from the Network Information Table (NIT), but I don't think I have any way of testing it.
7 Rockhampton has "no valid NIT". ... Hm, with the two logs I still have, all channels have "no valid NIT",

I see "no valid NIT" when I scan, too, but unless a valid NIT is eventually found, the services on that frequency won't be found.

adoxa wrote: ↑
Tue Aug 07, 2018 12:45
had SBS with it, as I saw an unprocessed tag 0x73 (the default authority).

That's a bit odd, because at least in Canberra, SBS transmits its CRIDs in full and doesn't need a default authority (and doesn't transmit one). When I scan, I only see logging of unprocessed tags 0x41 and 0x4a, respectively Service List Descriptor (in the NIT) and Linkage Descriptor (in the SDT).

adoxa wrote: ↑
Tue Aug 07, 2018 12:45
7 still didn't have it, but again, I thought I saw it one time, but it hasn't appeared since. It's probably not worth it.

It really is all a bit of a shambles.

adoxa wrote: ↑
Tue Aug 07, 2018 12:45
For my purposes, all I intend to do is extract info from ABC ("/ZW1234A001S00" -> "i1234 e1"), Nine ("/12345D1_2" -> "s1 e2" or "/12345D2018_123" -> "2018 e123") and SBS ("/900123" -> "n900123"). I might have a play with 7's and WIN is just useless.

What are you intending to do with the series/episode data you extract?

prl · Post by **prl** » Tue Aug 07, 2018 14:59

IanSav wrote: ↑
Tue Aug 07, 2018 13:12
If you guys can identify any errors in the Network Seven broadcasts then I would like to know. I think my contact at Seven could help to get any issues resolved. (Adoxa: I did ask how to decode the CRID obfuscation for Seven but my questions went unanswered. Now that we have code to access the CRID data I can always ask again.)

I don't receive Seven. I get 7Prime.

Its errors are:

Using a Private Data Specifier outside the range allocated to Australian Broadcasters. It sends 0x233a, when the allocated range is 0x3200-0x320f.
It sends no CRID data at all. No Default Authority Descriptor, and no Content Identifier Descriptors. THis is unchanged since Dec 2017.

The last time I recall you posting about questions you asked of your Seven content, you never posted any response.

Also, it's not "new" to have code to extract CRID data: I posted a patch to do that in Dec 2017.

It now seems that using "data:" in a post is forbidden by our Sucuri overlords.

IanSav · Post by **IanSav** » Tue Aug 07, 2018 15:07

Hi Prl,

prl wrote: ↑
Tue Aug 07, 2018 14:59
I don't receive Seven. I get 7Prime.

Are they an affiliate or wholly owned by the Seven Network?

prl wrote: ↑
Tue Aug 07, 2018 14:59
Also, it's not "new" to have code to extract CRID data: I posted a patch to do that in Dec 2017.

The code may not be new but interest and development is now progressing and code is becoming available to more people.

prl wrote: ↑
Tue Aug 07, 2018 14:59
It now seems that using "data:" in a post is forbidden by our Sucuri overlords.

This forum is getting more and more annoying as more aspects of it continue to break.

Regards,
Ian.

adoxa · Post by **adoxa** » Tue Aug 07, 2018 15:17

prl wrote: ↑
Tue Aug 07, 2018 13:13
What are you intending to do with the series/episode data you extract?

Add it to the bottom of my FMW Single EPG description and hopefully include it with the title of an EPG search.

IanSav wrote: ↑
Tue Aug 07, 2018 13:12
[...] I did ask how to decode the CRID obfuscation for Seven [...]

It looks pretty consistent, so I hope to be able to extract season and episode numbers.

Grumpy_Geoff · Post by **Grumpy_Geoff** » Tue Aug 07, 2018 15:25

prl wrote: ↑
Tue Aug 07, 2018 14:59
I don't receive Seven. I get 7Prime.

Its errors are:
Using a Private Data Specifier outside the range allocated to Australian Broadcasters. It sends 0x233a, when the allocated range is 0x3200-0x320f.

It sends no CRID data at all. No Default Authority Descriptor, and no Content Identifier Descriptors. THis is unchanged since Dec 2017.

When I did the 'CRID' scan in December, the Seven Network in Perth only had CRID episode data for 0.2537% of events.
You posted here that it was also using a wrong Private Data Specifier.

Perhaps Seven only cares about the Sydney and Melbourne broadcast markets?

prl · Post by **prl** » Tue Aug 07, 2018 15:39

IanSav wrote: ↑
Tue Aug 07, 2018 15:07

prl wrote: ↑
Tue Aug 07, 2018 14:59
I don't receive Seven. I get 7Prime.
Are they an affiliate or wholly owned by the Seven Network?

I have absolutely no idea. The name is owned by a public company, Prime Media Group Limited. Whoever owns Prime7, they apparently don't care about OP-72.

IanSav · Post by **IanSav** » Tue Aug 07, 2018 15:48

Hi Grumpy_Geoff,

Grumpy_Geoff wrote: ↑
Tue Aug 07, 2018 15:25
When I did the 'CRID' scan in December, the Seven Network in Perth only had CRID episode data for 0.2537% of events.
You posted here that it was also using a wrong Private Data Specifier.

Perhaps Seven only cares about the Sydney and Melbourne broadcast markets?

I don't believe that is the case. I do know that Seven in Perth is under the same technical umbrella as Seven Sydney and Melbourne.

Regards,
Ian.

IanSav · Post by **IanSav** » Tue Aug 07, 2018 15:50

Hi Prl,

prl wrote: ↑
Tue Aug 07, 2018 15:39
I have absolutely no idea. The name is owned by a public company, Prime Media Group Limited. Whoever owns Prime7, they apparently don't care about OP-72.

I think they may not be directly related. Still, when I get your code update I will start running the tests and get back to my contact at Seven and see what influence he has with affiliates.

Regards,
Ian.

prl · Post by **prl** » Tue Aug 07, 2018 16:10

Here's a alpha patch (one for each Beyonwiz model) of the that collects CRID information and inserts it in the EPG (broadcast EPG only, not IceTV).

It also includes patches to the easy-skin-aus-hd skin that show the CRID information for the in-focus program in the Graphical EPG PIG and for the in-focus recording in the media selection screen. Of course, CRIDs can only be shown for recordings if the recording has been made after CRID information has been collected into the EPG, and only for recordings where the program being recorded has CRIDs in the EPG.

The CRIDs are labelled Series and Episode in the EPG screen, and are unlabelled in the media selection screen, but the series CRID is always the upper one and the episode always the lower.

The patch has been tested with firmware version 20180729 (beta).

This alpha firmware is experimental and is not intended for normal use.

To apply the patches, download the appropriate linked .ZIP file, and extract it. It will create a new directory/folder called crid-installer.model_name, which contains two files, installer.sh and uninstaller.sh.

Copy the two files somewhere convenient on a T series box (like /home/root), then log into the box using telnet or ssh, change directory to the place you put the installer.sh/uninstaller.sh files. If you put the files in /home/root you'll be in the right place as soon as you log in.

To install the patches run:

sh installer.sh

and restart the GUI (or reboot).

After installing, you need to rescan, otherwise the CRIDs won't have their correct form on most channels. Part of the scan process needed for getting correct CRID forms on Seven Melbourne are untested, and the scan could crash if there's a bug in that untested code. The conditions for testing it don't exist in Canberra broadcasts.

When the scan completes, I'l like to get some information to verify that it has correctly extracted the right CRID authority information.

The results of the following commands would be useful:
grep '\[eDVBScan\].*private' debug_log_file
grep '\[eDVBScan\].*da.au' debug_log_file
grep '\[eDVBScan\].* da ' debug_log_file

Once the scan completes, the EPG needs to be cleared and refilled. A simple way to clear it is to do MENU>IceTV>Enable IceTV, MENU>IceTV>Disable IceTV.

Then simply step through viewing a channel from each broadcaster, staying on each channel for long enough for the EPG for the broadcaster to fill, about a minute or so.

Then CRID data should be visible in the modified EPG and media selection screens as noted above.

CRID data for programs is only printed to the debug log when event information is fetched from the EPG, e.g. by the EPG screen.

You can get a full dump to the log of all CRID information in the EPG by setting your current bouquet to Last Scanned and opening the Graphical EPG, and then repeatedly pressing 6 to skip forward until you display the last page of the EPG data. This way of harvesting CRID data will give full coverage, but CRID entries that cross a page edge will be logged more than once. Using the Last Scanned bouquet will log EPG CRID data from both radio services and TV services.

CRID information in the log looks like this:

Code: Select all

{2153{7390}< 19202.696> [Event] crid  3202:0350:0352  SBS VICELAND        11.08, 08:00  crid://canberra.sbstwo.sbs.au/214450               crid://canberra.sbstwo.sbs.au/S73401520180811      Romanian News

The useful data part is: crid ONID:TSID:SID <channel_name> dd.mm, HH:MM series_crid episode_crid <program_title>

You can extract all the CRID data with:
grep '\[Event\] crid' debug_log_file
NB: two spaces between crid and the ONID.

You can make it specific to a network by adding the ONID to the search, so for all Canberra SBS CRID data:
grep '\[Event\] crid 3202' debug_log_file

To count the number of matches use the "-c" option to grep.

E.g. to count the total number of logged CRID items for Canberra SBS:
grep -c '\[Event\] crid 3202 ' debug_log_file

To uninstall the patches, log in and go to the directory as you did to install, and run

sh uninstaller.sh

And restart the GUI.

After uninstalling, do another service scan to remove the additional data that the alpha firmware puts in /etc/enigma2/lamedb.

Make sure you uninstall before doing an upgrade. Don't run the installer if you've already installed & don't run the uninstaller if you've already uninstalled.

You can check whether the patches are installed by logging in and running this on the box:

find /usr/lib/enigma2 /usr/bin /usr/share/enigma2 -name \*.bak

It should print nothing if the patch isn't installed, and it should print

//usr/lib/enigma2/python/Components/Converter/EventName.pyo.bak
/usr/lib/enigma2/python/enigma.pyo.bak
/usr/bin/enigma2.bak
/usr/share/enigma2/easy-skin-aus-hd/skin_epg.xml.bak
/usr/share/enigma2/easy-skin-aus-hd/skin_videos.xml.bak

if the patch is installed.

Comments welcome!

prl · Post by **prl** » Tue Aug 07, 2018 16:12

IanSav wrote: ↑
Tue Aug 07, 2018 15:50
[W]hen I get your code update I will start running the tests and get back to my contact at Seven and see what influence he has with affiliates.

Alpha patches for all models are now available.

They have all the changes that Adoxa made in what he's made available, and also, because the CRID data is not lowercased, extra code to make sure that the test to see whether the crid:// needs to be prepended has been made case-insensitive.

Grumpy_Geoff · Post by **Grumpy_Geoff** » Tue Aug 07, 2018 16:14

IanSav wrote: ↑
Tue Aug 07, 2018 15:48
Hi Grumpy_Geoff,
Grumpy_Geoff wrote: ↑
Tue Aug 07, 2018 15:25
When I did the 'CRID' scan in December, the Seven Network in Perth only had CRID episode data for 0.2537% of events.
You posted here that it was also using a wrong Private Data Specifier.

Perhaps Seven only cares about the Sydney and Melbourne broadcast markets?
I don't believe that is the case. I do know that Seven in Perth is under the same technical umbrella as Seven Sydney and Melbourne.

Regards,
Ian.

The broadcast transmission must lose 99.7463% of its CRID episode data when it heads across the Nullarbor Plain then

Grumpy_Geoff · Post by **Grumpy_Geoff** » Tue Aug 07, 2018 16:20

prl wrote: ↑
Tue Aug 07, 2018 16:10
Here's a alpha patch (one for each Beyonwiz model) of the that collects CRID information and inserts it in the EPG (broadcast EPG only, not IceTV).

Is this any different to the one I ran last December? Do you need me to run it again for Perth data?

prl wrote: ↑
Tue Aug 07, 2018 16:10
The patch has been tested with firmware version 20180827 (beta).

You're ahead of yourself

prl · Post by **prl** » Tue Aug 07, 2018 16:30

Grumpy_Geoff wrote: ↑
Tue Aug 07, 2018 16:20

prl wrote: ↑
Tue Aug 07, 2018 16:10
Here's a alpha patch (one for each Beyonwiz model) of the that collects CRID information and inserts it in the EPG (broadcast EPG only, not IceTV).

Is this any different to the one I ran last December? Do you need me to run it again for Perth data?

Only in minor ways, and only to see whether anything has improved/got worse. I only got into this because Adoxa wanted CRID source code to play with.

Grumpy_Geoff wrote: ↑
Tue Aug 07, 2018 16:20
[
prl wrote: ↑
Tue Aug 07, 2018 16:10
The patch has been tested with firmware version 20180827 (beta).

You're ahead of yourself

You have to ask Gully to get that version

Fixed. I dhouldn't enter firmware versions from memory

adoxa · Post by **adoxa** » Tue Aug 07, 2018 16:33

prl wrote: ↑
Tue Aug 07, 2018 16:10
The useful data part is: SID:TSID:ONID <channel_name> dd.mm, HH:MM series_crid episode_crid <program_title>

You can make it specific to a network by adding the ONID to the search, so for all Canberra SBS CRID data:
grep '\[Event\].*:3202 ' debug_log_file

I swapped those around, so it's ONID:TSID:SID and fgrep '[Event] crid 3202' (note there's really two spaces after crid).

Gully · Post by **Gully** » Tue Aug 07, 2018 16:47

prl wrote: ↑
Tue Aug 07, 2018 16:30

Grumpy_Geoff wrote: ↑
Tue Aug 07, 2018 16:20
[
prl wrote: ↑
Tue Aug 07, 2018 16:10
The patch has been tested with firmware version 20180827 (beta).

You're ahead of yourself
You have to ask Gully to get that version

Sorry it is a closed beta for the next 20 days.

prl · Post by **prl** » Tue Aug 07, 2018 16:57

adoxa wrote: ↑
Tue Aug 07, 2018 16:33

prl wrote: ↑
Tue Aug 07, 2018 16:10
The useful data part is: SID:TSID:ONID <channel_name> dd.mm, HH:MM series_crid episode_crid <program_title>

You can make it specific to a network by adding the ONID to the search, so for all Canberra SBS CRID data:
grep '\[Event\].*:3202 ' debug_log_file
I swapped those around, so it's ONID:TSID:SID and fgrep '[Event] crid 3202' (note there's really two spaces after crid).

Fixed now. Thanks.

adoxa · Post by **adoxa** » Tue Aug 07, 2018 17:20

prl wrote: ↑
Tue Aug 07, 2018 16:10
grep '\[eDVBScan\].* au ' debug_log_file

Should that be da, not au?

Ah, here's Seven's default authority:

Code: Select all

{2083}<348076.912> [eDVBScan] descr<41>
{2083}<348076.912> [eDVBScan] descr<73>
{2083}<348076.912> [eDVBScan]   NIT TS private 00000029

and missing_authority works.

About the Forum

Join us