Next "official" release

[peteru]

The current beta has been around for a long, long time and is a significant improvement over the last "official" or production release. I had a chat to "management" and it was agreed that it is time to polish the current beta a little more and bless it for general release.

I have one more feature that I would like to investigate and add to the firmware before the current beta is promoted to a production release. I anticipate that if there are no complications I will be done in a couple of weeks. I hope that during this time the current picon reshuffle, due to the regional broadcaster restructures, will hopefully get to a stage where the production release will have a set of picons that don't need updating within weeks.

At this stage, I would like to put a freeze on any major changes. I'd rather avoid anything that could have negative impact on the release candidate testing process and the eventual promotion to a full production release. New features or enhancements should probably be saved until the next beta cycle, unless you have a very compelling reason for why they must be in the upcoming production release. On the other hand, bug fixes will be very welcome at this stage, especially if the changes are minimal (rather than major refactors), well documented and easy to follow. Issues in the BitBucket bug tracker and detailed commit messages always help. The usual practices apply, we don't want to break something just before pushing the release out the door.

If you have code or ideas that relate to the above scheduling, please feel free to discuss here so that we all have visibility.

[Gully]

Hi Peter

That sounds great, given the number of posts from people asking about the latest official release.

One question - with all the discussion recently about OverlayHD and how it has problems due to updates from OpenATV, etc that haven't made it into the Beyonwiz build as yet, as well as other improvements people have been posting about, are you planning to incorporate any of this?

I realise that might be more a matter of next step after this gets released but still interested.

[MrQuade]

I realise that might be more a matter of next step after this gets released but still interested.

I think you could run a post search on IanSav's name to read the whole gory history behind this.

The updates required are *extremely* non-trivial from what I understand.

[prl]

One question - with all the discussion recently about OverlayHD and how it has problems due to updates from OpenATV, etc that haven't made it into the Beyonwiz build as yet, as well as other improvements people have been posting about, are you planning to incorporate any of this?

Different Peter - but anyway - I've submitted the first part of the fix for this. I'm pretty sure it fits peteru's "minimal ..., well documented and easy to follow."

However, that just enables the fix that IanSav and I worked out for OverlayHD, and that fix will go into a version of OHD that won't (initially) be available for Beyonwiz. Getting that fixed OHD version will probably depend on my getting the new Euro distributions' VirtualKeyboard implementation (largely IanSav's work) into Beyonwiz firmware. I haven't looked at that closely yet, and I'm a bit worried that it might be something that has other changes needed to support it.

I realise that might be more a matter of next step after this gets released but still interested.

I'm not intending to try to merge in the new VirtualKeyboard before the official release is finalised unless peteru suggests that I do.

I will probably submit my changes to address the shutdown timer/record timer problem that Grumpy_Geoff reported in the next couplee of days, they also seem to pass that test. However, they also modify core function (i.e. recording) code, so I wouldn't be surprised if they weren't taken up by peteru before making an official release.

[Gully]

I think you could run a post search on IanSav's name to read the whole gory history behind this.

The updates required are *extremely* non-trivial from what I understand.

No need for the search. I'm well aware of the history if not the technical details and understand they were unlikely to be trivial so it was as much a question of where do they sit on the planning timeline as what aspects might be incorporated this time.

Different Peter - but anyway - I've submitted the first part of the fix for this. I'm pretty sure it fits peteru's "minimal ..., well documented and easy to follow."

Good to know. Not that I'm an OverlayHD user.

However, that just enables the fix that IanSav and I worked out for OverlayHD, and that fix will go into a version of OHD that won't (initially) be available for Beyonwiz. Getting that fixed OHD version will probably depend on my getting the new Euro distributions' VirtualKeyboard implementation (largely IanSav's work) into Beyonwiz firmware. I haven't looked at that closely yet, and I'm a bit worried that it might be something that has other changes needed to support it.

I'm not intending to try to merge in the new VirtualKeyboard before the official release is finalised unless peteru suggests that I do.

I will probably submit my changes to address the shutdown timer/record timer problem that Grumpy_Geoff reported in the next couple of days, they also seem to pass that test. However, they also modify core function (i.e. recording) code, so I wouldn't be surprised if they weren't taken up by peteru before making an official release.

Fair enough. As I said, good to know the plans ahead.

[peteru]

I merged the BER changes that prl submitted as they were clearly an obvious bug with a well documented commit that is low risk. If that change enables OverlayHD to work (or be updated to work) correctly, then great. If there are more bug fixes that have a similar characteristics (obvious coding errors, with simple, easy to understand fixes and low risk), I will also merge them.

Major code changes that add new features are unlikely to be merged before the next production release. For example, adding a whole new on screen keyboard implementation is definitely out of scope for the upcoming release candidates.

Once the next release is out and widely deployed, I'll deal with urgent bug fixes (hopefully the number of these will be zero ) before looking at merging new features.

Although I would like to start a new series of firmware, I don't think the upstream is in a state to go down that path yet. There is a major effort afoot to transition from python2.7 (which has been EOL for a while now) to python3, but unfortunately that effort is nowhere near good enough to make the switch. At this stage, the quality of the most recent upstream development version is only good enough for people who are willing to tinker with nightly builds. It is not of acceptable quality for a consumer device that needs to have every day reliability.

My main goal for the Beyonwiz firmware is to make it robust enough for everyday use, without having to baby sit the installation.

[MrQuade]

Although I would like to start a new series of firmware, I don't think the upstream is in a state to go down that path yet. There is a major effort afoot to transition from python2.7 (which has been EOL for a while now) to python3, but unfortunately that effort is nowhere near good enough to make the switch. At this stage, the quality of the most recent upstream development version is only good enough for people who are willing to tinker with nightly builds. It is not of acceptable quality for a consumer device that needs to have every day reliability.

Yea, I was trying to run the Python3 version of OpenATV on my second V2 for almost a year and the progress has been hilariously slow. I can't believe it is taking this long.

I keep thinking back to one of your old proposals to fork a separate version of firmware that was closer to upstream and was thinking it was a bad idea at the time since it would have essentially split the local community.

However, after having the V2 for a while and playing with both firmwares on different boxes, I keep thinking how great it would be to have the upstream firmware, but with just those few really nice quality of life improvements we have in ours (time skipping improvements, regional preferences and the like). (Things like getting rid of our incompatible and problematic main menu, which while I think it looks nice, sounds like it is quite hacky under the hood)

Though I know that is a dangerous line of thought, because where do you draw that line, how do you then maintain upstream compatibility long term, and do you want to start having disagreements with the upstream maintainers about the value of our improvements and merging into their projects....

[prl]

Things like getting rid of our incompatible and problematic main menu, which while I think it looks nice, sounds like it is quite hacky under the hood.

IanSav (and to a lesser extent, I) have cleaned up the main menu a bit from what it was. The code is still pretty hard to follow in places, though.

[prl]

I've submitted a fix to the problem that Grumpy_Geoff found of PowerTimers shutting town the PVR if they ran at exactly the same time that a recording started. It'll be a judgement call by peteru on whether that's too much of a change at the moment.

While I was digging around in the PowerTimer code, I noticed that, unlike RecordTimers, PowerTimer logs aren't also written to the debug log.

I've done a separate submit that adds debug logging of the PowerTimer user logs to the debug log. It also puts timestamps on thest debug log entries for both kinds of timer. This is a trivial fix.

[peteru]

Merged both and pushed out an online update so that it can get some testing.

[prl]

Thanks.

[prl]

20210731 crashes at startup if there are any timers in pm_timers.xml.

Working on it, should be a simple fix.


If you have any power timers, I suggest not updating! If you only have disabled power timers (as in the factory defaults), that should be OK (thanks to Grumpy_Geoff for confirming that disabled power timers won't cause the crash).

[adoxa]

Dammit, I should have said something earlier. Could you update the plugins, too, so my IMDb update can be included?

[prl]

20210731 crashes at startup if there are any timers in pm_timers.xml.

Working on it, should be a simple fix.

Fix submitted.

Sorry, everyone!

[Grumpy_Geoff]

20210731 crashes at startup if there are any timers in pm_timers.xml.

Working on it, should be a simple fix.


If you have any power timers (even disabled ones), I suggest not updating!

I've just a single disabled Power Timer (the default Auto Deep Standby one) on the one box that I've just updated to 20210731 without it crashing.

[prl]

20210731 crashes at startup if there are any timers in pm_timers.xml.

Working on it, should be a simple fix.


If you have any power timers (even disabled ones), I suggest not updating!

I've just a single disabled Power Timer (the default Auto Deep Standby one) on the one box that I've just updated to 20210731 without it crashing.

OK, thanks. That's at least a little better. PowerTimerEntry.activate() mustn't be being called for disabled timers (which isn't surprising, I guess). I'll amend my post.

[stevebow]

it was agreed that it is time to polish the current beta a little more and bless it for general release.

Thank you, very much looking forward to that!

[peteru]

Could you update the plugins, too, so my IMDb update can be included?

Done.

[peteru]

20210731 crashes at startup if there are any timers in pm_timers.xml.

Fix submitted.

Merged, building, will push overnight.

I will also upload USB images for a quicker recovery.

[adoxa]

Could you update the plugins, too, so my IMDb update can be included?

Done.

Thank you. Another thing I should have mentioned, could you update your git tag? This would make Enigma2's about information appear more up-to-date, too.

[prl]

The overnight update to 20210801 has fixed the power timer-induced crash at startup in the previous version.

[peteru]

could you update your git tag?

Which one?

There's many dozens of repos. Some are tagged rather well, some are not tagged at all and many are somewhere in between, with ad-hoc tag tactics employed. If you let me know which particular repo you are thinking of (or at least what you are looking at), I'll see what I can do.

[peteru]

The overnight update to 20210801 has fixed the power timer-induced crash at startup in the previous version.

Thank you for confirming the fix.

I've got some more updates cooking. I want to push several online only updates over the next few weeks to see how my changes to that part of the firmware are working. I want to make the online update functionality a little more robust, a bit more secure and slightly more flexible. Bit at a time.

[adoxa]

could you update your git tag?

Which one?

easy-ui-4 - Enigma2's version in the about shows as 20200822-153-g6bc301d.

[adoxa]

I want to make the online update functionality a little more robust, a bit more secure and slightly more flexible. Bit at a time.

I made a change a while ago to better reflect the package count.

[peteru]

I have split this topic to move the unrelated EPG discussion, which can now be found in the General Discussion section.

[peteru]

I think that the current beta is getting close to becoming a release candidate. The changes to feeds seem to work as well as can be expected. I've done some work on the network mount manager as it has been a major source of many pain points recently. I also fixed all crashing bugs that were reported with usable logs and were reproducible.

I had a quick look through the issues and found a few that may be worth revisiting before wrapping this up. Can I please get some help with updating / fixing the following issues?

668/link-deletion-deletes-target-not-link-in
607/crash-in-skip-forward-back-min-on-very
737/file-system-check-always-uses-fsckext4
729/skip-back-operation-skips-right-on-v2

If you decide to work on a fix, please assign the issue to yourself so that we don't end up duplicating effort.

Thanks.

[adoxa]

I don't know if this is just my test V2 (20210823), but I'm getting a package error with ntpdate whenever I install a plugin.

Code: Select all

{2293}<   598.223> [Ipkg] parseLine /bin/sh: ///var/lib/opkg/info/ntpdate.postinst: Permission denied
{2293}<   598.438> [Ipkg] parseLine Installing enigma2-plugin-extensions-Movie-Meta (0.1) on root.
{2293}<   598.439> [Ipkg] parseLine Configuring enigma2-plugin-extensions-Movie-Meta.
{2293}<   598.439> [Ipkg] parseLine Configuring ntpdate.
{2293}<   598.439> [Ipkg] parseLine Collected errors:
{2293}<   598.439> [Ipkg] parseLine  * pkg_run_script: package "ntpdate" postinst script returned status 126.
{2293}<   598.439> [Ipkg] parseLine  * opkg_configure: ntpdate.postinst returned 126.

[prl]

668/link-deletion-deletes-target-not-link-in
737/file-system-check-always-uses-fsckext4

If you decide to work on a fix, please assign the issue to yourself so that we don't end up duplicating effort.

I'm happy to work on these, but I can't see how to assign them to myself in the tasks. I see an Assignee field, but I can't find a way to change it.

[peteru]

668/link-deletion-deletes-target-not-link-in
737/file-system-check-always-uses-fsckext4

If you decide to work on a fix, please assign the issue to yourself so that we don't end up duplicating effort.

I'm happy to work on these, but I can't see how to assign them to myself in the tasks. I see an Assignee field, but I can't find a way to change it.

You Edit the issue, then scroll down and there should be an Assign to me link, just under the search field for Assignee.

However, I'm not sure if only an admin can assign issues on the free plan.

I assigned issues 737 and 668 to you.

[peteru]

I fixed 607/crash-in-skip-forward-back-min-on-very as well as #606.

[peteru]

729/skip-back-operation-skips-right-on-v2 also fixed.

[prl]

You Edit the issue, then scroll down and there should be an Assign to me link, just under the search field for Assignee.

When I do that, I don't see any Assign to me link.

However, I'm not sure if only an admin can assign issues on the free plan.

Seems quite likely to be the cause.

I assigned issues 737 and 668 to you.

Thanks.

[prl]

668/link-deletion-deletes-target-not-link-in

The problem here is in MovieSelection.deleteDirConfirmed():

Code: Select all

		cur_path = os.path.realpath(current.getPath())
		if not os.path.exists(cur_path):
			# file does not exist.
			return

I can't think of a good reason why realpath() is being used here, and it's the reason why the directory, rather than the link to it is deleted when a a link is deleted.

[adoxa]

I don't know if this is just my test V2 (20210823), but I'm getting a package error with ntpdate whenever I install a plugin.

Does just seem to be the test V2.

[prl]

737/file-system-check-always-uses-fsckext4

On a tangentially related note, in looking at what needs to be done, I see that the old assumption in Screens.About that if a mount is file system type fuseblk, then the mounted filesystem is NTFS is no longer true. It might also be exFAT, which is a potentially fsck-able filesystem.

I'm thinking that Harddisk.getProcMounts() should look for the actual filesystem of FUSE-mounted filesystems (e.g. by running blkid) and put that in the returned list instead of "fuseblk".

[prl]

fsck.exfat/exfatfsck on the Beyonwiz doesn't have a -p option, but worse, it doesn't seem to do much useful: "exfatfsck checks an exFAT file system for errors. Note that it cannot repair corrupted FS, it just reports found errors."

The ability to do so seems to be in the pipeline, though.

[peteru]

Don't go overboard with the implementation. For the next release, minimal set of changes to prevent the firmware from doing stupid things, like running fsck.ext4 on FAT32 drives, is the best option. Further enhancements can come later, once this release is out the door.

I don't think that running fsck on NTFS or exFAT (or even HFS+) is a good idea. These foreign file systems should be repaired in their native environments where the tools are likely to be well tested and more robust. I'd rather not have the PVR make a mess of a drive that could have been repaired had it been connected to a full blown computer.

Repairing native Linux file systems (and possibly FAT32) should be OK.

Please let me know if additional packages need to be pre-installed in the firmware image. blkid, e2fsprogs and dosfstools are already there, which is probably enough.

As I mentioned in issue #737, you can use blkid to get a reliable list of device names and their filesystems:

Code: Select all

root@devv2:~# blkid -c /dev/null -s TYPE
/dev/mmcblk0p3: TYPE="vfat"
/dev/mmcblk0p13: TYPE="ext4"
/dev/sdb1: TYPE="ext4"

You can then use fsck.vfat and fsck.ext4 on the supported file systems and disable fsck on all other file systems. Alternatively, you could instead show the user a message for NTFS or exFAT file systems, suggesting that in order to check and repair the file system, they should connect the drive to a PC.

[prl]

Don't go overboard with the implementation. For the next release, minimal set of changes to prevent the firmware from doing stupid things, like running fsck.ext4 on FAT32 drives, is the best option. Further enhancements can come later, once this release is out the door.

I don't think that running fsck on NTFS or exFAT (or even HFS+) is a good idea. These foreign file systems should be repaired in their native environments where the tools are likely to be well tested and more robust. I'd rather not have the PVR make a mess of a drive that could have been repaired had it been connected to a full blown computer.

Repairing native Linux file systems (and possibly FAT32) should be OK.

The implementation (as far as it is written) will allow fsck to be done on ext2-4 & vfat, but is easily extended to handle other checkers.

Please let me know if additional packages need to be pre-installed in the firmware image. blkid, e2fsprogs and dosfstools are already there, which is probably enough.

OK, thanks, but I think that's what's there already is enough.

As I mentioned in issue #737, you can use blkid to get a reliable list of device names and their filesystems

The current changes already do that, but it's only needed to identify what's really under a fuseblk mount.

You can then use fsck.vfat and fsck.ext4 on the supported file systems and disable fsck on all other file systems. Alternatively, you could instead show the user a message for NTFS or exFAT file systems, suggesting that in order to check and repair the file system, they should connect the drive to a PC.

The current implementation will only show an error when a check is run on a filesystem that doesn't have check support. I'm looking at adding the check at a higher level, perhaps even making filesystems that can't be checked not selectable in the list where the user chooses which filesystem to check. That's made slightly messier by the fact that that is a list of devices, not a list of partitions/filesystems.

[prl]

The new filesystem check device selection screen. Filesystems that can't be checked are greyed out and can't be selected. The list box has been re-dimensioned so that it's the same size as normal configuration screen config lists, which also makes more horizontal space available so that there's room to display the filesystem type. The SD slot comes up as "Unknown" because it's set up for multiboot and it doesn't mount.

The disk format selection screen looks the same, except that all devices can be selected in that screen.

Should I use a darker grey for the greyed-out entries?

[IanL-S]

Darker may be more likely to grab attention - but then does clearly say unknown. But it may assist seniors such as myself who sight is failing (actually the cataract operations worked like a charm - I seem to be not as alert as I once was). Red flashing text would be an overkill - assuming it is possible.

[sub3R]

Should I use a darker grey for the greyed-out entries?

I think it would depend on which skin is being used. On the nice OverlayHD where the background is black without any bleed through from live TV it would probably be ok as is. But on the other skins with the annoying live TV bleed through it depends on what’s on live TV at the time.

[prl]

It's not only unknown filesystem types that can't be checked - exFAT (bottom entry) can't be either. Sounds like the "can't be checked" text does need to be darker.