Feed server question/issue...

[IanSav]

Hi All,

I am trying to create a feed server to serve the OverlayHD skin and any other future packages I write for the Beyonwiz and OpenViX communities. I have a test environment on my internal web server and a production environment on the BitBucket web server. My test server uses the http protocol while the production server uses https.

I can use the "opkg update" command from the Enigma2 command line on both servers and it runs to completion without error:

Code: Select all

root@beyonwizt3:~# opkg update
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/3rdparty/Packages.gz.
Updated source 'beyonwiz-3rdparty'.
Downloading https://iansav.bitbucket.io/feeds/Packages.gz.
Updated source 'IanSav-Plugins'.
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/all/Packages.gz.
Updated source 'beyonwiz-all'.
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/beyonwizt3/Packages.gz.
Updated source 'beyonwiz-beyonwizt3'.
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/inihdx/Packages.gz.
Updated source 'beyonwiz-inihdx'.
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/inihdx_3rdparty/Packages.gz.
Updated source 'beyonwiz-inihdx_3rdparty'.
Downloading http://feeds.beyonwiz.org/beyonwiz/17.5/inihdx/mips32el/Packages.gz.
Updated source 'beyonwiz-mips32el'.

When I point opkg to my test feed and run "MENU > Setup > Software manager > Check for updates now" everything works fine. However, when I point opkg to my production feed I get an error that results in the message "Can not retrieve data from feed server. Check your Internet connection and try again later.". I believe this message is being generated from line 70 of "Screens/SoftwareUpdate.py". This is basically reporting that the "opkg update" command is returning a message containing 'wget returned 1' or 'wget returned 255' or '404 Not Found'. As demonstrated above this is NOT what I get when I manually run the opkg command.

As far as I can see the only difference is that one server is using http and the other https. The command line version of "opkg update" seems fine with the difference.

Does anyone have any ideas or suggestions on what may be going wrong?

Regards,
Ian.

[adoxa]

You could try running the command from a script - that's a cross between the two environments. If it fails there, you'll at least be able to see the error messages; if it works, well, I'm at a loss.

[prl]

...
When I point opkg to my test feed and run "MENU > Setup > Software manager > Check for updates now" everything works fine. However, when I point opkg to my production feed I get an error that results in the message "Can not retrieve data from feed server. Check your Internet connection and try again later.". I believe this message is being generated from line 70 of "Screens/SoftwareUpdate.py". This is basically reporting that the "opkg update" command is returning a message containing 'wget returned 1' or 'wget returned 255' or '404 Not Found'. As demonstrated above this is NOT what I get when I manually run the opkg command.
...
Does anyone have any ideas or suggestions on what may be going wrong?
...

Really only a meta-suggestion: print the full contents of result in UpdatePlugin.checkNetworkStateFinished(). Possibly also add '-d' (debug) as an option to the "opkg update" command in UpdatePlugin.checkNetworkState(), so "opkg -d update. I don't have access to a Beyonwiz at the moment, so I can't check whether the Beyonwiz version of wget has the '-d' option, so you might want to test that manually before trying to use it in the code.

[IanSav]

Hi All,

It has been suggested that the shell invoked from the command line may not match the shell as invoked by Enigma2. Any differences may be affecting the way opkg or wget work.

I have been sidetracked writing a perl tool to build the package list from all the modules stored on the feed server. I will get back to looking at the http vs https issue shortly.

Regards,
Ian.

[prl]

Hi All,

It has been suggested that the shell invoked from the command line may not match the shell as invoked by Enigma2. Any differences may be affecting the way opkg or wget work.

On what evidence? When a command string "cmd" is run from Components.Console via eConsoleAppContainer, it's effectively running:

/bin/sh -c 'cmd'

I don't have the access to see what the root shell is in the Beyonwiz firmware, but if you want to check from the command line how the command is run from enigma2, then just do:

/bin/sh -c 'opkg update'

in the commandline interface, because that's what enigma2 is doing. However, there might be differences in the environment variables for enigma2 (which is run via /usr/bin/enigma2.sh from init) than from a normal user shell.

I think it may be more constructive to try to find out more information about what the error is.

[IanSav]

Hi Prl,

The "/bin/sh -c 'opkg update'" command runs fine.

I don't see any suspicious shell variable present in the interactive shell. As you say I will be digging into this more to identify what is generating the error when the catalogue builder is complete.

Regards,
Ian.

[prl]

...
I don't see any suspicious shell variable present in the interactive shell. ...

The problem may be because environment variables are set in the enigma2 environment that aren't set (or set to something innocuous) in the interactive shell environment. The environment of the caller process will be inherited by callec processes (whoch may modify them before passing them on to the children).

U*ix environment variables behave differently from Windows ones: in Windows they are global or user global (IIRC), in U*ix they are local to the process and inherited by sub-processes (possibly inherited with modification, or even replacement).

I don't think it's likely that environment variables are the cause of the problem, but they're the only thing I can think of that would be different for a process run from the shell vs one run from enigma2.

[IanB]

Problem is caused by environment LD_PRELOAD=/usr/lib/ libopen.so.0.0.0 in enigma.sh

This is a tools stub that overrides all the libc open/fopen calls in an application and sets the FD_CLOEXEC flag on the returned fd so they cannot leak into any child processes.

The gotcha is the LD_PRELOAD environment is inherited by all child processes spawned from enigma2 like the calls to opkg.

Opkg calls wget to do it's repo file fetching.

Busybox wget calls out to openssl to do it's https handling and you guessed it, the opened stdin and stdout fd's are of course flagged as closed on exec.

Enigma2 probably should unset the LD_PRELOAD environment before it spawns child processes.

[IanSav]

Hi,

IanB beat me to the post.

Thanks to his help the bug is identified and could well affect *anything* that is run in a shell spawned by Enigma2.

The problem is the Enigma2 supplied libopen.so.0.0.0.

libopen.so.0.0.0 hack was added to Enigma2 in 2006?  It changes the default clib calls to open files or pipes.  It forces any files open files or pipes to be closed on an exec system call.  This library is invoked in the Enigma2 startup script.  When Enigma launches a shell the shell inherits this hack.

In this problem case calls to opkg use a call to wget to fetch files.  Wget uses openssl to get https files.  Because of libopen.so.0.0.0 the open file channels that should be between wget and openssl get closed causing the link between wget and openssl to fail.

The fact that we found this bug in my call to opkg is just a random failure. The underlying bug could cause erratic behaviour is *any* programs launched within Enigma2.

Does libopen.so.0.0.0 still serve and useful purpose or is it a leftover from some previous issue.  If there is a desire to protect some file channels from any sub processed then those specific handles should have the close on exec bit specifically set.  This should not be a global Enigma2 hack.

What can be done to fix this bug?  Is it as simple as removing the libopen.so.0.0.0 hack?  Can we find out why it is there or what it is trying to protect.  Git blame points at who added the hack (Andreas Monzner committed Oct 18, 2006) but there is no documentation as to why.

Regards,
Ian.

[IanSav]

Hi,

Feedback from the OpenViX developers is that this was done to stop Enigma2 from passing on the many file handles it opens on to all of its children. This was causing a significant number of unused file handles to be created. This wastes the limited resources on typical PVR hardware.

Hopefully when Prl returns from caring for his daughter he may be willing to create a test build of Enigma2 that unsets the variable when Enigma2 has started. This should solve my problem and the many other potential failures that could occurs in shell scripts invoked from Enigma2.

Regards,
Ian.

[prl]

...
Hopefully when Prl returns from caring for his daughter he may be willing to create a test build of Enigma2 that unsets the variable when Enigma2 has started. This should solve my problem and the many other potential failures that could occurs in shell scripts invoked from Enigma2.
...

That shouldn't be hard to do. Could you write up a bug in the issue tracker?

[IanSav]

Hi Prl,

Issue #602 created.

Please note that this issue affects Beyonwiz, OpenViX and probably all builds of Enigma2. I would welcome a fix for both the Beyonwiz and OpenViX builds.

Regards,
Ian.

[prl]

It should be the same fix for both Beyonwiz and OpnViX.

Could you add a sample feed URL to the issue for me to use for testing?

[IanB]

Recommend don't stuff around, nuke it first thing.

main/enigma.cpp

Code: Select all

int main(int argc, char **argv)
{

#ifdef MEMLEAK_CHECK
	atexit(DumpUnfreed);
#endif

#ifdef OBJECT_DEBUG
	atexit(object_dump);
#endif

	unsetenv("LD_PRELOAD"); //    <------- First thing

	gst_init(&argc, &argv);

	// set pythonpath if unset
	setenv("PYTHONPATH", eEnv::resolve("${libdir}/enigma2/python").c_str(), 0);

[IanSav]

Hi Prl,

The example URL was concealed in my sample and bug report.

The production URL will be "https://iansav.bitbucket.io/feeds".

To test the feed add "IanSav-feed.conf" to "/etc/opkg" containing:

Code: Select all

src/gz IanSav-Plugins https://iansav.bitbucket.io/feeds

If you would like the test feed that does work please send me a PM. (I don't want the http test server to become public.)

By the way, I would be very pleased to host the Australian picons package on my feed server. This will allow users to update picons whenever you make an update without having to wait for PeterU to create a full new firmware build.

Regards,
Ian.

[prl]

IanSav, thanks.

IanB, yes, that's what I was intending to do.

[IanSav]

Hi,

One of the OpenViX developers proposes the adding the following to "mytest.py" as an appropriate fix:

Code: Select all

# LD_PRELOAD will(?) be set when we start-up, but we don't want to pass
# it on to sub-processes
#
try:
        del os.environ["LD_PRELOAD"]
except:
        pass

I am about to give this a try.

Regards,
Ian.

[prl]

Doing it there should work fine.

IanB's suggested method does the same thing, just a bit earlier.

[IanSav]

Hi,

The code has been tested and is working fine!

I will file pull requests to both repositories shortly.

Regards,
Ian.

[IanSav]

Hi,

Pull requests to Beyonwiz and OpenViX repositories submitted.

Regards,
Ian.

[peteru]

Please make sure that you test this very thoroughly with as many plugins and code paths that launch external programs as you can find. While this change may fix one specific problem for your use case, it has the potential to break a multitude of other things. There could also be potential security implications.

Off the top of my head a few areas to test would be: disk partitioning, disk formatting, fsck, mounting, init script invocations, mcut, createapsc, File Commander, file copying/moving/downloading, HbbTV/YouTube/web browser and streaming. Ideally your commit messages (not the pull request messages) should contain information on the testing methodology and the findings.

The LD_PRELOAD hack is ugly, but worse than that, it lacks enough documentation to determine if it still needs to exist. If we knew what specific manifestation of a problem it was addressing, we could probably get rid of it altogether, instead of adding another hack to provide a workaround for some side effect. As a first step in getting some answers, it may be interesting to instrument libopen.c to report instances (including stack trace) where it has forced the FD_CLOEXEC flag.

[prl]

I'm not sure that the alarm is warranted. My understanding is that this change won't change the close-on-exec behaviour of enigma2 files. Just sub-processes of enigma2 will now not have forced close-on-exec, which is their normal expected behaviour, for example when run from a login shell.

However, it would be interesting to know just what problems the libopen library is trying to work around to see whether it can be avoided altogether.

[IanSav]

Hi,

My apologies for neglecting to post the reason for the LD_PRELOAD hack.

According to an OpenViX developer (Birdman) the reason for the hack was to try and control the number of open file handles being propagated through the system. Enigma2 can open a significant number of files while in operation. Duplicating all these file handles was placing an unacceptable burden on some of the platforms running Enigma2.

By the way, this fix was actually proposed by Birdman and the pull request has already been merged on OpenViX.

Regards,
Ian.

[IanSav]

Hi,

I had a chat to Rowland about the LD_PRELOAD fix that has been merged into OpenViX. I am concerned about the memory leak warning in the Python documentation regarding the use of os.environ. I have given Rowland IanB's suggested cpp fix and asked his opinion on which way we should go.

I suspect that Rowland will, at my request, revert the Python version of the fix and go with the cpp version. This change will ensure that we do not suffer the potential memory leak in Python version.

I will update this thread with the pull request that PeterU can cherry pick to add the fix to the Beyonwiz firmware.

Regards,
Ian.

[prl]

I think the cpp code is the better way to fix this for lots of reasons.

[IanSav]

Hi,

The Python fix has been reverted and the cpp fix is being merged. It is going via Rowland's repository so the code can be built and tested before being merged into OpenViX.

Regards,
Ian.

[IanB]

I think the cpp code is the better way to fix this for lots of reasons.

[prl]

Do I need to do a corresponding fix in the Beyonwiz code?

I'm heading back home tomorrow (Fri 25 Aug), and I can probably do it on the weekend.

[IanSav]

Hi Prl,

Do I need to do a corresponding fix in the Beyonwiz code?

Yes please. Rowland can only assist with the OpenViX build.

I'm heading back home tomorrow (Fri 25 Aug), and I can probably do it on the weekend.

How is your daughter?

Regards,
Ian.

[prl]

...
How is your daughter?
...

She's OK, thanks. It's been one step in an uncomfortable and sometimes confronting but not dangerous, process.

[IanSav]

Hi Prl,

I am now running a test build of enigma on my OpenViX box. All looking good. Rowland wants me to check that the box remains stable for a while before the change will be merged into OpenViX.

I can't find HbbTV on OpenViX to test that and we don't use TeleText here any more. I will be loading the web browser shortly to test that. I wonder what else I should be checking?

By the way, Rowland asked me to draw your attention to your OpenViX commit for the boot time fix that has had to be reverted due to a crash. See https://github.com/OpenViX/enigma2/comm ... 702eb9fc55

Regards,
Ian.

[prl]

...
I can't find HbbTV on OpenViX to test that and we don't use TeleText here any more. I will be loading the web browser shortly to test that. I wonder what else I should be checking?

I'd try some other operations that run subprocesses. Peteru suggested a few. I doubt that the fix would cause problems, but it's well worth testing.

By the way, Rowland asked me to draw your attention to your OpenViX commit for the boot time fix that has had to be reverted due to a crash. See https://github.com/OpenViX/enigma2/comm ... 702eb9fc55
...

OK, thanks for that.

[IanSav]

Hi Prl,

Have you had an opportunity to build the updated Enigma2 binary and try it out?

The code has been accepted into OpenViX and will be included in the next Dev and Production releases.

I am hoping this can be part of the Beyonwiz build ASAP.

Regards,
Ian.

[prl]

...
Have you had an opportunity to build the updated Enigma2 binary and try it out?
...

No, not yet. I'll probably be able to do it today.

[IanSav]

Hi Prl,

No, not yet. I'll probably be able to do it today.

Thank you.

Please note that I have split my feeds into Beyonwiz and and OpenViX feeds. The Beyonwiz feed is now https://iansav.bitbucket.io/feeds/Beyonwiz.

By the way, I declined the mytest.py version of the fix pull request so that only the preferred solution is pending on the Beyonwiz repository.

Regards,
Ian.

[prl]

Hi, IanSav.

I'm unable to test this. The required file doesn't seem to be available on your feeds server:

Code: Select all

root@beyonwizt4:/etc/opkg# wget https://iansav.bitbucket.io/feeds/Packages.gz
Connecting to iansav.bitbucket.io (104.192.143.11:443)
wget: server returned error: HTTP/1.1 404 Not Found
root@beyonwizt4:/etc/opkg#

[cwiggles]

Hi, IanSav.

I'm unable to test this. The required file doesn't seem to be available on your feeds server:

Code: Select all

root@beyonwizt4:/etc/opkg# wget https://iansav.bitbucket.io/feeds/Packages.gz
Connecting to iansav.bitbucket.io (104.192.143.11:443)
wget: server returned error: HTTP/1.1 404 Not Found
root@beyonwizt4:/etc/opkg#

If it's any help PRL, the following does result in a smallish 1.5k gz file using a web browser

https://iansav.bitbucket.io/feeds/Beyonwiz/Packages.gz

but the gz files seems corrupt, nothing I have can unpack it. Out of interest I opened it in a text editor and lo and behold it's a text file and not a compressed zip or tar archive as the extension would have you believe.

Craig

[prl]

...
If it's any help PRL, the following does result in a smallish 1.5k gz file using a web browser

https://iansav.bitbucket.io/feeds/Beyonwiz/Packages.gz

That works for me. I missed the new URL that IanSav posted today.

but the gz files seems corrupt, nothing I have can unpack it. Out of interest I opened it in a text editor and lo and behold it's a text file and not a compressed zip or tar archive as the extension would have you believe.
...

When I download that link, I get a 706-byte file Packages.gz that unpacks using gunzip to a 1610-byte file Packages, with the sort of content that I'd expect.

[prl]

I have checked that IanSav's pull request fixes this problem and doesn't cause any problems with some of the areas that peteru asked to be checked. For more details, see my comment in the pull request. I have also tticked off my review for the pull request.

[IanSav]

Hi Prl,

Thank you for your testing and verification.

Sorry about the URL change but it became apparent that offering Beyonwiz packages to OpenViX users and vice-versa was not optimal.

I hope this pull request gets merged in soon so that I can make future updates of OverlayHD *much* easier for all its users.

Regards,
Ian.

[IanSav]

Hi Craig,

If it's any help PRL, the following does result in a smallish 1.5k gz file using a web browser

https://iansav.bitbucket.io/feeds/Beyonwiz/Packages.gz

but the gz files seems corrupt, nothing I have can unpack it. Out of interest I opened it in a text editor and lo and behold it's a text file and not a compressed zip or tar archive as the extension would have you believe.

I have noted this behaviour before. The browser is being too smart and deciding that compressed text files should be expanded on download. If you use something like wget then the file will come across without molestation.

Regards,
Ian.

[IanSav]

Hi PeterU,

I notice that you are making changes to the Beyonwiz repository while the next firmware release has not yet been made. Can you please merge Pull Request #380 so that this fix is also included in the next release.

Prl has already tested this fix in the Beyonwiz context:

I have checked that IanSav's pull request fixes this problem and doesn't cause any problems with some of the areas that peteru asked to be checked. For more details, see my comment in the pull request. I have also tticked off my review for the pull request.

I and the whole OpenViX community have tested the fix in the OpenViX context. No issues detected or reported.

Regards,
Ian.