getWizPnP - Beyonwiz recording downloader

[prl]

So you will be rate limiting the application on all platforms to cope with the shortcomings of the lowest common denominator? If so, should you be also taking into account DOS limits?
...

I was intending to use

Code: Select all

if($^O eq 'MSWin32')

to set the limits. And possibly larger limits, or unlimited, on more forgiving systems. I need to check whether I can differentiate WinXP from Win7 and other more modern systems.

I've never thought about supporting getWizPnP on DOS.

[peteru]

Perhaps POSIX::uname() will give you more detail. Alternative solution may be $Config{osname} and $Config{osver}.

[peteru]

On a second read, perhaps $Config and $^O are not the right way to go, since they appear to hold configure/build values, rather than execution environment values.

[prl]

I have the opportunity to test for build/run environment differences between Win XP and Win7 for POSIX::uname() and the Perl builtin variables. I've already written a script to do it.

[IanB]

Hm, that's not how I read the function of SO_DONTLINGER.

It's a little rude, in that it slams shut the TCP socket with a RST, which in windows apparently avoids the TIME_WAIT state.

In the past, I've had mixed success in the past with SO_REUSEADDR (perhaps because I don't really understand what it's intended to do).

It is really intended for force reopening a port that you intend to do a LISTEN on when you have brain dead ACCEPT logic, but it can be used any-time to open an in use address-port pair. Of course if you do which socket actually gets the traffic is indeterminate.

Anyway, I can't use any of this without reaching deep into LWP::Protocol::http and messing with an intended-to-be-internal function. This means, amongst other things, having to worry about internal interfaces inside LWP.

Okay, we have the nub. I completely understand the slippery slope of fiddling with the standard black boxes.

I'm now inclining to a scheme to limit the number of ports in WAIT state by keeping track of how many ports have been used in the past 120 or so seconds, and keeping that below some number considerably less that the 3976 allowed by XP. It can be relatively simply implemented with a timestamp queue for HTTP requests, and sleeping while there are more that the given number of WAIT state ports in the queue. THe sleep can even be calculated from the current time and the timestamp in the oldest queue element, so polling should not be necessary.

If you have 2 apps chewing up ephemeral ports you could only have half that number available to you.

The KISS approach is to just just to spin on a socket in use error with a short sleep, say ~30ms, before the retry.

Burning 3976 ephemeral ports in 120 seconds is a fairly moderate rate of connects, ~31/sec.

Perhaps bundling a .REG file to set a more reasonable limits would be a service to your users.

[sub3R]

... If you have 2 apps chewing up ephemeral ports you could only have half that number available to you.

I got caught with something like that yesterday running check from YARDWiz 0.4.1 with IE8 open - the missing data file message came up even though I had TcpTimedWaitDelay set to 60sec. I couldn’t repeat this & to date haven’t had it appear running --check from the command line at the 60sec setting. I think under certain circumstances the setting of 60sec may be a bit too long for my use so I’ll drop this down to 45sec.

[prl]

...If you have 2 apps chewing up ephemeral ports you could only have half that number available to you.

I was intending to allow getWizPnP to have a "ports in WAIT state" queue about 10% of the total available ephemeral ports.

The KISS approach is to just just to spin on a socket in use error with a short sleep, say ~30ms, before the retry.

At the level of interface I'm using (LWP::Simple, and a bit of LWP::UserAgent), I don't directly get a "socket in use" error. I get a HTTP error (as described earlier):

Code: Select all

500 Can't connect to w.z.y.z:49152 (connect: timeout).

on the expiration of the standard 180sec timeout (though I can control the length of this timeout). The timeout error comes pretty much straight from IO::Socket::INET->new(), so there appears to be a timeout on trying to allocate an ephemeral port, perhaps in connect(2). There seems little point in adding an extra delay here, because the 180sec timeout exceeds the 120sec WAIT state duration, and so by the time the timeout expires, all the ports previously in WAIT state that originated from getWizPnP should have completed their close. This points to a fairly ordinary implementation of waiting for the required resources to complete the connection.

So my current retry-based code is simply a retry loop with a 15sec internal timeout on the socket open, and a repeat count on that to bring it up to a total maximum wait of 180sec. No extra delays are necessary. Because the situation arises because getWizPnP is using up ephemeral ports faster than they can close down, it doesn't matter much that there could be up to 15sec of additional delay, because the program will quickly catch up and stall again.

The reason why I have moved away from this idea (though I may retain it as a "last stand" protection), is that it is triggered only when getWizPnP has used up all the ephemeral ports, and is potentially interfering with other activities, as noted by Dennis (sub3r) above.

Burning 3976 ephemeral ports in 120 seconds is a fairly moderate rate of connects, ~31/sec.

The problem is caused by burning the 3976 ephemeral ports in less than 120 sec. Dennis reports that the problem happens after about 85sec, for a port use rate of ~45/sec. The reason that the ports are being burnt so fast is that --check, in contrast to any other getWizPnP functions, performs overwhelmingly HTTP HEAD requests, rather than GETs or POSTs. Probably part of the reason that it isn't even faster is the slowness of the Beyonwiz server, and the relative slowness of Perl over compiled code (even "compiled" versions of getWizPnP are interpreted, not compiled, even if they are in a .exe and superficially behave like compiled code).

Perhaps bundling a .REG file to set a more reasonable limits would be a service to your users.

A couple of reasons for not doing this - it complicates the installation, and I don't have either the technical knowledge or the requisite software to create Windows setup.exe packages. I had a look at acquiring both, and was dismayed. The other is that I'm reluctant to make changes to central system parameters "behind the back" of users, even if the change is relatively innocuous, like increasing the size of the pool of ephemeral sockets. The other Registry changes that have been suggested I would simply not do in an installer, and probably not even recommend.

I appreciate the input from everyone (and most especially Dennis's efforts in tracking down the source of the problem), but I now want to implement and try my port wait queue idea to see to what extent it can be tuned to the execution environment of getWizPnP, and what effect it has on the speed of running getWizPnP. I don't think that the current behaviour of getWizPnP --check is fit for polite system society.

[prl]

I've implemented the code limiting the maximum number of TIME-WAIT state ephemeral ports generated by getWizPnP. I've elected to apply the needed waits smoothly, which means that in a case where a large number of TIME-WAIT ports are generated, but the number is less than the limit, some delay might be applied unnecessarily. This seemed preferable to letting the requests be made as fast as wanted until the limit was reached, and then stalling for about two minutes.

Limiting the number of ports this way means that the minimum average time between requests is 120/(portlimit). If I make the port limit 397 (10% of the 3976 ports available in Win XP), then this time is about 0.3 sec, or alternatively, 3.3 requests/sec.

On my OS X Mac, if I run without the limits --check over my recordings folder (about 5380 requests or ports used) takes 1min32sec. If I apply the 376-port limit (using the fast as possible/stall algorithm), it takes 26min11sec. If I use 10% of the available ports on OS X (total 16383, use 1638), the time should be no more than 40 seconds longer than with no limit (not yet measured).

I'm thinking of allowing 20% rather than 10% on WinXP (which would bring the check time in my example down to about 14min).

Unfortunately, I haven't been able to find a portable way to check what the ephemeral port range is from Perl, so if I do it this way, hacking the WinXP registry will be to no avail.

I have found a way to test the run-time (rather than build-time) Windows version from Perl, so (say) Win7 will have the same sort of added time as for OS X. Similarly for Linux.

Because copy operations take about 10 sec/request, there should be almost no effect on copy speed (but I haven't tested that).

[sub3R]

After Peter’s (prl) suggestion I have posted the following to keep users following this topic informed (this line of communication started as a specific YARDWiz/getWizPnP query & got back onto the getWizPnP --check port issue, therefore the reason to continue on the forum):

Those getwizpnp --check timing figures are a concern. Seeing that --check has such a hunger for ports have you considered allowing it to use something like 75% instead of 10 or 20%, & leave say 25% (or about 1000) for other applications?
I don’t know if this helps, the handful of applications & sites I’ve checked with TCPView set at 1sec update use only a relatively small number of ports. Except for some of the online Live TV sites (which I can’t stream or download) the highest number I’ve seen so far is Coastalwatch surfcam which appears to use about 50.

Peter’s (prl) reply to this:

“The one of the reasons why I want to limit the number of ports to 20% is one raised by a poster on the main thread - someone running two or more copies of getWizPnP, e.g. to check two or more BWs at once, or someone running a second --check immediately another one finishes. Any reasonably long run of --check will exit getWizPnP with all of its allowed ports in TIME-WAIT (if it doesn't it's been pushing the request rate too low), so a second run will need to have a reasonable stock of ports available to use. There's no simple way for one run of getWizPnP to communicate its port use history to another run.

And just as a matter of clarification, the request rate limiting applies to all HTTP requests, not just to the HTTP GET operations used in --check. However, because in --copy and --move operations, each data file copy takes ~10 sec, that falls well below the average request rate that even the 10%/376-port limit applies (0.3 sec between requests). Increasing to 20%/795 ports makes that even less a problem. Other operations like --verbose and the --debug options don't make enough requests for this to be a serious issue.

However, with that known, I'm happy for the potential user base to inform me about what the port use percentage should be, especially on systems like WinXP that only use 1024-5000 as ephemeral ports. From my point of view it's mainly about getWizPnP to be a well-behaved part of the software ecology, and not cause other applications (including other runs of getWizPnP) to fail in unusual ways.” ... (end quote)

[sub3R]

The one of the reasons why I want to limit the number of ports to 20% is one raised by a poster on the main thread - someone running two or more copies of getWizPnP, e.g. to check two or more BWs at once, or someone running a second --check immediately another one finishes. Any reasonably long run of --check will exit getWizPnP with all of its allowed ports in TIME-WAIT (if it doesn't it's been pushing the request rate too low), so a second run will need to have a reasonable stock of ports available to use. There's no simple way for one run of getWizPnP to communicate its port use history to another run.

I hadn’t even considered running getwizpnp --check on two wizzes at the same time. I just tried it using the default WinXP port range & with the default TcpTimedWaitDelay reduced from the 120sec default to 45sec & didn’t experience any problems (I expect 60sec would bring it unstuck though).

Unfortunately I can’t reduce the port limit to 20% to see what times I get running getwizpnp --check on the S1 containing the most recordings (MaxUserPort doesn’t allow the upper limit to be reduced from the default of 5000). If I could reduce it to 20% I don’t know if it would even run without stalling because I can only reduce TcpTimedWaitDelay to 30sec so this may not be enough to do that test.

This is all I could check; the S1 with 367,879MB of recordings took 4min28sec to complete getwizpnp --check with the above reduced TcpTimedWaitDelay of 45sec & the default port range. The 2nd S1 contains 190,150MB of recordings & took 2min33sec to complete getwizpnp --check. All checks were run after allowing enough time for all ports to become available for use.

[prl]

Hi, Dennis, the idea behind limiting the port range use (and adding the retry-on connect failure code) is so that getWizPnP --check can be run without changing the registry. I think it's a lightly-enough used function that the additional delay as a trade-off against either intermittent stalls or modifying the registry is worth it. With the intermittent stalls, it will take even longer than with the request rate limiting (because the connect timeout is 180sec, while the TIME-WAIT timeout is 120sec).

I think that asking users to modify their registry is something I'd rather not do.

I may put in some facility for disabling and/or adjusting the rate limiting in getwizpnp.conf, so that people who do want to make the effort to tune their IP parameters can do so if they want, and have --check take advantage of it.

[sub3R]

Hi Peter. Just so there isn’t any misunderstanding, I wasn’t trying to convince you to recommend a registry change - I’m not in favour of any registry change in this situation. In fact one of the registry changes I made during these tests made certain internet browsing very sluggish. My intention was to try & give times on an XP PC when running --check on a wiz containing a reasonable number of recordings with the ports reduced to 20% - unfortunately I couldn’t find a way to do that so I just posted what I could time in case that was helpful.

... I think it's a lightly-enough used function that the additional delay as a trade-off against either intermittent stalls or modifying the registry is worth it. With the intermittent stalls, it will take even longer than with the request rate limiting (because the connect timeout is 180sec, while the TIME-WAIT timeout is 120sec).

I agree, anything is better than the stall condition & displaying an incorrect missing data file - I don’t think you have any other option than to reduce the ports available to --check.
I must admit, in the past I hadn’t given --check much use thinking that --debug=trunc would pick up most problems, but I wasn’t aware then that --debug=trunc lists the data files it expects to find & therefore can give the impression that a missing data file isn’t missing. So I’ll probably use --check more now.

I may put in some facility for disabling and/or adjusting the rate limiting in getwizpnp.conf, so that people who do want to make the effort to tune their IP parameters can do so if they want, and have --check take advantage of it.

That could be an advantage to users with a lot of recordings on large drives.

[prl]

...

I may put in some facility for disabling and/or adjusting the rate limiting in getwizpnp.conf, so that people who do want to make the effort to tune their IP parameters can do so if they want, and have --check take advantage of it.

That could be an advantage to users with a lot of recordings on large drives.

Now implemented in my development copy. And you can disable port limits completely by setting either the number of system ephemeral ports available, or the use fraction, to 0.

I still have retry on connection failure as a backup protection if you do run out of ephemeral ports, though of course that won't help other applications if they can't create connections because of getWizPnp.

[sub3R]

. That should keep everyone happy, including users using operating systems later than XP.

[peteru]

Power users than really want to hammer multiple Beyonwizes hard, can always use a real OS that uses the full 64k port range and additionally set up virtual interfaces to go even further.

[prl]

Power users than really want to hammer multiple Beyonwizes hard, can always use a real OS that uses the full 64k port range and additionally set up virtual interfaces to go even further.

Generally, a 16k range (49152/0xc000 - 65536/0x10000) rather than 64k. And --check for me (OS X Lion 10.7.3) on the whole of our main PVR's HDD (DP-Lite 500GB, 378GB used) makes nearly 12000 requests in 3min33sec, and so consumes an appreciable fraction of the number of available ports (given the 120sec TIME-WAIT timeout, roughly 2/3 of the 12000, so ~8000 of the available 16k).

Anyway, people who still use WinXP are presumably doing it for some reason (they didn't update to either Vista or WIn7), so I don't suppose that getWizPnP --check limitations would have much influence on their choice of whether to upgrade or not.

But yes, I've set getWizPnP up to take a fairly conservative approach to port use, but that can now be over-ridden by anyone who can copy-paste a Perl assignment expression and edit it a bit.

The number of requests made by getWizPnP is close to proportional to the amount of space used, because almost all the HTTP HEAD requests are made on recording data files, and there are generally far more of them than any other files in the recording. Because, then, there's ~1 request/32MB used, the number of requests can be readily estimated from the amount of HDD used.

One oddity in all of this is that I have yet to see a TIME-WAIT state entry in netstat generated by getWizPnP (or anything else) on OS X Lion.

[prl]

Version 0.5.3 beta3 limited release

Version 0.5.3 released. Do not use the beta versions. Links to beta versions removed.

A new limited release beta version of getWizPnP is available. Only the canned versions are available:

• Windows
• OS X
• Cygwin
• Linux

Changes from the last full release (0.5.2) are:

• Allow output of downloaded recordings to stdout.
• getWizPnP HTTP request rate limited so that it no longer uses up large numbers of available ephemeral ports. Port usage is limited to 20% of available ephemeral ports on Windows XP and older, and 10% of available ephemeral ports on other systems. If, despite the rate limiting above, getWizPnP does exhaust the host's ephemeral ports, it now retries the request if a request fails because of a connection timeout.
• Adds a consistency check to --check between the trunc entry sizes and the recording offsets in the trunc file. The recording offsets should be a running total of the sizes.
• Offsets printed in --debug=trunc and --debug=offsets now print correctly on Windows (this bug only appears to have affected ActivePerl on Windows, including the distributed "compiled" version for Windows).
• Fixes a bug that was causing an incorrect estimated start time for the last trunc entry printed by --debug=trunc if the last entry played for more than 10 sec.
• Changed the make_doc.sh (Unix shell script) to make_doc.pl (Perl script) to make documentation generation more portable to Windows, but the script has problems when run from within a Makefile on the Windows port of Gnu make. Some cosmetic improvements to the HTML output. These scripts are not included in"compiled" versions.
• Started on modifications to Makefile to make it usable on Windows using Windows ports of make and the Unix shell tools. Still not working correctly on Windows. Only affects developers. Not visible to users of "compiled" versions.
• Modified Makefile to include README-VERSION.txt in the "compiled" ZIP files.

Thanks to Dennis (sub3r) for tracking down the cause of incorrect reports of missing files in --check. This problem is the reason for introducing the limit on the number of IP ephemeral ports.

[sub3R]

Hi Peter,

You’ve done a fair bit of work there.

One item I want to confirm.
The getWizPnP.txt file mentions the following in regards to --check:
“... These limitations mean that --check will take about 4.5 sec/GB of recordings on the Beyonwiz on Windows XP or earlier), and will take about 2.1 sec/GB on systems with more ports available.”
and ... “It is possible to increase the fraction of available ephemeral ports that *getWizPnP* uses by setting $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac in the getWizPnP configuration file.”

Just checking to make sure I have this correct in ‘c:\documents and settings\username\application data\prl\getwizpnp\getwizpnp.conf’.
To set the port use to 3900 I added the following line ...
$Beyonwiz::Recording::HTTPAccessor::numEphemPorts=3900;
& to reduce this back to WinXP's 20% port use ...
#$Beyonwiz::Recording::HTTPAccessor::numEphemPorts=3900; or $Beyonwiz::Recording::HTTPAccessor::numEphemPorts=795;
Is this correct?

I’ll do some more checks in a few days time but before making any changes or additions to ‘getwizpnp.conf’, --check took 15min exactly to check the S1 containing 173,985MB of recordings & took the exact same 15min with 3900 ports as per above.
To see if I had this correct I replaced 3900 with 30 & TCPView shows getwizpnp --check only using 6 ports.
I’m obviously missing something or doing something wrong.

[prl]

Hi Peter,

You’ve done a fair bit of work there.

Thanks.

One item I want to confirm.
The getWizPnP.txt file mentions the following in regards to --check:
“... These limitations mean that --check will take about 4.5 sec/GB of recordings on the Beyonwiz on Windows XP or earlier), and will take about 2.1 sec/GB on systems with more ports available.”
and ... “It is possible to increase the fraction of available ephemeral ports that *getWizPnP* uses by setting $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac in the getWizPnP configuration file.”

Just checking to make sure I have this correct in ‘c:\documents and settings\username\application data\prl\getwizpnp\getwizpnp.conf’.
To set the port use to 3900 I added the following line ...
$Beyonwiz::Recording::HTTPAccessor::numEphemPorts=3900;
& to reduce this back to WinXP's 20% port use ...
#$Beyonwiz::Recording::HTTPAccessor::numEphemPorts=3900; or $Beyonwiz::Recording::HTTPAccessor::numEphemPorts=795;
Is this correct?

If you want to use 20% of 3900 available ephemeral ports, the way it's intended to be used is:

Code: Select all

$Beyonwiz::Recording::HTTPAccessor::numEphemPorts  = 3900; # Number of available ephemeral ports
$Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac = 0.2;  # Fraction to use; 20%

That way it's sort-of self-documenting. The number of ports and the fraction you want to use are (or at least can be) expressed separately.

And, of course, you need to remove the initial '#' comment character, though that's probably just a typo in your example. And 20% of 3900 ports is 780

[sub3R]

Now I understand - thanks. I didn’t read the section on --check in getWizPnP.txt carefully enough, I didn’t even notice $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac even though I quoted it. I read it as $Beyonwiz::Recording::HTTPAccessor::numEphemPorts

I was trying to do everything with $Beyonwiz::Recording::HTTPAccessor::numEphemPorts by setting the number of ports for getwizpnp to use (say to 3900) then having an option to return it to your 20% setting of WinXP’s available port range (20% of 3976 = 795) by ‘#’ commenting it or changing the number of ports to 795. Obviously not the correct way to do it.

Which leads to another question.
I assume $Beyonwiz::Recording::HTTPAccessor::numEphemPorts shouldn’t be set to exceed the default number of ephemeral ports (eg. 3976 in WinXP) or that set by ‘MaxUserPort’. If set to more than this it would fall back to the default number or that set by ‘MaxUserPort’ - is this correct?

[prl]

Now I understand - thanks. I didn’t read the section on --check in getWizPnP.txt carefully enough, I didn’t even notice $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac even though I quoted it. I read it as $Beyonwiz::Recording::HTTPAccessor::numEphemPorts

I was trying to do everything with $Beyonwiz::Recording::HTTPAccessor::numEphemPorts by setting the number of ports for getwizpnp to use (say to 3900) then having an option to return it to your 20% setting of WinXP’s available port range (20% of 3976 = 795) by ‘#’ commenting it or changing the number of ports to 795. Obviously not the correct way to do it.

If you don't change variables (any of them) in the getwizpnp.conf file, then they stay at their in-program defaults (getWizPnP sets the program default values, then executes getwizpnp.conf, then decodes the command line).

"Correct" is a bit stronger that I'd put it. That's why I said "intended" in my first reply. What's intended is that if you want getWizPnP to use all the ports that you've said are available, then set $Beyonwiz::Recording::HTTPAccessor::numEphemPorts to that number of ports, and set $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac to 1.0 (or 100/100 if you want to keep the percentage flavour).

Which leads to another question.
I assume $Beyonwiz::Recording::HTTPAccessor::numEphemPorts shouldn’t be set to exceed the default number of ephemeral ports (eg. 3976 in WinXP) or that set by ‘MaxUserPort’. If set to more than this it would fall back to the default number or that set by ‘MaxUserPort’ - is this correct?

getWizPnP has no access to the number of ephemeral ports that the system actually has set as the maximum. It simply finds out what OS it's running on, and sets fixed values assuming that the OS defaults haven't been changed (3976 on WinXP and earlier, 16384 on other systems). If (e.g on WinXP), you increase MaxUserPort, then it's intended that you should set $Beyonwiz::Recording::HTTPAccessor::numEphemPorts to that number.

$Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac is the "how close to the edge do I want to go" control. The program default is deliberately conservative.

If you allow getWizPnP to use more ephemeral ports than the system actually has, it will now handle that case more gracefully, by timing out on the HTTP request and retrying. But in doing this, it will have used up all the ephemeral ports, and this may have a detrimental effect on other applications that want to use networking.

Play with the numbers all you like, but don't tell getWizPnP porkies

[sub3R]

A very clear description Peter.
I can do all the fine tuning I need to do with just $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac = 0.n;

[prl]

A very clear description Peter.
I can do all the fine tuning I need to do with just $Beyonwiz::Recording::HTTPAccessor::ephemPortsFrac = 0.n;

Thanks. I'll have a look at the comments in the sample getwizpnp.conf file and the manual page and try to clarify them a bit.

[Luke]

Hi Peter, I've been testing 0.5.3 beta3 in YARDWiz with no issues I can see. The printing to stdout works well and I've implemented in YARDWiz (with the option to use a temp file if preferred which allows pause/rewind as I haven't figured out how to get VLC to "timeshift" when reading from stdin).

[prl]

Hi Peter, I've been testing 0.5.3 beta3 in YARDWiz with no issues I can see. The printing to stdout works well and I've implemented in YARDWiz (with the option to use a temp file if preferred which allows pause/rewind as I haven't figured out how to get VLC to "timeshift" when reading from stdin).

Thanks, Luke.

If no-one else reports any problems, I'll look for a proper release next weekend with only changes to the documentation.

[Luke]

I have noticed something. Testing on Win 7 Pro using the "compiled" versions, 0.5.3beta is considerably slower (2-3x) than 0.5.2 to list recordings (getWizPnP.exe --all -H <IP> -p <port>). My wiz currently has 25 recordings (~100GB). Timing the getWizPnP calls over 10 runs each I get:

Code: Select all

Version   Mean  Max  Min (seconds to list all recordings)
0.5.2      7.8  8.4  7.5
0.5.3beta 22.1 22.5 21.7

[prl]

I have noticed something. Testing on Win 7 Pro using the "compiled" versions, 0.5.3beta is considerably slower (2-3x) than 0.5.2 to list recordings (getWizPnP.exe --all -H <IP> -p <port>). My wiz currently has 25 recordings (~100GB). Timing the getWizPnP calls over 10 runs each I get:

Code: Select all

Version   Mean  Max  Min (seconds to list all recordings)
0.5.2      7.8  8.4  7.5
0.5.3beta 22.1 22.5 21.7

OK, I can see how that would happen. It's similar to --check: larger numbers of small requests. In fact it's exacerbated by another optimisation: I only fetch the parts of header.tvwiz that I need for any particular operation, which means that there can be more than one access of header.tvwiz per recording listed.

Looks like I need to have a bit more of a think on this.

Thanks

[prl]

Even without --verbose, getWizPnP --list does two HTTP HEAD and two HTTP GET operations per recording. One of the HEAD operations is the result of a HTTP redirect, so both are due to the same request in the getWizPnP code. One of the GETs is for the basic recording information and one for the episode name.

Even with the more generous number of ports used in non-WinXP systems, the ephemeral port restriction can cause some significant delay. On those systems, the average number of requests that are passed is ~13/sec, so only about 3 recordings/sec can be listed (respectively ~6/sec and 1.5/sec on WinXP).

I have already coded around the HTTP HEAD requests, and that improved the --list time for my recordings with the ephemeral port restriction from ~98 sec to about 43 sec. This compares with 27 sec for --list without any restrictions. 121 recordings, MacBook Pro 2.26GHz, OS X 10.7.3.

Some further improvement is possible. getWizPnP only decodes a span of about 84kB of the 512kB of the header file, so the "only fetch what's needed" optimisation may not actually gain much over always loading all the information getWizPnP ever uses (basic info, offsets table, bookmarks table, episode name, program synopsis) in a single rwquest. By making getWizPnP load all at once, then the number of requests would be halved over --list without --verbose, and be improved even more for more verbose output and/or --debug=offsets output. That would mean that listing all the recordings would be limited to about 13/sec, no matter what the --verbosity.

This change would simplify some of the getWizPnP code, but will take a bit of rewriting.

[prl]

Explanation of slow --list substantially rewritten.

[Luke]

I wondered if it might have something to do with the ephemeral port restriction. Leaving numEphemPorts at default and upping the ephemPortsFrac in getWizPnP.conf I managed to get 0.5.3beta3 listing at roughly the same speed as 0.5.2.

If you decide not to modify the getWizPnP code, I'll just note the listing speed issue and workaround in the YARDWiz help and point users to the getWizPnP doc.

[prl]

I have noticed something. Testing on Win 7 Pro using the "compiled" versions, 0.5.3beta is considerably slower (2-3x) than 0.5.2 to list recordings (getWizPnP.exe --all -H <IP> -p <port>). My wiz currently has 25 recordings (~100GB). Timing the getWizPnP calls over 10 runs each I get:

Code: Select all

Version   Mean  Max  Min (seconds to list all recordings)
0.5.2      7.8  8.4  7.5
0.5.3beta 22.1 22.5 21.7

Now fixed, by rewriting the code that loads and decodes header.tvwiz so that it only issues one HTTP request no matter what parts of the header file are actually required, and fixing the port use limitation code so that it works properly (operations run faster, in most cases).

The averages I get for ten rounds of getWizPnP.pl --host=10.1.1.8 --list --all with 124 recordings are:

Code: Select all

Version    Mean  (seconds to list 124 recordings)
0.5.2      37.1
0.5.3beta4 31.7  (limited to 10% of 16384 ephemeral ports)
0.5.3beta4 31.5  (port use unlimited)

Yes, for --list the port-limited code is now faster than the old non-limited code Just shows the inadvisability of optimisations without proper performance checking.

Of course, --check is still slowed considerably: 2min45 without the limiter, 11min:31 with the limiter.

[prl]

I wondered if it might have something to do with the ephemeral port restriction. ...

It does. I said so myself.

If you decide not to modify the getWizPnP code, I'll just note the listing speed issue and workaround in the YARDWiz help and point users to the getWizPnP doc.

Code modified. Now faster than 0.5.2, even with (non-WinXP) port restrictions.

[Luke]

It does. I said so myself.

I know you did. Which is why I used the past tense, wondered.

Code modified. Now faster than 0.5.2, even with (non-WinXP) port restrictions.

Good stuff!

[prl]

It does. I said so myself.

I know you did. Which is why I used the past tense, wondered. ...

So you did.

[prl]

Version 0.5.3 beta4 limited release

Version 0.5.3 released. Do not use the beta versions. Links to beta versions removed.

There was a serious bug in the first version of this release put on the download server. That bug has now been fixed and a new version of the release has been put on the server. If you downloaded this release before this notice appeared, download in again and use the newer version.

A new limited release beta version of getWizPnP is available. Only the canned versions are available:

• Windows
• OS X
• Cygwin[
• Linux

Changes from the last full release (0.5.2) are:

• Allow output of downloaded recordings to stdout.
• Adds a consistency check to --check between the trunc entry sizes and the recording offsets in the trunc file. The recording offsets should be a running total of the sizes.
• Fixed a bug in --check where header file size errors were being reported as wrong-name errors.
• When checking trunc file entries in --trunc, don't reload the recording file's header info if the entry is for the same recording file as the previous entry.
• Offsets printed in --debug=trunc and --debug=offsets now print correctly on Windows (this bug only appears to have affected ActivePerl on Windows, including the distributed "compiled" version for Windows).
• Fixes a bug that was causing an incorrect estimated start time for the last trunc entry printed by --debug=trunc if the last entry played for more than 10 sec.
• getWizPnP HTTP request rate limited so that it no longer uses up large numbers of available ephemeral ports. Port usage is limited to 20% of available ephemeral ports on Windows XP and older, and 10% of available ephemeral ports on other systems. If, despite the rate limiting above, getWizPnP does exhaust the host's ephemeral ports, it now retries the request if a request fails because of a connection timeout.
• Reduced the number of HTTP requests needed to fetch the header.tvwiz data to improve performance of common operations when the port usage is restricted.
• Changed the make_doc.sh (Unix shell script) to make_doc.pl (Perl script) to make documentation generation more portable to Windows, but the script has problems when run from within a Makefile on the Windows port of Gnu make. Some cosmetic improvements to the HTML output. These scripts are not included in"compiled" versions.
• Started on modifications to Makefile to make it usable on Windows using Windows ports of make and the Unix shell tools. Still not working correctly on Windows. Only affects developers. Not visible to users of "compiled" versions.
• Modified Makefile to include this file in the "compiled" ZIP files.
• Added the ability to checkModules.pl to only issue warnings about some optional modules on particular systems. Mainly so that builds on non-Windows based systems don't complain about the Win32 module not being available. Only affects developers. Not visible to users of "compiled" versions.

Also improved the speed of the port allocation restrictions over previous beta versions.

Thanks to Luke for raising the issue of the decreased performance in --list due to the ephemeral port restrictions and for pointing out the bug in the original version released.

[Luke]

I get an error message and can't download with 0.5.3beta4:

Code: Select all

getWizPnP.exe -R --BWName -O F:\Videos\WizDownloads -T "The Shadow Line - Episode 2 2012-03-30"  -H 192.168.0.4 -p 49152 -t "recordings/The Shadow Line Mar.30.2012_21.28"
ABC1: The Shadow Line/Episode 2 - Copy
Undefined subroutine &Beyonwiz::Recording::Recording::is_success called at Beyonwiz/Recording/Recording.pm line 488.

[prl]

I get an error message and can't download with 0.5.3beta4:

Code: Select all

getWizPnP.exe -R --BWName -O F:\Videos\WizDownloads -T "The Shadow Line - Episode 2 2012-03-30"  -H 192.168.0.4 -p 49152 -t "recordings/The Shadow Line Mar.30.2012_21.28"
ABC1: The Shadow Line/Episode 2 - Copy
Undefined subroutine &Beyonwiz::Recording::Recording::is_success called at Beyonwiz/Recording/Recording.pm line 488.

I hate run-time symbol resolution.

But thanks. I think I know what caused the problem. I did test dowmloading, but not with --BWName.

[Luke]

For info, downloading using --BWName works when --stdout is passed to getWizPnP.

[prl]

For info, downloading using --BWName works when --stdout is passed to getWizPnP.

I just re-checked, and the problem has nothing to do with --BWname, and yes, that particular error can't occur with --stdout, though I'm a bit surprised a similar one didn't pop up elsewhere.

The problem is due to trivial largely cosmetic changes I made after testing the copy function. I was just switching over from using the deprecated RC_* status codes in the HTTP::Status module to the recommended HTTP_* status codes, and the change was "so small" I didn't re-test

Fixed now, and testing

[prl]

New version now available that fixes the problem. Go to the original release post and download again.

[prl]

Version 0.5.3 beta4 limited release
There was a serious bug in the first version of this release put on the download server. That bug has now been fixed and a new version of the release has been put on the server. If you downloaded this release before this notice appeared, download in again and use the newer version.

Go to the original release post to download again.

Apologies for any inconvenience caused.

[Luke]

I've come across no issues with 0.5.3 beta4 in basic testing with YARDWiz. Seems to be working well now. Much faster when listing, good work!

[prl]

I've come across no issues with 0.5.3 beta4 in basic testing with YARDWiz. Seems to be working well now. Much faster when listing, good work!

Thanks!

[sub3R]

Unfortunately I can’t help with testing Peter. Still on holidays.

[prl]

Unfortunately I can’t help with testing Peter. Still on holidays.

Thanks for letting me know. I was wondering why I hadn't heard from you. I'll give Luke a few more days to bend, fold, spindle and mutilate it and then do a release if he doesn't find any more bugs.

Have a great holiday!

[Luke]

I've been playing with 0.5.3beta4 for a week now with no issues.

[prl]

Thanks, Luke. Looks like it might be time to crank out a release. Next weekend, possibly. I still want to play a bit with trying to get the Makefile running properly under Windows.

[prl]

Version 0.5.3

A new version of getWizPnP is available from the GetWizPnP Release page on the OpenWiz Software Pages.

This version adds the --stdout option for --copy and --move to allow getWizPnP to pipe recording data direct to another application (like a media player).

This version fixes the problems reported in 0.5.2 where files were incorrectly reported missing by --check. This has resulted in some performance loss for --check. Other operations remain unaffected or are improved: --list should now be faster. Fixes bugswhere recording offsets would be reported incorrectly on Windows XP using ActivePerl and the incorrect reporting of the start time of the last entry in a truc file (--debug=trunc). Some improvements to the build process - these should only be visible to users of the getWizPnP source package.

Thanks to Dennis (sub3r) and Luke for finding and helping locate the bugs fixed in this version.

[prl]

--resume does not work properly in 0.5.3. Work is under way to fix this.