getWizPnP - Beyonwiz recording downloader

prl · Post by **prl** » Mon Apr 27, 2009 23:38

A new version (0.4.3a) of getwizPnP is available from the GetWizPnP Release page on the OpenWiz Software Pages.

In future I'll make announcements of new releases of getWizPnP in this topic. Please post bugs & enhancement requests here.

The getWizPnP topic in Software Developers should now be reserved for implementation issues. I'll post replies to bug reports and enhanement requests that are posted in Sofwtare Developers here, quoting the original post. I'll link to the reply from the Software Developers topic.

Thanks to glow on the Beyonwiz forum for finding the bugs in the length of the --ts recording transfers for Keep A->B or Copy A->B to New File recordings, and the problem with transferring "topped" recordings.

Version 0.4.3a

Download getWizPnP 0.4.3a

New in 0.4.3a:

Worked around a bug in the Beyonwiz firmware that caused too much of a recording to be transferred in --ts mode if the recording had been edited/created with Keep A->B or Copy A->B to New File.
Changed the outputs of recording sizes, transfer rates and recording bitrates from MiB and MiB to MB and Mb.

Fixed in 0.4.3a:

Worked around a bug in the Beyonwiz firmware that caused too much of a recording to be transferred in --ts mode if the recording had been edited/created with Keep A->B or Copy A->B to New File. This is strictly a workaraound for a Beyonwiz bug, rather than a getWizPnP bug fix.
Recordings that have been "topped" now transfer correctly in --ts
mode.
If you specify --resume on a "topped" recording that hasn't been
started, the recording transfers correctly.

Bugs in 0.4.3a:

The changes to the file indexing from 1.05.283 mean that if you delete a recording on the Beyonwiz (--delete/--move), it will remain in the internal index (i.e. visible in the file player, and visible in WizFX) until you start a recording index rebuild using FILEPLAYER, SOUNDTRACK on the remote, or a HDD check is done. This can only be fixed in the Beyonwiz firmware.
In free-to-air EPGs, the information used by getWizPnP for the episode name is sometimes actually the program synopsis.

IanSav · Post by **IanSav** » Tue Apr 28, 2009 08:46

Hi Peter,

A few observations...

The newer names for the standard Beyonwiz folders are "Recordings", "Contents" and "Trashcan". That is, the first letters are capital letters.

If anyone uses an subnet mask of anything bigger than 255.255.255.0 then only using the last byte of the IP address will not uniquely resolve the device (if the name is also left as default).

Regards,
Ian.

download · Post by **download** » Tue Apr 28, 2009 08:57

Hi prl, looks great. Just one thing I never quite determined. PnP is just for downloading isn't it? There's no upload option (and won't be?)

Regards

Peter Gillespie

prl · Post by **prl** » Tue Apr 28, 2009 09:28

IanSav wrote:Hi Peter,

A few observations...

The newer names for the standard Beyonwiz folders is "Recordings", "Contents" and "Trashcan". That is, the first letters are capital letters.

That's true for the current betas, but not for the current released version. Since folder matching is case-insensitive, this is only a documentation issue anyway. I'm happy to update the documentation when the next public firmware release happens. Are the contents of the Trashcan available in index.txt? I've always assumed that they are not, but that may be incorrect, because I haven't tested it.

IanSav wrote: If anyone uses an subnet mask of anything bigger than 255.255.255.0 then only using the last byte of the IP address will not uniquely resolve the device (if the name is also left as default).

Regards,
Ian.

You appear to be assuming an algorithm from the examples. I chose the most common netmask value for the examples. The actual algorithm correctly extracts the host address from the IP address, no matter what the netmask is set to (i.e. hostAddr = ipAddr & ~netmask).

prl · Post by **prl** » Tue Apr 28, 2009 09:49

download wrote:Hi prl, looks great. Just one thing I never quite determined. PnP is just for downloading isn't it? There's no upload option (and won't be?)

Regards

Peter Gillespie

I've posted about this a number of times in the past, but in summary:

The HTTP server in the Beyonwiz has code in it that is intended to implement HTTP PUT to allow files to be uploaded.
The implementation of HTTP PUT in the Beyonwiz is buggy: any PUT operation results in an empty file being created, no matter what the PUT document contents are.
There is a corresponding bug in the source code of the version of shttpd used in the Beyonwiz.
I provided a patch fixing the bug in the source code version to both Beyonwiz (12 Jun 2008 on the beta forum) and to the author of shttpd.
So far, Beyonwiz has not fixed the bug in the shttpd embedded in the wizpnp server.
For beta forum members, the patch was posted in the topic SHTTPD put bug.

If HTTP PUT was fixed, then getWizPnP (and all its similar friends) could upload files to the Beyonwiz. Upload operations would suffer similar indexing problems to the getWizPnP delete operation problems that are listed in the Bugs section of the release notes, unless Beyonwiz also improves the communication between wizpnp and the wizpnp index generation process.

I use HTTP PUT in the bwhack tool in BWFWTools to allow me to remotely enable and disable hacks on the Beyonwiz by creating and deleting flag files.

download · Post by **download** » Tue Apr 28, 2009 11:08

prl wrote:I've posted about this a number of times in the past, but in summary:

Ta, I'd have been happy with no you can't upload, but this is very good too

Thanks

Peter Gillespie

prl · Post by **prl** » Tue Apr 28, 2009 22:10

craigh wrote:Hi Peter

The change you made to --device

Matching device names using --device is now a case-insensitive substring match. Can be made a prefix match if people prefer.

This has caused all my scripts to fail as below

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl
Device name '' matches more than one device:
[FamilyRoom_BW-192.168.0.153-49152, FishRoom_BW-192.168.0.152-49152] were found

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --device=FamilyRoom_BW
Device 'FamilyRoom_BW' isn't available.
[FamilyRoom_BW, FishRoom_BW] were found

have I missed something ?

No, I missed something. The case-insensitive matching for short names is broken. As a workaround use --longNames (or set it in the config file). You can still use --device=FamilyRoom_BW or --device=FishRoom_B.

This bug only affects device selection using --device. If you have only one device, and don't use --device, the bug shouldn't cause any problems.

I'll try to get a fix out ASAP. I did test this!

If you want to try to fix it yourself, in Beyonwiz/WizPnP.pm change line 417:

Code: Select all

    return grep { defined($_->shortName) && index($_->shortName, $name) >= 0 }

to:

Code: Select all

    return grep { defined($_->shortName)
                        && index(lc $_->shortName, $name) >= 0 }

And then re-install (on non-Windows).

craigh · Post by **craigh** » Wed Apr 29, 2009 07:13

prl wrote:
IanSav wrote:Hi Peter,

A few observations...

The newer names for the standard Beyonwiz folders is "Recordings", "Contents" and "Trashcan". That is, the first letters are capital letters.
That's true for the current betas, but not for the current released version. Since folder matching is case-insensitive, this is only a documentation issue anyway. I'm happy to update the documentation when the next public firmware release happens. Are the contents of the Trashcan available in index.txt? I've always assumed that they are not, but that may be incorrect, because I haven't tested it.
IanSav wrote: If anyone uses an subnet mask of anything bigger than 255.255.255.0 then only using the last byte of the IP address will not uniquely resolve the device (if the name is also left as default).

Regards,
Ian.
You appear to be assuming an algorithm from the examples. I chose the most common netmask value for the examples. The actual algorithm correctly extracts the host address from the IP address, no matter what the netmask is set to (i.e. hostAddr = ipAddr & ~netmask).

Peter

The contents of Trashcan are not in index.txt

prl · Post by **prl** » Wed Apr 29, 2009 09:09

craigh wrote:
prl wrote:... Are the contents of the Trashcan available in index.txt? I've always assumed that they are not, but that may be incorrect, because I haven't tested it.
...
The contents of Trashcan are not in index.txt

Thanks for confirming it, Craig.

craigh · Post by **craigh** » Wed Apr 29, 2009 18:20

prl wrote:
craigh wrote:...

Hi Peter

I have no issue with the files on the BW being handled that way.

Its just when I use --indir on a network drive that contains 1000+ files (but only 50 tvwiz) the list is a bit hard to handle.

Did you see my other post re root of drives as well ?
Would my suggested fix (ability to set your own media file extension list from the config file or command line) fix the problem for you? I didn't need any more justification to add the feature if it fulfills your need. On the other hand, perhaps you could make more use of folders on your media NAS?

Yes, I saw and responded to, both on the forum and in the code, your posts about the root of drives. What I missed was mentioning it in the release notes. However, I did mention in the discussion that I had coded the fix.

I'd prefer if this discussion went on in the topic in Content, Software and USB

Hi Peter

For me a simple command line option of

--media=all
or
--media=tvwiz

would do it for me (default is "all")

With regard to the root directory question I was not concerned re it being documented, more that it did not appear to be working as described in the thread. ( I still got an error)

prl · Post by **prl** » Thu Apr 30, 2009 16:08

Craigh posted in the Software Developer's topic on getWizPnP that the fix in version 0.4 that was supposed to allow the specification of whole devices in --indir (e.g. --indir=x:). He's correct. It appears that a change in the index generation code after I tested this feature undid my fix.

A workaround is to use --indir=x:\.

I have fixed the bug, but not distributed the fix yet:

Code: Select all

getWizPnP>.\getWizPnP.pl  --indir=e: --all --folder=Personal
Content: Personal/Computer/Beyonwiz/thehobbitenigma.mp4
Content: Personal/Computer/Beyonwiz/thehobbitenigma.wmv
Content: Personal/Computer/Bugs/Install from 9.1 hang/DSC00440.JPG
Content: Personal/Computer/Bugs/Install from 9.1 hang/screen.gif
Content: Personal/Computer/Bugs/Install from 9.1 hang/screen.jpg
Content: Personal/Music/dotmatrix1 control to efficiency.mp3
Content: Personal/Music/dotmatrix2 efficiency.mp3
Content: Personal/Music/dotmatrix3 new existential clause.mp3
getWizPnP>

On the weekend I hope to distribute an update version 0.4a that addresses this problem, the --device problem reported by craigh above, and his request there and in this topic to allow users to specify the file extensions that getWizPnP should recognise as media files in local PC folders (overriding the default, which should be the same extensions as recognised by the Beyonwiz itself). The option will probably allow something like --media=avi,ts and --media=default, and the specification of a user-defined default list in the configuration file.

craigh · Post by **craigh** » Thu Apr 30, 2009 17:12

Hi Peter

A bit more info from my testing on windows re the problem with recursive root directory and --indir

If the root string passed to --indir is contained in "" it fails
eg
--indir="F:\" fails
--indir=F:\ works

This works fine for any other string other than root drive.

prl · Post by **prl** » Thu Apr 30, 2009 17:33

craigh wrote:Hi Peter

A bit more info from my testing on windows re the problem with recursive root directory and --indir

If the root string passed to --indir is contained in "" it fails
eg
--indir="F:" fails
--indir=F:\ works

This works fine for any other string other than root drive.

Thanks.

It looks like it's part of the whole ungodly mess that is quoting in the DOS shell.

Please don't get me started on this.

Code: Select all

getWizPnP>.\getWizPnP.pl  --indir="e:\" --all --folder=personal
Can't find E:" --all --folder=personal: No such file or directory
Fetch of index for e:" --all --folder=personal failed

The " is being treated as an escaped " rather than a backslash followed by the closing double quote, and all the rest of the line is being treated as the argument to the --indir option. The DOS shell then apparently doesn't give a damn about the fact that the opening quote doesn't have a matching terminating quote.

There's nothing sensible that getWizPnP can do about this. Try: --indir="F:\" if you want to put the name in quotes.

Don't shoot me, I'm only the messenger.

If anyone can point me to a lucid explanation of how DOS shell quoting is supposed to work and which bits of it are handled by the shell, and which bits by the application, I'd appreciate it!

craigh · Post by **craigh** » Thu Apr 30, 2009 18:22

Getting closer Peter

With a second \ it sort of works but kicks up an error re the root and shows files in all sub directories but not the root.

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --indir="e:\\" --List --recursive

Can't opendir(E:\/System Volume Information): Invalid argument
at Beyonwiz/Recording/FileAccessor.pm line 316
DVI Capture/CAPTURE.AVI
DVI Capture/Defensive Driving 1.mpg
DVI Capture/Defensive Driving.avi
.....

in fact I get the same error with indir="e:" "e:\", e:, or e:\......possibly thats the other fix you mention re the root index.

craigh · Post by **craigh** » Thu Apr 30, 2009 18:30

prl wrote:On the weekend I hope to distribute an update version 0.4a that addresses this problem, the --device problem reported by craigh above, and his request there and in this topic to allow users to specify the file extensions that getWizPnP should recognise as media files in local PC folders (overriding the default, which should be the same extensions as recognised by the Beyonwiz itself). The option will probably allow something like --media=avi,ts and --media=default, and the specification of a user-defined default list in the configuration file.

Peter

As long as the --media allows me to choose just tvwiz I will be more than happy

DaveR · Post by **DaveR** » Fri May 01, 2009 21:19

Craig, try: --indir=e:/

DaveR · Post by **DaveR** » Fri May 01, 2009 21:24

prl wrote:If anyone can point me to a lucid explanation of how DOS shell quoting is supposed to work and which bits of it are handled by the shell, and which bits by the application, I'd appreciate it!

The problem is not DOS (or more precisely, the Windows shell). The problem is that you (we) are using perl in a Windows environment.

craigh · Post by **craigh** » Fri May 01, 2009 22:11

Dave? wrote:Craig, try: --indir=e:/

Hi Dave

No luck

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --indir=e:/ --List --recursive
Can't opendir(E:\/System Volume Information): Invalid argument
at Beyonwiz/Recording/FileAccessor.pm line 316
DVI Capture/CAPTURE.AVI
DVI Capture/Defensive Driving 1.mpg
DVI Capture/Defensive Driving.avi

DaveR · Post by **DaveR** » Fri May 01, 2009 22:39

Part (all?) of the problem seems to be that 'Recording/FileAccessor.pm' is expecting a file or folder, and not a drive.

craigh wrote:Can't opendir(E:\/System Volume Information): Invalid argument
at Beyonwiz/Recording/FileAccessor.pm line 316

After the E: is that a \/ (backslash and slash) or the capitol letter Vee? It looks like a backslash and slash, which is strange (unless it's a perl or cywgin thing?).

Also, "E:\System Volume Information" is a valid Windows system folder (for anyone with an E: drive). So the question is "Why is E:<insert any single character here> being seen as "E:\System Volume Information" !?!?!?

Craig, what's in your environment path? Does the same thing happen if you try it with a different drive letter? (like C:\\ or F:\\)

Since your passing the command to a perl script, try 'E:\' (with single quotes). Or try a pair of double quotes: ""E:""

craigh · Post by **craigh** » Sat May 02, 2009 09:06

Dave? wrote:Part (all?) of the problem seems to be that 'Recording/FileAccessor.pm' is expecting a file or folder, and not a drive.

craigh wrote:Can't opendir(E:\ /System Volume Information): Invalid argument
at Beyonwiz/Recording/FileAccessor.pm line 316
After the E: is that a \/ (backslash and slash) or the capitol letter Vee? It looks like a backslash and slash, which is strange (unless it's a perl or cywgin thing?).

Also, "E:\System Volume Information" is a valid Windows system folder (for anyone with an E: drive). So the question is "Why is E:<insert any single character here> being seen as "E:\System Volume Information" !?!?!?

Craig, what's in your environment path? Does the same thing happen if you try it with a different drive letter? (like C:\\ or F:\\)

Since your passing the command to a perl script, try 'E:\' (with single quotes). Or try a pair of double quotes: ""E:""

Hi Dave

After the E:\ that it /System Volume Information.

Path =
C:\Perl\site\bin;C:\Perl\bin;d:\downloads\beyonwiz\getwizpnp;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem

results...

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --indir=""e:"" --List --recursive

Can't find E:" --List --recursive : No such file or directory
Fetch of index for e:" --List --recursive failed

D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --indir='e:\' --List --recursive
Can't find 'e:\': No such file or directory
Fetch of index for 'e:\' failed

DaveR · Post by **DaveR** » Sat May 02, 2009 15:53

craigh wrote:D:\Downloads\Beyonwiz\getWizPnP>getwizpnp.pl --indir='e:\' --List --recursive
Can't find 'e:\': No such file or directory
Fetch of index for 'e:\' failed

That's as close as you've got to getting it working. It's no longer adding the following switches, or system folder names, to the directory path that it's looking for. But unfortunately it's actually looking for 'e:\' instead of e:\. The 'Can't find xyz: No such file or directory' seems to be coming from the Perl interpreter, as the wording is not from Peter's Perl scripts or the Windows shell.

The closest match to Can't find $path: No such file or directory that grepping found is from line 592 in Active Perl's :\Perl\lib\Cwd.pm

Code: Select all

unless (-e $path) {
 _croak("$path: No such file or directory");
 }

All other error messages that end with 'No such file or directory' start with something different.

EDIT Can't find is coming from line 143 in FileIndex.pm

DaveR · Post by **DaveR** » Sat May 02, 2009 16:07

Craig, is this a new problem specific to the latest version of getwizpnp?

I just tried it with a win32 compiled getwizpnp.exe version 0.3.4.0 with the following result (posted as returned, unedited):

Code: Select all

>getwizpnp.exe --indir=e:\ --List --recursive
Can't opendir(E:\/System Volume Information): Invalid argument
 at Beyonwiz/Recording/FileIndex.pm line 144
Movies/ABC DiG Radio Oct.7.2008_22.21 - Tue Oct  7 2008.radwiz Feb.21.2009_

Movies/Doctor Who Feb.10.2009_22.32
Movies/Doctor Who - Tue Feb 10 2009 Feb.10.2009_22.1
Movies/Freeview - Tue Nov 25 2008 Mar.1.2009_23.35
Movies/Heroes - Thu Jan  8 2009 Jan.18.2009_0.47
Movies/Heroes - Thu Jan 15 2009 Mar.13.2009_21.24
Movies/Heroes_Dec.4.2008_22.25 Feb.22.2009_2.6
Movies/NCIS - Tue Mar 10 2009 Mar.13.2009_19.34
Movies/NCIS_cut - Tue Mar 10 2009 Mar.14.2009_20.8
Movies/N_A Channel Nine - Sun Jan 11 2009 Mar.11.2009_17.45
Movies/One HD_Apr.14.2009_3.40 Apr.14.2009_18.27
Movies/Scrapheap Challenge - Thu Feb 19 2009 Feb.20.2009_1.40
Movies/South Park - Mon Feb 23 2009 Apr.18.2009_16.34
Movies/Underbelly_ A Tale Of Two Cities - Mon Feb  9 2009 Feb.27.2009_23.14
Movies/abc logos2 - Fri Nov 28 2008 Mar.12.2009_1.6
Movies/test - Tue Jul  1 2008 Mar.14.2009_15.15
Movies/Heroes/Heroes Mar.7.2009_13.20
Toppy-Progs/As It Happened Jan.10.2009_0.12
Toppy-Progs/As It Happened.tvwiz.bak Dec.6.2008_18.15
Toppy-Progs/As It Happened2 Dec.6.2008_18.18
Toppy-Progs/Prison Break Dec.6.2008_18.11
Toppy-Progs/Two And A Half Men Dec.6.2008_17.48

>

So apart from the following error message it worked perfectly.

Code: Select all

Can't opendir(E:\/System Volume Information): Invalid argument 
 at Beyonwiz/Recording/FileIndex.pm line 144

I'm not surprised it returnes an error message when trying to access 'E:\System Volume Information' because Windows won't even let admins view it's contents. It looks like all that is needed is for the code starting at line 125 in FileIndex.pm to better handle folders that it does not have permission to open. Or to make it easy, just skip any folder called System Volume Information or skip any folders with the H attribute.

craigh · Post by **craigh** » Sat May 02, 2009 19:01

Dave? wrote:Craig, is this a new problem specific to the latest version of getwizpnp?

I just tried it with a win32 compiled getwizpnp.exe version 0.3.4.0 with the following result (posted as returned, unedited):
Code: Select all
>getwizpnp.exe --indir=e:\ --List --recursive
Can't opendir(E:\/System Volume Information): Invalid argument
 at Beyonwiz/Recording/FileIndex.pm line 144
Movies/ABC DiG Radio Oct.7.2008_22.21 - Tue Oct  7 2008.radwiz Feb.21.2009_

Movies/Doctor Who Feb.10.2009_22.32
Movies/Doctor Who - Tue Feb 10 2009 Feb.10.2009_22.1
Movies/Freeview - Tue Nov 25 2008 Mar.1.2009_23.35
Movies/Heroes - Thu Jan  8 2009 Jan.18.2009_0.47
Movies/Heroes - Thu Jan 15 2009 Mar.13.2009_21.24
Movies/Heroes_Dec.4.2008_22.25 Feb.22.2009_2.6
Movies/NCIS - Tue Mar 10 2009 Mar.13.2009_19.34
Movies/NCIS_cut - Tue Mar 10 2009 Mar.14.2009_20.8
Movies/N_A Channel Nine - Sun Jan 11 2009 Mar.11.2009_17.45
Movies/One HD_Apr.14.2009_3.40 Apr.14.2009_18.27
Movies/Scrapheap Challenge - Thu Feb 19 2009 Feb.20.2009_1.40
Movies/South Park - Mon Feb 23 2009 Apr.18.2009_16.34
Movies/Underbelly_ A Tale Of Two Cities - Mon Feb  9 2009 Feb.27.2009_23.14
Movies/abc logos2 - Fri Nov 28 2008 Mar.12.2009_1.6
Movies/test - Tue Jul  1 2008 Mar.14.2009_15.15
Movies/Heroes/Heroes Mar.7.2009_13.20
Toppy-Progs/As It Happened Jan.10.2009_0.12
Toppy-Progs/As It Happened.tvwiz.bak Dec.6.2008_18.15
Toppy-Progs/As It Happened2 Dec.6.2008_18.18
Toppy-Progs/Prison Break Dec.6.2008_18.11
Toppy-Progs/Two And A Half Men Dec.6.2008_17.48

>
So apart from the following error message it worked perfectly.
Code: Select all
Can't opendir(E:\/System Volume Information): Invalid argument 
 at Beyonwiz/Recording/FileIndex.pm line 144 
I'm not surprised it returnes an error message when trying to access 'E:\System Volume Information' because Windows won't even let admins view it's contents. It looks like all that is needed is for the code starting at line 125 in FileIndex.pm to better handle folders that it does not have permission to open. Or to make it easy, just skip any folder called System Volume Information or skip any folders with the H attribute.

Hi Dave

This is not new to the latest build but Peter had tried to correct in this build so behavior is different from previous.

I think you may also find in your example that while it appeared to work it will not return any media in the root o the drive only subdirectories.

DaveR · Post by **DaveR** » Sun May 03, 2009 03:56

craigh wrote:Hi Dave

This is not new to the latest build but Peter had tried to correct in this build so behavior is different from previous.

I think you may also find in your example that while it appeared to work it will not return any media in the root o the drive only subdirectories.

Hi Craig,

It works for me. Version 0.4.0 does list tvwiz recordings and other media file types that are in the root (if you ignore the error on the first 2 lines). Version 0.3.4 only listed tvwiz recordings. They both work as expected here. Note: 0.3.4 is a compiled exe and 0.4.0 is a normal perl script.

Recursive list including root level media files:

Code: Select all

>getwizpnp.exe --indir=e:\ --List --recursive
Can't opendir(E:\/System Volume Information): Invalid argument
 at Beyonwiz/Recording/FileIndex.pm line 144
Blue Oyster Cult - Don't Fear The Reaper.mp3
NCIS - Tue Mar 10 2009 May.3.2009_2.45
test.jpg
test0000.gif 
Movies/ABC DiG Radio Oct.7.2008_22.21 - Tue Oct  7 2008.radwiz Feb.21.2009_17.44
Movies/Doctor Who Feb.10.2009_22.32
Movies/Doctor Who - Tue Feb 10 2009 Feb.10.2009_22.1
etc ...

Non-recursive list containing only root level media files:

Code: Select all

>getwizpnp.pl --indir=e:\ --List
Can't opendir(E:\/System Volume Information): Invalid argument
 at Beyonwiz/Recording/FileAccessor.pm line 316
Blue Oyster Cult - Don't Fear The Reaper.mp3
NCIS - Tue Mar 10 2009 May.3.2009_2.45
test.jpg
test0000.gif

Interestingly, if I redirect the output to a file, the error message appears onscreen and just the file list (without error message) is sent to the file.

Code: Select all

getwizpnp.pl --indir=e:\ --List > test.log
Can't opendir(E:\/System Volume Information): Invalid argument
 at Beyonwiz/Recording/FileAccessor.pm line 316

Contents of test.log:

Code: Select all

Blue Oyster Cult - Don't Fear The Reaper.mp3
NCIS - Tue Mar 10 2009 May.3.2009_2.45
test.jpg
test0000.gif

craigh · Post by **craigh** » Sun May 03, 2009 09:06

Thats interesting Dave.

Up to now I have been using it with "E:\" as my code always wraps the string that way to handle spaces and as such I got no results on root directories.

When I removed the "" it does return files but I found no root directory ones.

Turns out its was because my root files were all TS files and it must not count them as media.

It does however return some other interesting stuff like

RECYCLER/S-1-5-21-842925246-1336601894-682003330-1003/De1 Dec.3.2007_15.42

craigh · Post by **craigh** » Sun May 03, 2009 09:20

Hi Peter

Probably not worth spending a heap of time on as it can be coded around by removing the "" on ay root directories.

The exclusion of system folders would be good though

RECYCLER/S-1-5-21-842925246-1336601894-682003330-1003/De1 Dec.3.2007_15.42

Also are TS files ignored for some reason ?

craigh · Post by **craigh** » Sun May 03, 2009 09:51

Hi peter

I just noticed that if you copy a non tvwiz media file and had a flag like --Date on it will place the date after the filename suffix.

eg image.jpg becomes image.jpg 2003-12-25 10-46

prl · Post by **prl** » Sun May 03, 2009 12:06

Dave? wrote:...
Interestingly, if I redirect the output to a file, the error message appears onscreen and just the file list (without error message) is sent to the file.
...

That's because getWizPnP follows the normal Unix convention and sends "normal" output to standard output, and error output to standard error. If you redirect with '>', then only standard output is redirected to the file. Error messages aren't redirected and still appear on the screen. That's intended behaviour; usually you want to be informed of errors even if you are capturing output. Unix shells have ways of redirecting standard error, too, if you want. I think the DOS shell can, too.

I'll try to reproduce the e:\/System Volume Information error. The error message is quite likely coming from inside the perl File::Find package, and it may not be easy to work around. In any case, the warning is valid. The folder was found, but its contents apparently can't be accessed.

Missing TS files is a known bug (the 'ts' extension was left out of the list of media file extensions), already fixed in the version I'll release as soon as I have the "e:\/" problem sorted out.

I'm not gong to try to compile a list of "folders getWizPnP shouldn't look in" for all of Windows, MacOS and Linux. For various reasons, I don't put my media in the root folder on my Mac media volume. Putting media into a subfolder would probably work around the problems mentioned in Windows, too.

prl · Post by **prl** » Sun May 03, 2009 12:16

craigh wrote:Hi peter

I just noticed that if you copy a non tvwiz media file and had a flag like --Date on it will place the date after the filename suffix.

eg image.jpg becomes image.jpg 2003-12-25 10-46

It shouldn't be doing that!

prl · Post by **prl** » Sun May 03, 2009 12:22

Dave? wrote:
prl wrote:If anyone can point me to a lucid explanation of how DOS shell quoting is supposed to work and which bits of it are handled by the shell, and which bits by the application, I'd appreciate it!
The problem is not DOS (or more precisely, the Windows shell). The problem is that you (we) are using perl in a Windows environment.

No, the particular problem (getting an error with getWizPnP --indir="e:" --more --options) is definitely a Windows shell problem, because the shell isn't reporting the fact that the trailing quote is missing (the second quote having been quoted itself). Perl is being passed a single option containing the string --indir=e:" --more --options, and there's no way, in general, for it to work out what the user intended.

I note you dodged the challenge

prl · Post by **prl** » Sun May 03, 2009 12:30

Dave? wrote:...
I'm not surprised it returnes an error message when trying to access 'E:\System Volume Information' because Windows won't even let admins view it's contents. It looks like all that is needed is for the code starting at line 125 in FileIndex.pm to better handle folders that it does not have permission to open. Or to make it easy, just skip any folder called System Volume Information or skip any folders with the H attribute.

FileIndex.pm (FileAccessor.pm in version 0.4) doesn't open any folders itself, apart from looking inside .tvwiz, .radwiz and .wiz folders. Recursion through the folders is done by File::Find. It produces warning messages when it can't open a folder for reading. This is good, not bad. You want to know, usually, if you're missing anything in the --indir subtree. I don't think that some exclusion list is the best way to go. It's inherently non-portabla. As is looking at attributes of Windows-specific file systems.

prl · Post by **prl** » Sun May 03, 2009 16:44

craigh wrote:...
Probably not worth spending a heap of time on as it can be coded around by removing the "" on ay root directories.
...

I don't understand what problem is being worked around here.

craigh · Post by **craigh** » Sun May 03, 2009 17:31

prl wrote:
craigh wrote:...
Probably not worth spending a heap of time on as it can be coded around by removing the "" on ay root directories.
...
I don't understand what problem is being worked around here.

Hi Peter

Only having to not use "" to encapsulate the path when its just a root of a drive.

prl · Post by **prl** » Sun May 03, 2009 19:39

craigh wrote:
prl wrote:
craigh wrote:...
Probably not worth spending a heap of time on as it can be coded around by removing the "" on ay root directories.
...
I don't understand what problem is being worked around here.
Hi Peter

Only having to not use "" to encapsulate the path when its just a root of a drive.

As I've tried to explain, the fatal error due to using --indir="e:" has nothing to do with getWizPnP, apart from the current need to supply the backslash. In the new version, the root of a file system can be specified as --indir=e:, --indir="e:\" or --indir=e:\. --indir="e:" will continue to cause an error, and there's not much that getWizPnP code can do about it.

The warning about the System Volume Information folder and the bug that causes getWizPnP 0.4 to ignore TS files are independent issues.

prl · Post by **prl** » Sun May 03, 2009 19:40

I've reconsidered my perhaps too hard-line stand about ignoring the contents of some system folders. It won't get into the next release, though, and I'll need some sort of consensus about what folders to ignore.

On OSX, .Trashes and .Trash are obvious candidates; on Linux, lost+found, and the trash folders for the more popular filesystem browsers; on Windows, System Volume Information and RECYCLER. But what about the cache folders for browsers? What if you actually want to transfer files out of them? What else? I fear this may become a bit of a quagmire.

The new release won't be tonight. I still need to have a bit more of a look at how the filenames with e:\/ in them are being generated. I didn't expect to see that, even though it's probably innocuous.

craigh · Post by **craigh** » Sun May 03, 2009 19:55

prl wrote:I've reconsidered my perhaps too hard-line stand about ignoring the contents of some system folders. It won't get into the next release, though, and I'll need some sort of consensus about what folders to ignore.

On OSX, .Trashes and .Trash are obvious candidates; on Linux, lost+found, and the trash folders for the more popular filesystem browsers; on Windows, System Volume Information and RECYCLER. But what about the cache folders for browsers? What if you actually want to transfer files out of them? What else? I fear this may become a bit of a quagmire.

The new release won't be tonight. I still need to have a bit more of a look at how the filenames with e:\/ in them are being generated. I didn't expect to see that, even though it's probably innocuous.

Hi Peter

I think you have hit it on the mark any, sys vol/trash/recycle bin equivalents on each OS is more than enough in my view.

craigh · Post by **craigh** » Sun May 03, 2009 20:01

prl wrote:
craigh wrote:
prl wrote:
craigh wrote:...
Probably not worth spending a heap of time on as it can be coded around by removing the "" on ay root directories.
...
I don't understand what problem is being worked around here.
Hi Peter

Only having to not use "" to encapsulate the path when its just a root of a drive.
As I've tried to explain, the fatal error due to using --indir="e:" has nothing to do with getWizPnP, apart from the current need to supply the backslash. In the new version, the root of a file system can be specified as --indir=e:, --indir="e:\" or --indir=e:\. --indir="e:" will continue to cause an error, and there's not much that getWizPnP code can do about it.

The warning about the System Volume Information folder and the bug that cau
ses getWizPnP 0.4 to ignore TS files are independent issues.

Hi Peter

yes once it is broken down into the 3 issues the "" is mine as I use Windows and I have coded around causing that

The other 2 just clowded what I was seeing in results while testing it.

prl · Post by **prl** » Mon May 04, 2009 17:35

craigh wrote:...
yes once it is broken down into the 3 issues the "" is mine as I use Windows and I have coded around causing that

The other 2 just clowded what I was seeing in results while testing it.

All of these now work on my latest (not yet released) version:

Code: Select all

beyonwiz\getWizPnP>.\getWizPnP.pl --indir=e: --folder=Personal --all --list
Content: Dilbert non-standard.jpg
Content: thehobbitenigma.mp4
Content: thehobbitenigma.wmv
Content: DSC00440.JPG
Content: screen.gif
Content: screen.jpg
Content: dotmatrix1 control to efficiency.mp3
Content: dotmatrix2 efficiency.mp3
Content: dotmatrix3 new existential clause.mp3
ABC1: *Summer Heights High

beyonwiz\getWizPnP>.\getWizPnP.pl --indir=e:\ --folder=Personal --all --list
Content: Dilbert non-standard.jpg
Content: thehobbitenigma.mp4
Content: thehobbitenigma.wmv
Content: DSC00440.JPG
Content: screen.gif
Content: screen.jpg
Content: dotmatrix1 control to efficiency.mp3
Content: dotmatrix2 efficiency.mp3
Content: dotmatrix3 new existential clause.mp3
ABC1: *Summer Heights High

beyonwiz\getWizPnP>.\getWizPnP.pl --indir="e:\" --folder=Personal --all --list
Content: Dilbert non-standard.jpg
Content: thehobbitenigma.mp4
Content: thehobbitenigma.wmv
Content: DSC00440.JPG
Content: screen.gif
Content: screen.jpg
Content: dotmatrix1 control to efficiency.mp3
Content: dotmatrix2 efficiency.mp3
Content: dotmatrix3 new existential clause.mp3
ABC1: *Summer Heights High

beyonwiz\getWizPnP>

As before, this doesn't work, and for the same reason:

Code: Select all

beyonwiz\getWizPnP>.\getWizPnP.pl --indir="e:" --folder=Personal --all --list
Can't find E:" --folder=Personal --all --list: No such file or directory
Fetch of index for e:" --folder=Personal --all --list failed

beyonwiz\getWizPnP>

Due to a quirk in the way the perl File::Find module works, on Windows (and Cygwin) this also works (as do all mix & match variants with '/' or '\')!

Code: Select all

beyonwiz\getWizPnP>.\getWizPnP.pl --indir="e:/" --folder=Personal --all --list
Content: Dilbert non-standard.jpg
Content: thehobbitenigma.mp4
Content: thehobbitenigma.wmv
Content: DSC00440.JPG
Content: screen.gif
Content: screen.jpg
Content: dotmatrix1 control to efficiency.mp3
Content: dotmatrix2 efficiency.mp3
Content: dotmatrix3 new existential clause.mp3
ABC1: *Summer Heights High

beyonwiz\getWizPnP>

The opposite doesn't work on Linux or MacOS, because '\' is a legitimate file name character on those systems.

Confusingly, if File::Find generates an error or warning message, it uses '/' as the path separator on Windows.

The dotmatrix* files are recordings of [The User] symphony for dot matrix printers. Quite entertaining, especially if you've used a dot matrix printer.

prl · Post by **prl** » Mon May 04, 2009 17:41

prl wrote:...
The warning [from File::Find] about the System Volume Information folder ....

is very curious. Cygwin Perl can do an opendir on c:\System Volume Information, but ActivePerl running native on Windows cannot (returns "Invalid argument" error).

DaveR · Post by **DaveR** » Tue May 05, 2009 23:11

prl wrote:... on Windows, System Volume Information and RECYCLER. But what about the cache folders for browsers? What if you actually want to transfer files out of them?

On Windows, Recycled, RECYCLER and System Volume Information will do. I thought your previous argument of it not being a good idea to store files in a root directory was a valid reason for not creating a list folders not to look in. If someone is silly enough to try to list their Windows system drive then they deserve to get error messages from their IE cache etc.

craigh · Post by **craigh** » Wed May 06, 2009 08:04

Dave? wrote:
prl wrote:... on Windows, System Volume Information and RECYCLER. But what about the cache folders for browsers? What if you actually want to transfer files out of them?
On Windows, Recycled, RECYCLER and System Volume Information will do. I thought your previous argument of it not being a good idea to store files in a root directory was a valid reason for not creating a list folders not to look in. If someone is silly enough to try to list their Windows system drive then they deserve to get error messages from their IE cache etc.

Storing files in the root directory may not be a good idea, however choosing a drive and doing a recursive search is normal situation will arrise.

tonymy01 · Post by **tonymy01** » Wed May 06, 2009 08:22

What are you trying to recursively do on the PC filesystem? Since the beginning of this "indir" discussion, I have always wondered what the issue what about here?

prl · Post by **prl** » Wed May 06, 2009 08:28

tonymy01 wrote:What are you trying to recursively do on the PC filesystem? Since the beginning of this "indir" discussion, I have always wondered what the issue what about here?

As I understand it, Craig simply wants to dedicate an entire HDD to his media library, and be able to search the whole of the drive. You don't have to be searching over a system HDD to stumble across folders like Recycled, RECYCLER and System Volume Information.

I don't think that there's any ideal solution to the problem, but I am willing to try a "don't look in these folders" list.

tonymy01 · Post by **tonymy01** » Wed May 06, 2009 08:47

Search how? For what?

IanSav · Post by **IanSav** » Wed May 06, 2009 09:43

Hi,

I am surprised that this is such an issue. Without looking at the code I would assume that you are opening and reading the root directory. That should be fine and may include directories that you may or may not be able to access. As you list each entry you would be checking to see if it is a file or a directory. Listing of the files in the root should proceed without error, if it is a file then print its name etc and move on. If it is a folder then call yourself recursively to process the directory. If an attempt to open a directory fails then you can just silently ignore that directory or display an error message and move on. Loop this procedure until the tree has been walked. In this case it appears that the choice has been made to report the error. It seems like all people want is for unreadable directories to be ignored. If you are using the opendir, closedir, readdir, etc commands then this should not be an problem.

Hard coding directories that should or should not be read is a bad idea. Just cope with any errors you encounter as you walk the tree.

Regards,
Ian.

prl · Post by **prl** » Wed May 06, 2009 09:46

tonymy01 wrote:Search how? For what?

When it accesses local files, getWizPnP searches the --indir directory (or directory sub-tree with --all) to build the information that appears in the WizPnP index.txt file. This was the simplest means to implement local file operations within the already existing structure of getWizPnP as primarily a Beyonwiz WizPnP client.

The selection mechanisms used in getWizPnP for WizPnP operations would necessitate a search of the --indir directory (sub-tree) anyway. If I want to find all Beyonwiz recordings and media files in a media archive whose titles contain "Underbelly", how else is it to work?

tonymy01 · Post by **tonymy01** » Wed May 06, 2009 10:37

Ahh, thanks for that. I guess my confusion was that this was intended to list and grab files from a Wiz, and dump them somewhere locally, so I couldn't understand why the search discussion etc came into it. Perhaps Dave/Craig should code their side of the tool (windows gui) to do some of the local hard work, to simply tell getwizPNP "here is a tvwiz file, now give me the goss on it", rather than leave all the local filesystem grunt work to getwizPNP which is really going to fail miserably in a multi-operating system environment with access permissions, folders you should never be traversing etc.

prl · Post by **prl** » Wed May 06, 2009 12:42

tonymy01 wrote:... rather than leave all the local filesystem grunt work to getwizPNP which is really going to fail miserably in a multi-operating system environment with access permissions, folders you should never be traversing etc.

I don't think that it's a bug in getWizPnP that it prints a warning when it isn't able to scan a folder within the subtree that you've told it to scan. On the contrary, I think it's a bug not to give a warning. However, it wouldn't be difficult to portably code getWizPnP to silently avoid trying to scan any folders that it's not allowed to scan. I just happen to think that that's the wrong solution.

On the other hand, there seems agreement that it shouldn't be looking in folders like Recycled & RECYCLER on Windows, or in folders like lost+found, .Trash or .Trashes on OS X or Linux. There's an argument that it shouldn't be looking at any files or folders whose names start with '.' on Mac OS or Linux. It possibly shouldn't be trying to look inside System Volume Information on Windows, since it's not allowed to look inside anyway, and it's not going to contain anything interesting.

In any case, if you decide to run getWizPnP --indir=c:, you're going to get a lot of warning messages, and it's going to take a long time.

A mechanism actually already exists (--BWName) that could be used by WizZilla to nominate individual local recordings and media files whose paths are known, without having to do the search.

If I do implement a "don't look in these folders" list, it will be configurable from the getWizPnP configuration file for sophisticated users who don't like my choices. Any such list will have some problems anyway, because some of these folders should only be ignored when they appear in a particular context (such as in the top-level folder in a file system). That sort of context is difficult to detect portably.

craigh · Post by **craigh** » Wed May 06, 2009 18:38

prl wrote:
tonymy01 wrote:Search how? For what?
When it accesses local files, getWizPnP searches the --indir directory (or directory sub-tree with --all) to build the information that appears in the WizPnP index.txt file. This was the simplest means to implement local file operations within the already existing structure of getWizPnP as primarily a Beyonwiz WizPnP client.

The selection mechanisms used in getWizPnP for WizPnP operations would necessitate a search of the --indir directory (sub-tree) anyway. If I want to find all Beyonwiz recordings and media files in a media archive whose titles contain "Underbelly", how else is it to work?

Tony

In simple terms I have a couple of NAS's as well as storage on a local machine that contain various media.

Some of that media is tvwiz files other is TS, avi etc etc.

My programme currently searches your network for BW's then builds a single list of all media on them that you may want to copy off as TVwiz or TS using Peter's getWizPNP behind the scenes.

It then cleans any TS streams with videoredo before finishing.

I also allow you to choose any local drive or network resource and see what is stored there to also process.

So if your selective in my programme and choose a specific directory we have no issue but if you just choose a root of a local drive we had some issues.

Some turned out to be Peters some mine some Windows.

They can be coded around and in fact I now do that but Peter is trying to neaten up the returned list so you dont see deleted files etc.

The issue of should you store media in the root is in some ways irrelevant as even if you dont you will see any files in the system folders.