WizPNP server for (possibly) most platforms.....

tonymy01 · Post by **tonymy01** » Mon Jun 01, 2009 22:57

Guys/Gals,
You know how I have been bugging the forum sometimes about when someone would pull their finger out and make a WizPNP server to "simulate" a Beyonwiz so the Wiz could stream using PNP rather than silly windows networking??
Well.... if you want something done, do it yourself as the saying always goes.
I have just reached a breakthrough, and can now make my slug (NSLU2) show up in my Wiz menu system under "Network" and it happily streams recordings!
Edit:
Now can do a whole lot more than that, the streaming of recordings and media is possible:
If you don't want to read the ins/outs of the development process, get a download here:
README:
http://tonyspage.abock.de/beyonwiz/wizm ... README.txt
FILES:
http://tonyspage.abock.de/beyonwiz/wizmongoosev2.zip
This includes the source, a PC .exe binary you can run if you have a PC (copy the cygwin1.dll into the same directory as the exe file) and an NSLU2 binary to run this on a SLUG (no need for the cygwin1.dll).
The source can be compiled for Macs & other linux distros very easily with the included Makefile.

Technobabble below:
Ok, so my socket and threading C capabilities are about .... zip, and at first when I looked at PRL/Peters getWizPnP I didn't follow what the heck was going on... so I spent about 5 half days doing a *tonne* of googling and trying to find the perfect solution without a huge amount of coding effort from scratch, and have modded a *severely* basic http webserver called "mongoose" to do the PNP stuff also.

First challenge was to get WizFX working with it... wow, what a pain. While I happily got PRL/Peters getWizPnP working with it early on (well, late Saturday), I couldn't get WizFX and the Wiz to do much at all.
Turns out the solution was easier than I thought... WizFX (looking at the pascal source code for it today... grrrr....) is looking for windows CR/LF line terminators on everything. Fixed that this afternoon, and bingo, WizFX sees my slug like another Wiz, and downloads etc.
I still had the issue of the actual Wiz not seeing my SLUG. Anyway, was about to do the f/w upgrade, and thought "lets do a final check in the fileplayer" and what do you know... the SLUG is there in the Network menu, under the name I gave it, with a sexy different icon. One thing I did notice was the Wiz expects the Contents directory to be "contents" but the index.txt file has it named "Contents" so have to work out what is happening there. Similarly the "Recording" directory that we all know about is actually listed as "recording" in the WizPnP client interface on the Wiz, but this time will display and stream the recordings, albeit with "blah Mmm.DD.YYYY_HH.MM.tvwiz" names. Is this what they look like on the H1? As I don't have 2 wizes here at home, I am only going by the protocol document provided with WizFx and also the polling that I can see coming from the Wiz on the SSDP port/ip address on a regular basis, and what I can see in WizFX and the Wiz itself (with the "client" enabled).

Things to do before final release:
Actually make the application index a folder rather than downloading a few tvwiz files and the index.txt file off the Wiz and making a fake directory structure with Recordings and Contents folders..... DONE.
i.e. make it actually work with a folder on a "server" rather than mimick a Beyonwiz "Recordings/Contents" folder structure. DONE.
The thing I freaked out about was that the Wiz might use some obscure PNP streaming stuff, but as far as I can tell, it uses http get/head/etc as per how WizFX works... phew... thus thankfully using a *very* basic webserver to do all those bits of it already exists in the site provided...

I did find some other web servers that were pretty basic, but still had their source code scattered amongst 8 to 10 .c and .h files which is impossible for a newbie to socket programming and threading to follow... thankfully mongoose is so simple it is only 3 files, main.c, mongoose.c mongoose.h.... so simple in fact that I compile it on my 32meg slug (i.e not using a cross compiler). Luckily with a few small examples of UDP multicasting on the web, combined with the WizFX protocol notes, combined with the great work PRL/Peter has done for getWizPnP combined with the source of WizFx has finally netted a solution that is almost there....

Stay tuned, but I reckon a day or two away..... sorry for jumping the gun and posting on the forum before actually providing a solution, but I was very excited to see something other than a Windows share listed in the "Network" file list finally, and had to post here.
DONE.

edit:
aha... I thought the index.txt file from the Wiz for my Wiz recordings/contents had a corrupted line, turns out that the "<tab>idehdd/contents<cr>" line before your contents stuff is not a corruption and is necessary for the Wiz to list the contents folder (but not WizFX which is happy with the index.txt file without that line).
Cheers

Half Round · Post by **Half Round** » Tue Jun 02, 2009 00:33

tonymy01 wrote:Guys/Gals,
....
Stay tuned, but I reckon a day or two away..... sorry for jumping the gun and posting on the forum before actually providing a solution, but I was very excited to see something other than a Windows share listed in the "Network" file list finally, and had to post here.

Thats fantastic,

(Much more interesting reading than the blame game 'discussions' over the S1 DVD). I can see I have a lot to learn about networking /streaming.

efry · Post by **efry** » Tue Jun 02, 2009 07:43

Hey Tony,

Nice one. I'm looking forward to trying it out on my QNAP TS-109 II

Regards,
Eric

download · Post by **download** » Tue Jun 02, 2009 10:00

efry wrote:Hey Tony, Nice one. I'm looking forward to trying it out on my QNAP TS-109 II Regards, Eric

An impressive effort indeed. I'm actually in the market for a NAS at the moment and was considering the original QNAP TS-109. If I understand correctly I could install Tony's server onto it (?) and stream content direct over the network from it.

Currently I'm finding streaming episode length MKVs over USB to be pretty shaky. WOuld the network stream likely be more reliable, or Tony could this server also provide a way to transfer files from the NAS to the BW CONTENTS?

Regards

Peter Gillespie

PS Would the extra performance of the II model be of much use in my 2 PC home?

prl · Post by **prl** » Tue Jun 02, 2009 10:11

Half Round wrote:
tonymy01 wrote:Guys/Gals,
....
Stay tuned, but I reckon a day or two away..... sorry for jumping the gun and posting on the forum before actually providing a solution, but I was very excited to see something other than a Windows share listed in the "Network" file list finally, and had to post here.

Thats fantastic, (Much more interesting reading than the blame game 'discussions' over the S1 DVD). I can see I have a lot to learn about networking /streaming.

Well done, Tony. I believe that you are correct that the only part of UPnP that WizPnP uses is SSDP. I think everything else is done using HTTP.

If you have any questions about getWizPnP, please feel free to PM.

em5500 · Post by **em5500** » Tue Jun 02, 2009 22:29

tonymy01 wrote:You know how I have been bugging the forum sometimes about when someone would pull their finger out and make a WizPNP server to "simulate" a Beyonwiz so the Wiz could stream using PNP rather than silly windows networking??
Well.... if you want something done, do it yourself as the saying always goes.
I have just reached a breakthrough, and can now make my slug (NSLU2) show up in my Wiz menu system under "Network" and it happily streams recordings!

Good work Tony. Is the goal here to simplify streaming by removing the need to setup network accounts / passwords / shares?

I can see it now: Cross-platform, portable app versions of your code on USB memory sticks - just plug one in and wait for it to autorun to turn any networked PC into a WizPNP server!

tonymy01 · Post by **tonymy01** » Thu Jun 04, 2009 00:15

Any C gurus here?
The mongoose webserver fails to "stat" anything bigger than 2G (actually some things less than this seems to also be a problem, such as filenames with spaces in them... well, I think this is the problem, one step at a time here).
So trying to work out how to solve this little "got-ya" as I try to apply the server to index a directory full of avi, ts & tvwiz files.... (initial tests were with a few avi files and a few little wiz recordings... actually, the wiz recording side of it is fine, as we know they are only small files and so stat works fine there).
I would have thought that stat64() would be used rather than stat for most environments based on possibly what is going on in the #defines/#ifdef's in sys/stat.h... but maybe not

The Slug certainly supports 64bit stat's as, e.g, the ls -al gives full file sizes etc without a drama.
I can see the author of mongoose has an ifdef win32 and rewrites the routine that calls stat (mg_stat) by making all the structures surrounding it 64bit support also.... perhaps I have to do something similar.

peteru · Post by **peteru** » Thu Jun 04, 2009 02:39

Try something along the lines of:

Code: Select all

#define _LARGEFILE64_SOURCE
#define _FILE_OFFSET_BITS=64

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <utime.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <sys/file.h>
#include <unistd.h>
#include <fcntl.h>
#include <asm/byteorder.h>

int function(char *srcPath)
{
  int result = -EPROTO;
  int src = -1;
  struct stat64 srcStat;
  __u64 fileSize;

  src = open64(srcPath, O_RDONLY);
    if(src < 0)
    {
        fprintf(stderr, "ERROR: Can not open source file: %s\n",
                strerror(errno));
        return errno;
    }

    if(0 != fstat64(src, &srcStat))
    {
        fprintf(stderr, "ERROR: Can not examine source file: %s\n",
                strerror(errno));
        result = errno;
        goto out;
    }

    fileSize = srcStat.st_size;
    if(fileSize == 0)
    {
        fprintf(stderr, "ERROR: Source file is empty - not transferring.\n");
        result = -ENODATA;
        goto out;
    }


/*
 *
 * Do whatever with the file...
 *
 *
 */


  out:
    close(src);
    return result;
}

You probably don't need all those includes - just experiment with what's needed.

tonymy01 · Post by **tonymy01** » Thu Jun 04, 2009 08:22

Thanks Peter. I woke up after fiddling last night and was about to post that my next prob was when I made the routine that calls stat always return an ok value rather than the stat response (and actually still datafill the size/isdirectory/mtime with as much as I could), I hit a similar obstacle with "fopen":

Code: Select all

Error 500: Internal Server Error
fopen(/share/flash/data/beyonwiz/LOTR-FellowshipOfTheRing(ExtEd)HD1080PXvidAC3.avi): File too large

.. but it appears that your example accomodates that also. Ok, will give your ideas a shot (I did actually put #define __USE_FILE_OFFSET64 1
#define __USE_LARGEFILE64 1 last night and these didn't appear to do much, but I didn't call the 64 versions of the routines as I thought that defining the 64 would be enough and the sys/stat.h would work around my calls of stat with those #defines...). Actually your example is subtly different to mine anyway..

tonymy01 · Post by **tonymy01** » Mon Jun 15, 2009 12:07

Ok, I have something that is suitable finally.
http://tonyspage.abock.de/beyonwiz/wizmongoose.zip

What I have provided:
The source code of what I worked on and everything I didn't mod in the original mongoose distro for "mongoose-2.6.tgz", from here: http://code.google.com/p/mongoose/ and an executable that runs on a PC (along with the cygwin1.dll required to sit with the executable if you don't run a suitable cygwin environment on your PC). I can provide the SLUG binary also if any of you want to give it a go on the SLUG.

I have not spoken to the author of the original yet about this, so for the time being this is by no means an official release.

How to run it:
Put the mongoose.exe, cygwin1.dll, tvpictureSCPD.xml, tvdevicedesc.xml & mongoose.conf into the directory you want to share out to make your PC look like a Beyonwiz.
Edit the mongoose.conf file to put your PC IP address into the "ports" section (i.e. change the 192.168.0.120 into your PC ip address). Leave the 1900 right there, and the :49152 right there. If "C" could determine your PC IP address with 100% certainty I could have coded it to not worry about putting your PC IP address into the config file, but I don't believe "C" can reliably tell what your PC IP address is, so this is unfortunately what you need to do.
Remove the line "root /cygdrive/c/storedtoppycaptures" or edit the 2nd part of the line to have the directory which you want to share (line not required if you put mongoose.exe into the directory you want to share).
Edit the Wiz friendly name in tvdevicedesc.xml to be the name you want your PC based Beyonwiz to show up as.
Run the mongoose.exe and check the error messages etc it spits out. I am sorry it spits out everything it does at the moment, I put fprintf(stderr, for nearly everything it does while debugging it.... I guess a final release will make it much less "noisy".

That should be it! You should start seeing it send out "ssdp:alive" messages once a minute, with "send() returned 360" (the number will change a little depending on the size of the Wiz friendly/NICK name you setup for it). If you see "send() returned 0" or "send() returned -1" then this means the multicast UDP broadcasts are failing for some reason. I had lots of fun and games getting this to work in both cygwin and on my SLUG... I believe I got it working for both environments now.

When you enable the Beyonwiz WizPNP client, the Wiz will send a couple of broadcast "SEARCH" SSDP requests, and mongoose should respond with ssdp:alive's back to the Beyonwiz after each search. When the Beyonwiz is happy it got a response, it requests the index.txt file using standard web requests from the server, and this is (mostly) where the stock mongoose stuff kicks in.
The Wiz also starts regularly sending SEARCH with "WANT:blah" and blah matches the details of each WizPNP server on the network that the Beyonwiz client knows about. I faked an ID here simply by turning the IP address of the PC into a hex number (and didn't even fix the endedness of it... it only needs to be a unique 8 digit HEX number from what I can tell). I guess this is to keep tabs on the server. When the server goes to standby (i.e. a Wiz with WizPNP server enabled), it sends a ssdp:byebye, and so I got mongoose to do the same when you type "q" then carriage return. When the Beyonwiz client sees these bybye messages, it no longer continually polls the "WANT" messages. Mongoose did used to quit well by typing ctrl-c on the SLUG, but cygwin just kills the exe immediately when you do this, so that is why I check for stdinput q now....

The mongoose.conf file also contains the line "wizfx 1". This tells the indexing part of the application to not index any filenames that wizfx doesn't like. It apears that although the Wiz itself is quite happy with characters >ASCII 127 (e.g umlauts and asian characters etc), wizFX isn't, and won't even display anything if it spots any chars like this in the index.txt file. If you remove the line, or have wizfx 0, then this will index filenames with full 8bit ascii and the beyonwiz itself seems ok with these.

Mongoose will create the index.txt file only on startup. I haven't made it regularly create this file, but shouldn't be too hard to add. It creates the index.txt by recursively scanning the root folder that you setup in the mongoose.conf or the folder you run mongoose from if you don't specify a root.

I have also made mongoose do one more trick... it turns ".rec" files into ".ts" filenames in the index.txt file, so the Beyonwiz can see them. When the beyonwiz goes to download them, mongoose, if it fails to find the file, rather than reporting a 404 File Not found, will try going thru the extension mapping tree to see if it can find a file of similar name, and try that instead. So this means Toppy owners need not rename the .rec files into .ts any more. I only added this feature late last night, so haven't tested it much yet....

Finally.... phew... ok, this thing seems to have some issues with threading, and I believe it is the native mongoose code that I didn't touch that has this issue. While it seems quite fine with streaming native tvwiz stuff to the wiz (possibly because of the small file sizes), anything larger seems to give some problems. I don't think the threads are properly terminating when, say, you cancel a download with WizFX or you hit "stop" on the Beyonwiz (or it reports a "unsupported format" message when trying to play). I think the mongoose server is attempting to stream the whole file in the background still, so when you go to keep playing files, it gets slower and slower.... and takes many minutes to quit with the q enter key as it closes down all the threads. I can see both from WizFX and from a web browser and from the Beyonwiz they send a "GET" without any argument which *may* be to close down/cancel the download, dunno yet.
The Beyonwiz sends a number of heads & gets to determine aspects of the file it is going to play which doesn't help at all either...

Oh, two more final things... you get a lot of "unsupported format" on media files. Especially AVI files. The Wiz seems to do a GET for bytes 0-0, which means only get the 1st byte of the file. For MPGs the Wiz does a HEAD (to find out the filesize) and a get of the last 260kbytes or so of the file (perhaps to get further file info) and then starts playing. So if someone has two wizes hooked up and a TCP analyser, they may be able to help me here on exactly how the Wiz plays AVI files over WizPNP/http? MKVs give similar probs, as do a lot of media files. I don't believe the bytes range of 0-0 is a legitimate request anyway...

WizFX is only coded with 32bit numbers, and not unsigned at that. So single files over 2gigabytes in size are shown as "-1byte" in WizFX for the filesize. If you try to download the file with WizFX, it only downloads 2gigabytes of the file. I haven't played a file>2G thru on the Beyonwiz yet to see if it has the same limitation. If you want to double check this works ok on your PC, you can see a full index in your web browser by typing http://ipaddressofyourpc:49152 and the mongoose web server will serve out your root specified directory to you, showing directories and files and filesizes. You can even check the index.txt file that is created from the webbrowser like this... the standard mongoose webserver doesn't use this file, only the Beyonwiz and WizFX use the file, so it doesn't impact how the web browser supports displaying your files.

I think I have covered everything here?
Cheers

tonymy01 · Post by **tonymy01** » Mon Jun 15, 2009 12:19

Argh... a few more more points.
I tested this only with:
make linux (for the slug)
make mingw (for cygwin).

I guess I should test with "make linux" for Kubuntu, but when I was testing this in the early days two weeks ago, my PC PNP environment was stopping me to be able to bind to port 1900, but I found out later I needed to get the socket setup as re-use and never went back to verify that the fixes I added worked for Kubuntu after that.

The "rec" into "ts" will not work with "make windows" yet I wouldn't think, as I haven't modded the mg_stat routine that I believe is called if you make this in a native windows environment rather than in a cygwin windows environment. Because I don't have a native windows compiler (and don't really want one!), I decided not to fiddle with this side of things because I really don't know how my changes will influence it.

I also haven't sorted the index.txt file in any kind of way (apart from wiz recordings at the top, media files at the bottom). I also haven't supported radwiz recordings yet (I should record a couple so I can see what they look like). The wiz files show being in a "recording" directory, and the media files show being in a "contents" directory, despite your Wiz server not needing these directories to be set up. This is just the way the Beyonwiz client works.
The Beyonwiz client itself happily sorts the Wiz recordings either chronologically or by filename (using the date part added to the index.txt file), and I think it takes the index.txt sorting for media files to be the chronological sort order, but does sort the files by name OK in the WIz display.
Oh, and I went one better than a stock wiz, I put the program title and sub-title into the index.txt stored filename for native wiz recordings

Regards

tonymy01 · Post by **tonymy01** » Mon Jun 15, 2009 12:27

Finally...
Thanks to Peter/PRL for his efforts to get information on the openwiz.org website, which I used to, for example, get info out of the header.tvwiz file (eg I used PRLs MJD to localtime suggested example, I used PRLs getWizPNP "supported wiz extensions").
Thanks to Peter/Peteru for helping explain a few things about 64bit support that helped put me in the right direction also...
Thanks to google for helping me learn about sockets, about broadcast protocols, about multicasting, about SSDP protocol, about nearly everything, and Kernighan & Ritchie for your well worn "C Programming Language" book which I treasure

Regards

IanSav · Post by **IanSav** » Mon Jun 15, 2009 20:08

Hi Tony,

If you want Beyonwiz to address any of the WizFX issues you found then please append points and observations to my WizFX bugs thread in the beta area.

Regards,
Ian.

tonymy01 · Post by **tonymy01** » Mon Jun 15, 2009 20:22

The issues I found may not be encountered by interrogating a Wiz, it is more a "ooh, it behaves like this when a file can't be accessed at the time" (e.g when creating an index.txt file I did the ".rec to .ts" naming, but not the other changes to the webserver, and so when wizfx asked for "toppyfile.ts" and the webserver went "nope, that isn't here", wizfx essentially showed the name in the list with a "server hand" and all entries below that the same, and it hangs on close.... anyway, this kind of issue might explain the "hang on close" problems others have talking to a real wiz.

Playing one of my 3Gig Toppy TS files (testing the ts to rec smart auto-renaming tonight... it works... kind of), I found the Wiz only thinks the 1hour.10min files are 35mins, which appears that both WizFX and the Wiz itself are only coded to support <2G filesizes over WizPNP. This again is not a problem as Beyonwiz ensured files served from the Wiz are <2G in size...

Finally, the umlaut issue with fonts in files served via Mongoose. The index.txt file is happy to have these in, and the Beyonwiz is happy to display the files and stream the files, but WizFX won't have a bar of it and displays NO files. But I am not sure a Beyonwiz will ever get into a situation of having a file with an umlaut on it's HDD. I tried renaming a file on the Wiz HDD with an umlaut (the letter ? in particular), and the wiz filesystem changed it to a "v". I see the Wiz, when indexing files, also gets rid of colons and "*" etc, but is happy to show my WizPNP server files with ":*"etc in the filenames, so I decided that since you aren't going to use WizFX to download from my WizPNP server, there isn't much point in doing the same with the mongoose PNP server, and so I kept the names listed for the files with all the non-windows friendly chars (well, in particular, tvwiz recordings, given the name is actually not the filename but imbedded in the header.tvwiz file). I actually improved on what you see streaming between two Wizes, by putting the program subtitle name into the filename so you see prog_subtitle-date in the listings.
The only thing I haven't tested is how wizfx copes with umlauts in the tvwiz section of the index.txt file, but I would imagine that given Beyonwiz turns everything nasty to "_" in there, it also turns files recorded with umlauts in the program title into the same char.
Regards

craigh · Post by **craigh** » Mon Jun 15, 2009 20:23

Hi Tony

I have done a quick test on a couple of machines. (xp) Worked fine on one and i know why the other one did not work - firewall.

When a sub directory was chosen it seemed to work as well.

I found it interesting how it placed tvwiz in recordings and ts etc in contents. Meant I saw the same directories twice which confused me to start.

Played a number of files fine.

Seems to have an issue with setting the root of a drive for the data (builds index and at some stage shuts down)

Overall very impressive.

Can you look to allow multiple data sources and user configurable server names ?

tonymy01 · Post by **tonymy01** » Mon Jun 15, 2009 20:36

The server name is configurable, simply edit the tvdevicedesc.xml file (friendly name section).
Aha, yes, I didn't think of that about the directories. Not much that can be done there, as the Wiz expects to find files in these folders, and expects tvwiz in recordings, and media in contents. So if you have them spread around a bit on your HDD on the "shared" folder, they will "divide" up into the 2 streams in the Wiz display.
As I said earlier, it is a work in progress still, but I have done most of the cludging to get the SSDP stuff in. Trying to get the server to behave when doing media file streaming is the next issue, it slowly seems to "eat" threads streaming multiple versions of each file to your TCP port and making the CPU go up and up. If you only do one thing, or play only tvwiz recordings, it seems to work quite well with no major probs. It is only when the wiz does its "lets open the file 3 different ways and lets try downloading now" that the mongoose side seems to struggle, as I don't believe there is a way with HTTP to cancel a "get" request. The server (well, according to googling about cancelling "gets") is meant to have the grunt to deal with as many "get" requests as you throw at it (within the limit of whatever you hard set in the server), which means ..because these are all big files we are talking about here.. the HDD gets a real thrashing and the CPU starts going up and up. If you wait 10 or 15mins, all the threads slowly start closing as the "file transfer to no-where" finishes and the server goes back to playing nicely again.
I will check why it crashes with whole drives, I can imagine it is running out of buffer space for filenames as I specified a limit to the string size for a "filename" and that unfortunately includes the full path in that name too! What I didn't do was test to see whether the filename length was exceeded when trying to do some string stuff, so will try to make that bit more robust I think.
In the meantime, just stick with doing it in smaller directory trees, not your whole huge HDD (it could take a LOOONG time to index your whole HDD anyway).

Multiple data sources is probably something that can be done relatively easily, but I will have to check if mongoose supports serving multiple "roots", if not, it may be a tad difficult (the indexing bit I pretty much wrote myself I could imagine could do it reasonably easily... just the webserving bit is the bit that may require lots of changes to make it work).
Regards

prl · Post by **prl** » Tue Jun 16, 2009 09:34

tonymy01 wrote:...
... But I am not sure a Beyonwiz will ever get into a situation of having a file with an umlaut on it's HDD. I tried renaming a file on the Wiz HDD with an umlaut (the letter ? in particular), and the wiz filesystem changed it to a "v". I see the Wiz, when indexing files, also gets rid of colons and "*" etc, but is happy to show my WizPNP server files with ":*"etc in the filenames, so I decided that since you aren't going to use WizFX to download from my WizPNP server, there isn't much point in doing the same with the mongoose PNP server, and so I kept the names listed for the files with all the non-windows friendly chars (well, in particular, tvwiz recordings, given the name is actually not the filename but imbedded in the header.tvwiz file). I actually improved on what you see streaming between two Wizes, by putting the program subtitle name into the filename so you see prog_subtitle-date in the listings.
The only thing I haven't tested is how wizfx copes with umlauts in the tvwiz section of the index.txt file, but I would imagine that given Beyonwiz turns everything nasty to "_" in there, it also turns files recorded with umlauts in the program title into the same char.
Regards

I don't think there's a way to get an umlaut into the name of any recording file in the Recording folder on the Beyonwiz (apart, possibly, from using telnet). When it makes a recording, the recording folder name is composed from the service name, date and time, and two apparently random (16-bit?) numbers. When you rename a recording, the new name is used as part of the folder name, but there's no way to enter characters with diacriticals, like umlauts and other accents, in the input window. The files inside the recording folder have names that are strictly 7-bit printable ASCII, so again, no characters with diacriticals.

The part of the index.txt to the left of the '|' contains a sanitised version of the recording title, but I don't know whether it ever contains any characters that aren't in the printable 7-bit ASCII set (' '..'~'). The part of the line to the right of the '|' shouldn't contain any characters with diacritical marks for the same reasons as I detailed for the folder name - there's no way I know that those characters can get to be part of the recording folder name.

The titles of Beyonwiz recordings may have characters with diacriticals, so these may be an issue for any file transfer program uses the title to name the folder or TS file when it copies the recording to a host file system. However, I can't think of any reason why there would be diacriticals in the names of any of the folders or files that make up any Beyonwiz recordings in the Recordings folder (except via telnet or a hack).

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 10:01

Hi Peter, thanks.
As I said, the Wiz over WizPNP is completely happy with umlauts in that index.txt file. Of course tvwiz files aren't going to be named with these chars (well, not the raw files on the HDD), but the left hand part of the "|" can be anything really (for tvwiz files, NOT for contents files), this is just what other Wiz's use to put in their display about what is on the server. As such, I put the name_subtitle_date.tvwiz. The date is actually necessary, as the Wiz uses that date to sort the titles in cronological order over WizPNP.
It is more about contents files... these could have umlauts (and proven by running this mongoose server and the Wiz is happy both with what is in the index.txt file, and parses the correct http request to the webserver to obtain that file), but WizFX is not happy with it, and I tried to rename a media file in the Wiz contents folder with an umlaut via telnet (well, the serial console, same diff) and the Wiz changed it to a "v". So it appears the sanitising is done by the Wiz for contents files by the filesystem anyway, and probably due to the "they only half support international characterisation".

Can you see if this compiles for the mac, and see if it works? Then you can have a fiddle and see what I have achieved also, and will give an option for Mac people also (instead of using Windows networking). It will give you a simple test environment for your getWizPNP too

.

Regards

prl · Post by **prl** » Tue Jun 16, 2009 10:10

tonymy01 wrote:...
Can you see if this compiles for the mac, and see if it works? Then you can have a fiddle and see what I have achieved also, and will give an option for Mac people also (instead of using Windows networking). It will give you a simple test environment for your getWizPNP too .
...

I'll try it out. However, the gold standard for "getWizPnP works" is against the Beyonwiz server

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 10:34

But given you are a "two wiz owner" you can give me some hints about inconsistencies in the way the two wizes talk to each other and display lists vs what the wizmongoose shows. I have found for example that mpg, ts etc files often don't have a duration time when playing (well, bringing up the progress bar), dunno if this is just the way the Wiz streams over wizPNP or something else.
Something that I was churning over in my head last night while trying to get to sleep.... this mongoose http server is exactly that, when a "get" request is sent, it just fires off the file at webserving pace. How does the Wiz client deal with this, does it buffer the data from the server and play it from a buffer, or can it somehow throttle the TCP connection speed? And skipping is the next prob. Skipping tvwiz files works fine over WizPNP, as it just needs to call on one of those little 32meg files, but skipping fails to work on media files, because the Wiz is not constantly asking for "Range bytes=%llu-%llu"... it just does a get, which means it wants the whole file (it does do a range to determine some file params at the beginning of the determination about the file). I would have thought it would do multiple 32meg ranges of the media file, makes a lot more sense (and then the mongoose server won't cough and die when you start and stop playing lots of large 1G media files before they finish streaming completely!).

Regards

IanSav · Post by **IanSav** » Tue Jun 16, 2009 10:41

Hi Tony,

Does the Beyonwiz close the data socket when you press stop during file play. If the socket is closed then mongoose should see that it has lost its connection with the client and stop sending the data.

As for data flow control, I would have though that the TCP/IP protocol with packets being acknowledged/throttled/etc would provide the synchronisation that you are questioning.

Regards,
Ian.

prl · Post by **prl** » Tue Jun 16, 2009 10:45

tonymy01 wrote:... when a "get" request is sent, it just fires off the file at webserving pace. How does the Wiz client deal with this, does it buffer the data from the server and play it from a buffer, or can it somehow throttle the TCP connection speed? And skipping is the next prob. ...

The TCP protocol has windowed acknowledgment and flow control. The server can't independently "fire off a file at webserving speed". The client side driver code is limited by whether the client side has space in the window (and free kernel buffers to receive the data). The protocol tries to make sure that the server can't send data faster than that limit. The client will probably also want to do user-space buffering as well, because the window size for TCP may not be large enough to ensure the throughput continuity that a streaming client needs. That's how I'd do it, anyway

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 11:24

IanSav wrote:Does the Beyonwiz close the data socket when you press stop during file play. If the socket is closed then mongoose should see that it has lost its connection with the client and stop sending the data.

I think it might. When the mongoose server starts getting bogged down with the multiple files streaming/taking grunt, I see the many sockets slowly tearing down over time, but the data in them is invalid. I am guessing mongoose doesn't properly cope with these closed sockets, and waits till it thinks it finished streaming the full "get" request before seeing that it should have closed the socket down... maybe. I am only a newbie with socket programming, the whole FD_SET/FD_ZERO/select() thing is foreign to me, and only just know enough now from one or two google sessions to be dangerous

This is an example of the socket info closing down:

Code: Select all

pulled from socket:21 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:23 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:29 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:27 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:25 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:35 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:31 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:33 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
pulled from socket:41 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:39 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:51 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:47 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:49 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:43 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:45 fd:-1 buf:[rey's Anatomy - Wish You Were Here.ts]len:8192 nread:0
pulled from socket:7 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:19 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:11 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:9 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:17 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:15 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:13 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:37 fd:-1 buf:[rey's Anatomy - All by myself.ts]len:8192 nread:0
 ...finish done.

The nread normally has valid numbers (is the number read from a real request to the server). So I am guessing it getting "0" is that the socket didn't get anything new in it from the server. There is a socket timeout in mongoose, but I am thinking that it doesn't start to look at this timeout until it thinks it has done the whole streaming. The "finish done" is because some minutes before that, I tried to gracefully quit the app (as it was using all my poor slug CPU and chugging the slug USB connected HDD to a halt). The string in "buf" is what is supposedly read from the socket. The mechanism for determining anything about sockets is the select() option. I put printfs in many of the send and receive routines to try to understand the protocol a bit more. This also implies over 50sockets were opened in that session (I was testing it for about 20mins trying to play Toppy "rec" files to see what limitations it had with files greater than 2G in size), as the socket number tends to start from about 4 or 5 when you launch the app and it first opens listening sockets (in this case, one on 1900, and one on 49152).
Now I have done all the hard work to get the PNP & indexing side done, I can probably start to dig into the mongoose stuff a bit more and see if I can come up with some ideas.

So those considering using this... it works very well for small files and tvwiz (native Beyonwiz recordings), but will struggle at the moment with large files, esp when you are opening and closing many files.

I am guessing since the Wiz seems to not like (well, not 100% compat) anything over 2G in size over wizPNP I am probably going to have to "falsify" a .wiz directory structure to fake out that the AVI file is less than 2G to the beyonwiz. Given now I can "falsify" a ".rec" file to look like a ".ts" file (in filename only), I don't think it would be too hard to do similar with files greater than 2G.... wish me luck

Regards

prl · Post by **prl** » Tue Jun 16, 2009 11:40

tonymy01 wrote:...
This is an example of the socket info closing down:
Code: Select all
pulled from socket:21 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
pulled from socket:23 fd:-1 buf:[rey's Anatomy + Scrubs 05-02.ts]len:8192 nread:0
...
The nread normally has valid numbers (is the number read from a real request to the server). So I am guessing it getting "0" is that the socket didn't get anything new in it from the server. ...

fd=-1 is an illegal file descriptor. I'm surprised that you get nread=0 as the return value from a read() or recv() when the fd is -1. I'd expect to get -1 as the read/recv return value, indicating an error, and an error value of EBADF, for bad file descriptor.

nread=0 normally means "end of file", or "sender closed socket and all data read".

It's also a bit strange that you're getting an fd of -1 to match in the "ready to read" bits in a select() call. The select call uses bit 0 (2^0) for fd 0, bit 1 (2^1) for fd 1, etc.

If there is data available on the socket, read() and recv() will return it, but it may be less data than you requested, but it should only return 0 when the socket has been closed by the sender and all the data buffered in the transmission chain from the sender to the receiver has been read.

select() should mark a socket with a "closed" condition as ready for reading, so you shouldn't have to do anything special to make sure your reads on the socket "see" the end of file.

IanSav · Post by **IanSav** » Tue Jun 16, 2009 11:45

Hi Tony,

Is "fd=-1" meant to be the file channel descriptor? If so, "-1" looks like an error return to me. That is, I think you are potentially accessing a closed channel. What is errno etc set to after each call?

Edit: SNAP I think Peter is on the same path.

Regards,
Ian.

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 11:50

Nope... Mongoose sends stuff to fd:-1 for any socket stuff, and to fd:something to any file sockets. There are other routines over the top of where I stuck the printfs that essentially have an "if fd==-1, then send to network socket, else send to file socket".

Here is the "pull" routine:

Code: Select all

static int
pull(int fd, SOCKET sock, SSL *ssl, char *buf, int len)
{
	int	nread;
	if (ssl != NULL) {
		nread = SSL_read(ssl, buf, len);
	} else if (fd != -1) {
		nread = read(fd, buf, (size_t) len);
	} else {
		nread = recv(sock, buf, (size_t) len, 0);
	}
	printf("pulled from socket:%d fd:%d buf:[%s]len:%d nread:%d\n",sock,fd,(char *)buf,len,nread);
	return (nread);
}

See it will read from a file if it gets a real file descriptor, else reads from the socket (and if SSL, uses that var also to influence the type of read).

prl · Post by **prl** » Tue Jun 16, 2009 12:10

You do know that you can do a read() from a socket, and that it behaves quite nicely?

The only difference is that recv() can be passed flags, but you're not using that.

But with the code as it is, nread=0 means EOF/socked closed, and so you should just close it and drop it from the select() set. As far as I can tell, since the sockets that have had an nread=0 don't appear again in the debug output, the socket handling is working as it should here.

Also your printf is printing whatever stuff was lying around in the buffer, whether or not it was actually read by the last recv (and since the recvs you're reporting had nread=0, nothing was read). You should probably use:

Code: Select all

printf("pulled from socket:%d fd:%d buf:[%.*s]len:%d nread:%d\n",sock,fd,(nread >= 0 ? nread : 0),buf,len,nread);

You also don't need to cast buf as a (char *), it's already declared that way. You should only use casts when you actually need them.

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 12:22

Blame the mongoose author for that routine, not me. I only added the printf... (I am sure I had a reason for casting that thing... maybe I was trying something else and left it there, can't remember). Good point on the stale buffer, I was trying to simply see what it was at that point in time, just in case anything subsequently didn't look at the nread=0...

edit:
Oh, the author had his reasons too... I think:
#define read(x, y, z) _read(x, y, (unsigned) z)
(This is only if the environ is a particular one in the makefile.. trying to follow the nested #ifdef's related to the makefile and knowing what a compiler also passes into the environment to understand the other #ifdefs is a nightmare for me!)

prl · Post by **prl** » Tue Jun 16, 2009 12:38

tonymy01 wrote:...
edit:
Oh, the author had his reasons too... I think:
#define read(x, y, z) _read(x, y, (unsigned) z)
(This is only if the environ is a particular one in the makefile.. trying to follow the nested #ifdef's related to the makefile and knowing what a compiler also passes into the environment to understand the other #ifdefs is a nightmare for me!)

That in itself is not a reason for casting the buf (second) argument. The macro is casting the count (third) argument.

tonymy01 · Post by **tonymy01** » Tue Jun 16, 2009 12:53

I said I put the printf there, nothing else. I had my reasons at the time for the casting, maybe I had the printf elsewhere and the compiler complained about one argument, so I started casting stuff until the compiler shut up. Can we get off this non-issue please I will remove all unnecessary casts now (it may have been because it was complaining about "argument 3" and I miss counted, I can't remember as I added this a few weeks ago).

I meant the author had his reasons for making the pull from network socket different to the pull from file, clearly this was due to one of the compile environs not supporting one of these options properly perhaps, hence also the #define to alter that particular read request also.

j s · Post by **j s** » Wed Jun 17, 2009 03:27

On a laptop I'm getting the following error on startup of Mongoose....

Code: Select all

bind() failed for port:1900
setsockopt() add_membership failed

and the laptop cannot be discovered - though wizFX can see it fine if the IP address is entered as a favourite.

On my desktop it starts normally though discovery only works from the desktop itself - which is the major reason I'm playing with it. After having worked without issue for over a year a few months ago both the P1 and S1 stopped discovering each other and neither can be discovered by WizFX (adding them manually as favourites is fine). I'm hoping to use mongoose to try to identify the problem but I have to get it working first. Any ideas what might be causing the bind() problem?

off the topic now somewhat but still SSDP related....

Any ideas why discovery might have stopped working on the P1 and S1? The P1 and desktop (WizFX) connect to the same gigabit switch, the S1 on a 100megabit switch connected the gigabit switch. Media play works fine - it's only the SSDP stuff that seems to be not getting through. Both client & server are enabled on both (as they have been since day 1)

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 08:47

Is your router properly forwarding your multicast packets through? I think your laptop must already have something listening on 1900 and transmitting on 239.255.255.250, I spent a bit of time trying to get this working on Cygwin also (bind was failing on my PC and this was due to PNP already enabled for my Windows Media Player server/streaming to my Yamaha amp & Nokia phone). Then I found through lots of googling and trial & error that setting the created 1900 socket a "SO_REUSEADDR", it played nicely with the PNP setup on my machine. On my SLUG, I didn't even need to set that, and despite the SLUG having PNP enabled by default (and I could see all the PNP discovery messages being broadcast from it), it could see and send stuff on that socket.
So looking at the chain of events in the code, it appears that the SSDP stuff I put in mongoose is creating the 1900 socket ok, and setting the SO_REUSEADDR ok, but then is not able to bind to it (I don't think this is would cause it not to be able to see stuff, but may cause it not to be able to send stuff on that port). The add membership, as far as I can tell, is just making sure you have a route on your PC to that 239.255.255.250.

Back to your Wizzes not talking to each other... some routers have the option not to send multicast stuff around perhaps, make sure you didn't tick any option to block or perhaps send multicast stuff only thru the WAN port or something.

I wrote a little test prog that listens for any multicast stuff coming through, maybe try this on your machine that doesn't have the bind problem (and try it on the other one too..) and may give you some pointers about what you are seeing on your network:
http://tonyspage.abock.de/beyonwiz/mult ... n_test.zip (needs to run with the cygwin1.dll like mongoose).
For some reason I couldn't work out in the 5mins I had to set this back up again (I used this as a test when learning how to deal with multicast in the early part of my development of this) it isn't seeing the WizFX outgoing on the same machine it runs on, where as I could have sworn I could see these at one point (meh, maybe it was only the receiving I saw). I had 2 boxes, the PC and the SLUG both chugging away showing me what they saw when playing with WizFX, so it was information overload anyway.

Other "hints" on your PC about the bind/multicast issue is do a "netstat -an" from the command prompt.
On a PC with some kind of PNP enabled, you will spot:
UDP 127.0.0.1:1900 *:*
UDP 192.168.0.120:1900 *:*

But this still shouldn't block you to re-use and listen on those ports... and doesn't explain your Wiz-Wiz issues, which must be your router really blocking these.

Regards

j s · Post by **j s** » Wed Jun 17, 2009 10:36

Hi Tony

Thanks for the response - I'll have a play with the listener.

I know a fair bit about IP in general but not much about uPnP and multicast (it wasn't covered when I did a Novell CNE 15 years ago). What does the router have to do with it? This is all inside the private network.

Damn!! It looks like it's the gigabit switch blocking the multicast - put everything back on the 100baseT switch and can now discover the P1 and the P1 can see Mongoose.

The test listener has the same bind() issue as Mongoose on the laptop - not surprising I guess. On the desktop it works and can see the QNAP sending stuff - Twonkymedia server I assume.

Yep - definitely the switch - moved the QNAP back to the gbit switch and no longer see mediaserver packets. I feel a query to the vendor coming on - luckily Aussie - wiretek.com.au

Thanks for the tool - I'll probably use Mongoose on the desktop (where it works) - the notebook testing was just that - testing - have no need to run it there normally. Long term I'd like to get a QNAP version of your server though maybe I have too many media files there - SMB works fine anyway.

prl · Post by **prl** » Wed Jun 17, 2009 10:55

j s wrote:...
I know a fair bit about IP in general but not much about uPnP and multicast (it wasn't covered when I did a Novell CNE 15 years ago). What does the router have to do with it? This is all inside the private network.
...

The router has to recognise the multicast address range, 224.0.0.0 to 239.255.255.255, and treat packets with those destination addresses in a similar way to broadcast packets. It also has to do the correct TTL processing for the packets so they don't escape beyond their intended habitat

Switches need to do similar things.

IanSav · Post by **IanSav** » Wed Jun 17, 2009 10:56

Hi Tony,

Can another process on the PC exclusively open port 1900 and hence block shared access?

Regards,
Ian.

prl · Post by **prl** » Wed Jun 17, 2009 11:20

IanSav wrote:Hi Tony,

Can another process on the PC exclusively open port 1900 and hence block shared access?

Regards,
Ian.

This is how it's supposed to work (Linux setsockopt(2)):

Code: Select all

SO_REUSEADDR
      Specifies that the rules used in validating  addresses  supplied
      to bind() should allow reuse of local addresses, if this is sup‐
      ported by the protocol.  This option takes an int value. This is
      a Boolean option.

Of course, the setsockopt(2) call needs to be done before bind(2).

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 11:22

Possibly Ian. Until I worked out I had to do this "setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (char *) &one, sizeof(one))", I had the same problem with both the SLUG (having UPnP enabled on it blocked the bind or the receive, can't quite recall the sequence of events, so I disabled the SLUG UPnP in the early parts of testing) and my PC (which has lots of UPnP things interacting I think).
edit: I have contradicted myself about this SO_REUSEADDR behaviour for SLUG and PC... I think part of my issue was also that, even though I could listen on 1900, I couldn't send on it unless I could bind to it (and then it does need that SO_REUSEADDR enabled on the socket to behave with other UPNP apps/services enabled on the machine). The other issue was that cygwin "printf()" was not actually sending anything to screen. So while I thought that I wasn't getting anything because it wasn't listening on port1900, it was in fact because printf wasn't actually sending anything to the visible screen. I found "fprintf(stderr," was the way to get cygwin to dump stuff to screen. So it was a bit of "trial and error" about making socket and multicast changes in the way I tried it, with various different bits of sample code I found in google, then discovering printfs weren't displaying, then finding my sample listeners etc were working ok at the state that I had altered them to.

But multicasting should not have anything exclusively locking on port 1900, this is meant to be a sharable resource across many applications, and if one of those applications is playing nasty, then the author of that application should be told as it goes against the whole UPnP idea.
Here is a tidbit about UPnP for Windows though:
http://support.microsoft.com/kb/886257
It tells you how to enable your firewall (somewhat), but suggests something about Internet Connection Sharing being enabled should block any attempt to open up ports or something (skimmed it only).

As for the switch, ensure that UPnP passthrough is enabled, as Peter suggests. This is normally configurable on most better switches. It may be called SSDP or may be called UDP multicast... and may not be an enable, it may be a block instead...

What model Wiretek? Does it show up in Windows UPnP list when you connect it, maybe it also listens and sends UPnP and forgets to forward the ones on that it receives?
Regards

j s · Post by **j s** » Wed Jun 17, 2009 11:33

prl wrote:
j s wrote:...
I know a fair bit about IP in general but not much about uPnP and multicast (it wasn't covered when I did a Novell CNE 15 years ago). What does the router have to do with it? This is all inside the private network.
...
The router has to recognise the multicast address range, 224.0.0.0 to 239.255.255.255, and treat packets with those destination addresses in a similar way to broadcast packets. It also has to do the correct TTL processing for the packets so they don't escape beyond their intended habitat Switches need to do similar things.

I still don't understand Peter? Sure, the router would be involved in either letting (or not letting) multicast packets out into the big wide world but would have no effect on the internal LAN. Switches work at the Link layer and hence deal with MAC addresses only so surely AFAIK should just forward all multicast packets on.

Anyway - it seems I have found the problem - the gigabit switch - avoid wiretek switches is my conclusion.

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 11:53

Here is your story on whirlpool:
http://forums.whirlpool.net.au/forum-re ... 00164.html

prl · Post by **prl** » Wed Jun 17, 2009 11:56

j s wrote:
prl wrote:
j s wrote:...
I know a fair bit about IP in general but not much about uPnP and multicast (it wasn't covered when I did a Novell CNE 15 years ago). What does the router have to do with it? This is all inside the private network.
...
The router has to recognise the multicast address range, 224.0.0.0 to 239.255.255.255, and treat packets with those destination addresses in a similar way to broadcast packets. It also has to do the correct TTL processing for the packets so they don't escape beyond their intended habitat Switches need to do similar things.
I still don't understand Peter? Sure, the router would be involved in either letting (or not letting) multicast packets out into the big wide world but would have no effect on the internal LAN. Switches work at the Link layer and hence deal with MAC addresses only so surely AFAIK should just forward all multicast packets on.

Anyway - it seems I have found the problem - the gigabit switch - avoid wiretek switches is my conclusion.

Multicast also uses a specific set of Ethernet MAC addresses, so any link layer treatment also needs to recognise them. Switches need to decide whether a packet needs only to be sent to a single port (normal point-to-point packet) or to all ports, (broadcast or multicast packets). However, that should be able to be determined by the same MAC address bit for both broadcast and multicast, the LSB of the first octet of the MAC address.

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 14:33

I just had another idea for the "GE dodgy switch" problem.
You could simply "spoof" unicast responses to the two wizes from your PC and they might think the data came from each other depending on how they interpret the message.
I was not even thinking of spoofing your source IP address (although typing this now it may work better that way), I was merely thinking you could do the unicast response to a Wiz client that it would expect to see when it does a multicast "request" (ssdp:discover) to find servers on the network. The only issue I see with that is that you can't see the requests, so you effectively just have to send a repeating stream of "responses" and hope the Wiz client sees one of them and believes it to be a reponse based on its request. It also has to send regular ssdp:alive's, which normally go to the multicast address, but in this case will have to be unicast to each server. (ssdp/UPNP protocol is essentially a client will broadcast/multicast a discover with what it wants to discover, in our case, a "wizpnp-upnp-org:device:pvrtvdevice:1" device, and the UDP response back from the server (with "HTTP/1.1 200 OK") is actually unicast to the IP address of the multicast that it sees on its listening socket. I am guessing the unicast stuff will get thru your GE switch OK.
The other issue I see is that the Wiz, when it gets the response with the IP address of the http server/index.txt (which in our special "UPNPunblockertool" case is the IP address of your other Wiz server), it may also look at the IP address of the source UDP unicast packet, not sure on that. If it does, then we need to go one level lower and spoof the source address for these UDP unicasts... and the final possible "gotya" is the Wiz does a ssdp:discover with a WANT:blah and that blah appears to be a unique representation of the other Wiz and not something you will know yourself. But I can't see that being a big problem, as the Wiz only uses that info for the multicast discover/keep alives, and not the TCP HTTP index.txt & file streaming.
Let me knock something up and give you a shot at it... the only hassle I see in getting something out quickly is parsing arguments (the two IP addresses of your Wiz) is a bit of a pain... unless I trust the arguments you type of course... Normally all my coding I "hard code" the test environ as the compiling takes seconds only, that way I don't have to code all the argument/config file parsing routines.
Regards

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 18:00

Ok, here it is.
Wack your two Wizes back into your (dodgy) GE switch, and run this:
http://tonyspage.abock.de/beyonwiz/PNP_GE_Fix_text.zip
This essentially is the "PNP listener" and now a poller also. It will poll the 2 addresses you give it and "pretend" to be two wiz servers polling alives (every 60secs) and responding to "WANT" discovers (every 3 or so secs). It doesn't broadcast/multicast anything, as we know your GE switch kills it. It simply sends these messages to each of your Wiz IP addresses.
You should see your two wizes showing the IP address of the other Wiz in the fileplayer network list and be able to stream as per normal from each wiz.... maybe.
Cheers!

j s · Post by **j s** » Wed Jun 17, 2009 18:53

Hi Tony,

If the above is on my behalf then the dodgy switch is not really worth the effort of trying to work around it. The 100BaseT switch is quite adequate fpr tthe task so Ill just go back back to using that. Annoyed that I have to but not really a big deal.

Thanks for that listening test tool - made it much easier to confirm the cause.

tonymy01 · Post by **tonymy01** » Wed Jun 17, 2009 19:31

Please give the http://tonyspage.abock.de/beyonwiz/PNP_GE_Fix_text.zip tool a try though (thru the GE switch), if it works I will keep it as a solution for others who can't get their two wiz's speaking to each other.

j s · Post by **j s** » Thu Jun 18, 2009 16:02

tonymy01 wrote:I have also made mongoose do one more trick... it turns ".rec" files into ".ts" filenames in the index.txt file, so the Beyonwiz can see them. When the beyonwiz goes to download them, mongoose, if it fails to find the file, rather than reporting a 404 File Not found, will try going thru the extension mapping tree to see if it can find a file of similar name, and try that instead. So this means Toppy owners need not rename the .rec files into .ts any more. I only added this feature late last night, so haven't tested it much yet....

Love this feature Tony!!

Any chance, perhaps, of using an ftp source as root (or a subfolder of root)? I imagine you can guess why

I tried using ftp://192.168.0.101:2021/DataFiles as the root value (just in case it was that simple) but got "No such file or directory". (PHP can handle file system references like that)

I'll test GE-FIX either tomorrow or, more likely, the week-end. The HDD in the S1 failed earlier in the week. Replaced the caps yesterday (no diff) so popped a new WD5000AVJB in this morning. All fine now. The failed drive was an 7 month old WD5000AAKB.

To help with my testing do you know how long it takes for a Wiz to notice the disappeance/appearance of another Wiz? And do I need to refresh anything in the Wiz (standby cycle, exit/enter FileList, whatever) to see the change in Network?

tonymy01 · Post by **tonymy01** » Thu Jun 18, 2009 16:58

I find moving up or down in the filelist main view tends to do the trick, you will see it there when you, say, go to DVD, then back to network. I think the tool will work, last night while running it, I had *two* wizes listed in my Wiz, one for the "friendlyname" and one for the "ipaddress"

More wizpnp servers running during testing of things than I can poke a stick at

The server will appear at least within about a minute of it being enabled. Once the client starts seeing the server ssdp:alive messages (which the WizPNP server sends once a minute), it quickly polls discover messages asking for that Wiz server credentials. Once it starts doing that, they are effectively in a poll/response type cycle every 6 seconds or so.
The client also does a "wakeup call" with couple of generic discovers when the client wiz first boots up. I don't see these continuously getting sent, I think the Wiz client mainly sits there listening for the broadcasted ssdp:alive messages from the servers.

The server tends not to disappear from the list unless you power down the Wiz server (IIRC). Then it sends an ssdp:byebye before the power down and the Wiz client stops sending the 6second "WANT" polls for it also (and a menu refresh to DVD and back to network will show the server is no longer in the list).
The "quick and nasty GE PNP tool" doesn't send any "byebye" messages (I guess I could have added that, and made the polling interval a little less frequent... maybe on the weekend).

tonymy01 · Post by **tonymy01** » Thu Jun 18, 2009 17:02

j s wrote:Any chance, perhaps, of using an ftp source as root (or a subfolder of root)? I imagine you can guess why

hehe, yep.

I tried using ftp://192.168.0.101:2021/DataFiles as the root value (just in case it was that simple) but got "No such file or directory". (PHP can handle file system references like that)

eek, dunno what that would have done if it worked! It would have attempted to write an index.txt file to the Toppy for a start

. I would prefer that some other tool made the ftp drive like a directory on the PC, rather than attempt to support ftp in mongoose though, as there is a lot of parsing of the root directory structure to do things with (it uses stat etc on files it believes are on the local filesystem, won't work to an ftp server, surely).

All fine now. The failed drive was an 7 month old WD5000AAKB.

Oh dear... .hope my AAKB (about the same age) doesn't suffer a similar fate!

j s · Post by **j s** » Thu Jun 18, 2009 17:52

tonymy01 wrote:
j s wrote:Any chance, perhaps, of using an ftp source as root (or a subfolder of root)? I imagine you can guess why
hehe, yep.
I tried using ftp://192.168.0.101:2021/DataFiles as the root value (just in case it was that simple) but got "No such file or directory". (PHP can handle file system references like that)
eek, dunno what that would have done if it worked! It would have attempted to write an index.txt file to the Toppy for a start . I would prefer that some other tool made the ftp drive like a directory on the PC, rather than attempt to support ftp in mongoose though, as there is a lot of parsing of the root directory structure to do things with (it uses stat etc on files it believes are on the local filesystem, won't work to an ftp server, surely).

Yes, it did try to write Index.txt (failed of course - would have worked in PHP). Toppyweb ftp commands get the equivalent of stat - not as elegant as a stat call though.

There is a free product around - NetDrive- which will mount an FTP source as a drive letter - actually played with this - created a Windows share on the DataFiles directory and renamed a .rec file on it to .ts. The Wiz played it successfully.

The problem I found though is that the drive letter mapping on startup was a little unreliable - I had two drives mapped (2 toppies) and usually at least one of them failed to map which in turn meant that the share disappeared. And of course was of little use as long as the Wiz didn't recognise .rec files as playable. So I stopped playing with it.

It would be nice if your app could handle FTP directly as this would then work on other platforms - like our slugs and my qnap - eliminating the double transit of the network that running it in Windows would require.

Now that Mongoose s available I might have another play with NetDrive

j s · Post by **j s** » Thu Jun 18, 2009 21:51

mmm - Mongoose does work (sort of) with NetDrive but with a few issues...

Building index.txt is very slow - the server starts responding to the SSDP requests before the building is finished. This Toppy has ~60 rec files on it.

Trick modes don't work - or behave strangely when they try to work

Playback often gets "unsupported format" or will terminate inexplicably

All the above could just be due to the slowness of the Toppy USB transfer (non-turbo mode)

P.S. Seems like maybe some thread issues - after~15 minutes playing tried to end the server with "q" but 5 minutes later it was still waiting for threads to finish.