Slow SFTP transfer between T/U-series and FileZilla on Win10 PC

[Grumpy_Geoff]

I'd be interested to know if any one else is experiencing slow SFTP protocol file transfers similar to what I am.

Back in 4.4-series firmware days, using Samba and Windows Explorer to copy recordings to my PC, I was getting transfers that would max out at around 40MB/s if the T4 wasn't doing anything else. If I wanted faster transfers, I'd use the FileZilla FTP-client and get around 50MB/s. Good-o for dogs, no issue there.

Memory might be failing me as to when, but I think it was in the 16.1-series firmware we got a Samba speed boost to match the similar speed to what I was achieving via FTP, making FTP pretty much redundant for me (for recording transfers anyway).

Now with the the 17.5-series firmware and its security focus, I don't have the root user unlocked and I access the command line of the T/U-series units via PuTTY and private/public SSH keys.

For comparison, copying via Samba from the "Movie" share to my Win10 PC using Windows Explorer drag 'n drop I get around -
T2: 11MB/s (in media selection, not recording, not timeshifting). No complaints there, can't wring much more out of a 100Mb/s ethernet port.
T4: 47MB/s (in standby, one timer recording in progress). No complaints there.
U4: 75-80MB/s (in standby, no timer recordings). No complaints there.

Via FileZilla, using SFTP protocol and using the same PuTTY private/public SSH key files, I get very slow transfer speeds.
T2: 3.6-3.7MiB/s (it doesn't seem to matter if the T2 is recording/timeshifting or doing nothing).
T4: 7.4-7.5MiB/s (in standby, 1 recording active)
U4: ranges 29-31MiB/s (in live TV, timeshifting, no recording)

I don't know if I should be expecting better speeds via SFTP or it's just a protocol restriction. I ran top on the T2 whilst doing an SFTP test and found that the "sshd" process was maxing out the cpu - so likely the source of the slow speeds. For comparison, transfering the same recording a few minutes later via Samba had the "smbd" process using 28-30% cpu.

When I get some time, I'll see what unlocking the root user on one of the boxes does to the transfer speed when I can access using unsecured FTP.


This isn't much of an issue whilst we still have Samba access available. I'd likely only need to use SFTP to get to parts of the file system outside of the "Movie" share, and those files would be small in comparison.
If we lost this Samba access, then ouch...

Thoughts/suggestions, folks?


Cheers
Geoff

[MrQuade]

Encryption costs CPU cycles and will just naturally be slower on devices with limited power and no hardware assistance.

[prl]

Encryption costs CPU cycles and will just naturally be slower on devices with limited power and no hardware assistance.

That's the likey cause, and is consistent with the high CPU use of SFTP and scp. Ssh based protocols use public key server and client authentication, and secure private key exchange to encrypt all data transfers.

[IanB]

Yes it's cpu cycles doing the crypto. On a T3 it's basically useless, T2 bottom level, T4 just, U4 near acceptable.

You can mitigate it a little by selecting the arcfour cipher, this is reputed to be the quickest available, at a cost of crypto strength. Of course using the none cipher is the fastest but no-one enables that option these days.

Straight unencrypted FTP is still the fastest transfer available.

[Grumpy_Geoff]

I unlocked root on the T4. It was in standby, with no recordings in progress I started top and noticed the T4 was being flogged by the minidlnad process - argh! I had forgotten to turn that crap off and disable it.
I then used FileZilla - in unsecured FTP transfer mode. The result - Status: File transfer successful, transferred 2,456,738,316 bytes in 44 seconds
Which is 53MiB/s / 55MB/s (if my arithmetic is correct).

[MrQuade]

That sounds like the common experience. Personally I find it not significantly faster than SMB to warrant the minor hassle of using ftp though.

[prl]

... You can mitigate it a little by selecting the arcfour cipher, this is reputed to be the quickest available, at a cost of crypto strength. Of course using the none cipher is the fastest but no-one enables that option these days.
...

The only ciphers I can get to work with the T4 from OS X are: aes128-ctr aes192-ctr aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com chacha20-poly1305@openssh.com.

arcfour (or any of its varieties) isn't available. And, as you say, neither is none.

Here are some scp copy speed results for copies from a T4 for the ciphers (scp for ease of shell scripting):

Code: Select all

aes128-ctr:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB  10.3MB/s   03:27
aes192-ctr:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB   9.6MB/s   03:40
aes256-ctr:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB   9.3MB/s   03:48
aes128-gcm@openssh.com:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB   8.8MB/s   04:00
aes256-gcm@openssh.com:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB   7.9MB/s   04:27
chacha20-poly1305@openssh.com:
20170320 2129 - ELEVEN Canberra - Supernatura 100% 2121MB  14.0MB/s   02:32

I did an scp of the same file just before the loop to try to make sure the file buffer conditions were similar for all cases.

From the transfer speeds, it seems that my setup defaults to the chacha20-poly1305@openssh.com cipher. ssh -v doesn't appear to print the cipher choice.

Code: Select all

for i in aes128-ctr aes192-ctr aes256-ctr aes128-gcm@openssh.com aes256-gcm@openssh.com chacha20-poly1305@openssh.com; do
	echo $i:
	scp -o "Ciphers $i" beyonwizt4:'"/media/hdd/movie/20170320 2129 - ELEVEN Canberra - Supernatural.ts"' .
done

[IanB]

Ah! yes the new stream cipher chacha20 is the replacement for arcfour and still using the good old add-rotate-xor (ARX) methodology for fast implementation on standard processors.

[pbush1]

Hi, I've been noticing variable FTP transfer speeds to U4 from win10 to Draytek Vigor2860Vn plus using Filezilla (latest version) in unsecured FTP transfer mode. Can't pin the conditions down, but at times I'll get around 30MiB/s, usually only 10-12MiB/s or less. I noticed that Grumpy Geoff referred to 'unlocking root' on the T4 and this improved his transfer rates but don't know how this is done. Does anyone have anyy clues or suggestions ? Thanks

[MrQuade]

Hi, I've been noticing variable FTP transfer speeds to U4 from win10 to Draytek Vigor2860Vn plus using Filezilla (latest version) in unsecured FTP transfer mode. Can't pin the conditions down, but at times I'll get around 30MiB/s, usually only 10-12MiB/s or less. I noticed that Grumpy Geoff referred to 'unlocking root' on the T4 and this improved his transfer rates but don't know how this is done. Does anyone have anyy clues or suggestions ? Thanks

Grumpy Geoff wasn't saying that unlocking root improved speeds, he just needed to enable a root password so that he could use FTP. To use a SSH by authenticating with a key as he was doing, you don't need to have set a root password.

What sort of harddisk are you using? Are you sure you aren't capping out the speed on a low spec 2.5" model? Shingled perhaps?

I'm assuming this is over a hard wired network connection too?

[pbush1]

>he just needed to enable a root password so that he could use FTP.
Thanks - understood.
Using cat 6 ethernet between both laptop to modem & modem to U4.
Hard disks are the 1Tb internal on the U4 (the one it came with) and 250Gb m.2 in the laptop.
Not sure what 'Shingled' means (had to look it up) but no modification to HDDs.
If I could consistently get the higher speed I would be happy, but I seem more likely to transfer at the lower rate.

[MrQuade]

Hmm, that's strange then. With a non-encrypted FTP connection over gigabit wired connection, you should easily be getting 90MB/s to a U4 as long as the storage devices on either end can keep up.

I assume that you are using Gigabit ports on your modem/router?

Shingled drives are a newer technique for increasing data density. When tracks are written, they partially overwrite adjacent tracks, a bit like tiles on a shingled roof. The downside to that is that write speeds can suffer greatly as some of the partially overlapped data will need to be rewritten. This should only be a problem when writing to the U4 though, not if you are copying recordings FROM the U4.

There is a bottleneck in your setup somewhere, so its just a matter of locating it. It may be the U4's HDD, connection to the HDD, U4 CPU overloading (unlikely), network cables, router, or something in the laptop.

[Grumpy_Geoff]

If I could consistently get the higher speed I would be happy, but I seem more likely to transfer at the lower rate.

I haven't used FTP for quite a while. Usually I just use File Commander from the Wiz, or alternatively Windows to drag 'n drop.

What transfer speed do you get when using Windows to copy to the U4?
Are you using FTP because you're transferring to /media/hdd/Temp directory and can't get to it via Samba? (i.e. you don't have access to that directory from within Windows)?

Are you aware you can play/stream media on the U4 from Windows shares on the PC? This involves mounting the Windows shares on the U4. It's how I watch my downloaded content - I don't transfer anything to the Wiz.

[IanB]

I noticed in the screen snap you posted in the first you did a single download and got 32.4 MiB/s which is pretty much as expected.

In the second you are doing 3 parallel downloads and the first two are getting 4.9 MiB/s and 6.4 MiB/s, can't see the third, but I'll assume a similar speed. This is pretty much to be expected, you are experiencing contention by running to many things at once.

The T4/U4 are quite a powerful boxes for a PVR but are not in the same league as a modern PC.

[MrQuade]

I notice is the screen snap you posted in the first you did a single download and got 32.4 MiB/s which is pretty much as expected.

In the second you are doing 3 parallel downloads and the first two are getting 4.9 MiB/s and 6.4 MiB/s, can't see the third, but I'll assume a similar speed. This is pretty much to be expected, you are experiencing contention by running to many things at once.

The T4 is quite a powerful box for a PVR but is not in the same league as a modern PC.

He says he has a U4 which should be faster.

Though now that I look at the screenshot too, I see that he was writing to the U4's harddisk, which could indicate a shingling limitation.

And in fact, as you pointed out as well, the three parallel transfers would only serve to exacerbate that issue.

[Grumpy_Geoff]

Though now that I look at the screenshot too, I see that he was writing to the U4's harddisk, which could indicate a shingling limitation.

The single-file transfer was showing as 32.4MiB/s, and the U4 is only ~3 weeks old.

What speed did you get with your dirtied-up shingle-poxed drive in the U4, or did you never test it in that (and it's now in your T2, right?)

[pbush1]

Thanks Grumpy, Ian & MrQuade for your advice/thoughts.
Yes, plugged the U4 into GigaLAN 2 (refer the attached - port 1 is the connection to the laptop port replicator), however, looking closely at the modem, the right LED on port 2 was not on (refer attachment) so suggesting connected with 10/100 Mbps - not sure if this makes a difference. Another attachment also shows a transfer of one file only with 7.2MiB/s so, single file transfer sometimes goes slowly. Multiple transfers sometimes add up to about 30-35MiB/s.
I only use FTP because it was the simplest transfer method with my old Topfield. Don't know how to use any other method (apart from manual usb flash transfer) - I will read up on Samba, or another method any of you recommend that's simple enough (for an amateur like me). Thanks

[pbush1]

Just tried something - wondered if the port replicator had a problem so plugged the modem directly into the laptop ethernet (Gigabit) port - no change. Ethernet back to the port replicator and recycled the modem - both LEDs on port 2 are now on, and an FTP transfer shows 31MiB/s ! Problems appear to be not with the U4, but still keen for ideas on faster transfers using other methods.

MrQuade said '..I see that he was writing to the U4's harddisk..' - where else on the U4 would I be copying files ?

Thanks again

[Grumpy_Geoff]

I only use FTP because it was the simplest transfer method with my old Topfield. Don't know how to use any other method (apart from manual usb flash transfer) - I will read up on Samba, or another method any of you recommend that's simple enough (for an amateur like me). Thanks

In Windows File Explorer, the U4 should show up under 'Network' grouping (normally named as BEYONWIZU4 unless you've changed the U4's network name).

Another way to bring up a File Explorer window "exploring" the U4 is from a DOS command prompt start \\beyonwizu4 (or start \\192.168.2.10). Similarly \\beyonwizu4 from a WIN-R 'run' prompt).

The /media/hdd/movie/ directory is shared read/write by default. You can use Windows Explorer to then copy to/from the U4 (the default share on the U4 is named 'Movie').

You'll need a custom Samba config file on the U4 if you want to write to your /media/hdd/Temp/ (but /media/hdd/movie/Temp/ directory will be okay). We can get to that if you get faster speeds using Samba compared to FTP.

Here's my speed copying from my PC to a U4 via Windows command line. Commands were copied and pasted as a set to ensure accurate timing -

Code: Select all

E:\Temp>echo %TIME%
11:18:59.09

E:\Temp>dir %FIL%
 Volume in drive E has no label.
 Volume Serial Number is C874-4C33

 Directory of E:\Temp

18/02/2019  03:57 PM     1,364,764,974 Hawaii.Five-0.2010.S09E15.720p.HDTV.x264-AVS[eztv].mkv
               1 File(s)  1,364,764,974 bytes
               0 Dir(s)  204,209,491,968 bytes free

E:\Temp>copy %FIL% \\BEYONWIZU4\Movie\Test
        1 file(s) copied.

E:\Temp>echo %TIME%
11:19:15.17

E:\Temp>

1,364,764,974 bytes / 16.08 secs = 84,873,443 bytes/sec
Another test copy of the same file: 1,364,764,974 bytes / 14.78 secs = 92,338,631 bytes/sec

My U4 has a Toshiba 3TB MQ03ABB300 drive.

[MrQuade]

MrQuade said '..I see that he was writing to the U4's harddisk..' - where else on the U4 would I be copying files ?

My first assumption was that you were copying recordings FROM the U4 to your PC, and not tranferring movies from the Laptop to the U4. The difference being, that in your case, you are writing to the U4's disk instead of reading from it. With some harddrives, that makes a difference.

[MrQuade]

The single-file transfer was showing as 32.4MiB/s, and the U4 is only ~3 weeks old.

What speed did you get with your dirtied-up shingle-poxed drive in the U4, or did you never test it in that (and it's now in your T2, right?)

I think the main point of the shingled performance is that it not 100% deterministic from a high-level user experience. The single transfer might have been going to a cleaner area of the disk, whereas competing transfers might have been trying to stomp all over each-other. Don't know, just chucking it out there as an option.

The drive that I was experimenting with has been mostly untouched in the U4 for some time now. It only get very light usage and I haven't recently tested it. I never really found much of a performance problem with it in initial tests, but maybe another round of abuse is in order.

[pbush1]

Thanks again for the earlier advice - investigating Draytek issues re slow transfer speeds but some of the posts indicate that explorer drag & drop would be a simpler way of transferring files (Thanks MrQuade - I understand now - I usually download from usenet etc & transfer TO the U4 to watch on TV). I can map the 'Movies' directory, but Grumpy, would appreciate when you have time the method of sharing other U4 directories (specifically 'Temp') to allow this option. Cheers

[Grumpy_Geoff]

The Samba config extension file you need to create on the U4 is: /etc/samba/smb_user.conf, and to set a share on your current /media/hdd/Temp/ directory you use then the config contents should be something like:

Code: Select all

[Temp]
  comment = /hdd/Temp
  path = /media/hdd/Temp
  guest ok = yes
  read only = no

[pbush1]

Can't thank you chaps enough - got the Samba files in & working, now can transfer via Win Explorer to \tmp (thanks Grumpy) - much simpler than Filezilla. Found that the slow transfer rate was due to a dodgy ethernet cable (embarrased !!). Much appreciated !!! Is it just a repeat of the contents in smb_user.conf to share another directory in the U4 (/media/hdd/Gopro) ? eg:

[Temp]
comment = /hdd/Temp
path = /media/hdd/Temp
guest ok = yes
read only = no
[Gopro]
comment = /hdd/Gopro
path = /media/hdd/Gopro
guest ok = yes
read only = no

[IanB]

... Is it just a repeat of the contents in smb_user.conf to share another directory in the U4 (/media/hdd/Gopro) ? eg:

[Temp]
comment = /hdd/Temp
path = /media/hdd/Temp
guest ok = yes
read only = no

[Gopro]
comment = /hdd/Gopro
path = /media/hdd/Gopro
guest ok = yes
read only = no

Yep, just clone the section for each share you wish to export.

[raymondjpg]

Can't thank you chaps enough - got the Samba files in & working, now can transfer via Win Explorer to \tmp (thanks Grumpy) - much simpler than Filezilla.

Even simpler, in my view, is download via the OpenWebIf interface. Doesn't require any setting up, files transfer to the directories you specify in your web browser, or in your download manager if you use one. Can't speak to file transfer speed compared to Samba or FTP as I don't use the former (I have a mixed Windows 7 Windows 10 T/U PVR network and very often the BW PVRs just go AWOL from the Windows file explorers) and only use the latter to access T/U system files.

[MrQuade]

Even simpler, in my view, is download via the OpenWebIf interface. Doesn't require any setting up, files transfer to the directories you specify in your web browser, or in your download manager if you use one.

I don't think that applies when you are sending files to the Wiz.

[raymondjpg]

I don't think that applies when you are sending files to the Wiz.

I think you'd be right about that!

[Paul_oz53]

...
(I have a mixed Windows 7 Windows 10 T/U PVR network and very often the BW PVRs just go AWOL from the Windows file explorers) and only use the latter to access T/U system files.
...

Originally, I set the IP address on each Wiz box and that could be quite erratic with all sorts of strange disappearing acts. Ever since I allocated fixed IPs by MAC address in the router I have experienced very stable performance on the Windows network. The DHCP setting is Automatic on the boxes.

If you haven't done so, I'd recommend it.

[raymondjpg]

Originally, I set the IP address on each Wiz box and that could be quite erratic with all sorts of strange disappearing acts. Ever since I allocated fixed IPs by MAC address in the router I have experienced very stable performance on the Windows network. The DHCP setting is Automatic on the boxes.

If you haven't done so, I'd recommend it.

If I ever get around to it! I have always preferred the old "clunky" fixed IP addresses on individual devices. partly because some of the modem/routers I have had did not have the capability to store allocated MAC addresses in the router. For example the Huawei HG658, which I was using up until about six months ago, had to be run IP routed if you wanted VOIP functionality. I now have a Huawei HG659 and a TP-Link TD-W9970, both of which allow for DHCP reservation. I'll have a stab at setting reserved IPs just for the T2 and U4 at first, see if that makes a difference.

[Paul_oz53]

... some of the modem/routers I have had did not have the capability to store allocated MAC addresses in the router...

I didn't even have that excuse - just didn't dare interrupt the other half (the modems in her study - between Facebook, online surveys and genealogy barely a moment to get in there!).

When we swapped to the NBN I got my chance and have never looked back.