Series2Folder plugin

[prl]

...
I see discussions about long and short names. I am only new to BW and am not familiar with this option.

It's all about how the names for the files that represent a recording are generated. "Short" filenames use only the date and name of the program/series. "Standard" filenames (default) use the date, start time of the timer, channel name and name of program/series. "Event" filenames have the same fields as "Standard", but have the name first. "Long" filenames are standard filenames with he show description in the timer added at the end.

Where possible, Series2Folder gets its name information from the recording's .ts.meta file. If that is missing, it tries to get the name from the filename.

It all runs into problems with short filenames if there are two recordings on the same day, and one is moved into its folder by Series2Folder (or by user action), the other one records with the same filenames, and then when it's moved into the same folder, it replaces the first recording.

The setting is in MENU>Setup>TV>Recording settings>Composition of recording names. I suggest you leave it as the default.

Also, some might like to arrange recordings of different series in different folders. I know IceTV can distinguish between series but not sure if this carries through to the BW. If so, then the option to separate series might be good.

In general, no. The FTA guide carries little in the way of episode information (almost none). IceTV sends series/episode information to the Beyonwiz but it isn't used in the Beyonwiz guide. Series episodes in IceTV are sometimes named in the short description as Series N Episode M but this isn't uniform. Where episodes have names, that is used in the short description by IceTV in preference to series/episode numbers.

[KGL]

I wasn't aware of the similar Topfield TAP, but I'm certainly willing to steal good ideas from it

Here is a link FYI. http://oztoppy.wikidot.com/automove. Please note I am not advocating for all these features in the Series2Folder plugin.

Doing a "starts with" name match is easy, but opens the door to requests for other kinds of matches: "contains", "case sensitive/insensitive", even "regular expression".

Matching part of a filename to a folder to me is simply another way of putting a recording into a folder based on the filename. Ultimately it is up to you as the developer if you want to add extra features and if you don't want to then that is your prerogative. I certainly wasn't expecting anything beyond filename matching.

It all runs into problems with short filenames if there are two recordings on the same day, and one is moved into its folder by Series2Folder (or by user action), the other one records with the same filenames, and then when it's moved into the same folder, it replaces the first recording.

See how the Topfield TAP handles that but as you have outlined the BW has far more options in filenames. It sounds more complicated and it is best left up to you. You may have to make a decision on what filename format Series2Folder works with and make that a caveat for its use.

There are performance issue with "scanning subfolders", because it's often not stated where the scan should start from, and what its scope should be. Series2Folder is intended to be small and fast. It won't be if it starts wandering off searching through everything on someone's NAS.

The problem with moving from a single recording directory into a directory structure like the one KGL would like is that for each episode, multiple directories need to be searched in order to find where episodes should go, and if a new directory needs to be created for some new episodes, there's no way the code can tell where it should go.

I don't use a NAS but I appreciate some do. I am not sure how the Topfield TAP works but is seems to keep a historical record. Maybe the Series2Folder needs a mini text file DB on the hdd that filename patterns are recorded to and is used to locate the folder path to move the recorded file to. Maybe to create a new move pattern, a filename could be selected, open the menu, choose create folder, choose where to create and okay the creation. Then the setup is recorded into a DB. Then the plugin could compare filenames with the DB and move the file accordingly. If this setup was used, then probably a delete option maybe necessary also.

In the timer window, timers can be opened and locations specified to specific folders. Unfortunately doing this doesn't set up a pattern that subsequent recordings use the same folder location. It is tedious doing this and currently the simpler option is to move files in the Media window. The Series2Folder does automate this but it would be good if it could handle more levels in a directory tree. 2 levels would be great. (it seems the Topfield TAP only does 2 levels anyway).

Ultimately, my aim is to keep the recordings in some sort of organised structure rather than 100+ recordings in the default directory. Likewise I would like to avoid 20-50 folders in the default directory which could happen with the current Series2Folder setup. However, one advantage that the BW seems to have over the Topfield is that folders with files in them can be moved. I couldn't do this with the Topfield although I did try once using ftp and ended up deleting the folder with its files!
Regards
Greg

[prl]

....

Doing a "starts with" name match is easy, but opens the door to requests for other kinds of matches: "contains", "case sensitive/insensitive", even "regular expression".

Matching part of a filename to a folder to me is simply another way of putting a recording into a folder based on the filename. Ultimately it is up to you as the developer if you want to add extra features and if you don't want to then that is your prerogative. I certainly wasn't expecting anything beyond filename matching.

It differs from what's there now in that currently the full program name is used as the folder name. Simple. But, as you note later in your post, that's not suitable for what you want, especially if the match string is different from the destination name, for example, to collect Better Homes and Gardens into Lifestyle.

I'd also like to try to keep configuration to what configuration entries (like what's in settings screens) can do, and that limits the sort of configuration that's possible.

It all runs into problems with short filenames if there are two recordings on the same day, and one is moved into its folder by Series2Folder (or by user action), the other one records with the same filenames, and then when it's moved into the same folder, it replaces the first recording.

See how the Topfield TAP handles that but as you have outlined the BW has far more options in filenames. It sounds more complicated and it is best left up to you. You may have to make a decision on what filename format Series2Folder works with and make that a caveat for its use.[/quote]


The Beyonwiz already has a mechanism to avoid filename clashes when programs are recorded. Unfortunately, that scheme isn't used when recordings are moved or copied, either by Series2Folder or my manual actions in the UI. That lack is also related to Problems moving same directory names to trash (the first few posts, then it gets a bit esoteric).

...
In the timer window, timers can be opened and locations specified to specific folders. Unfortunately doing this doesn't set up a pattern that subsequent recordings use the same folder location.
...

It is precisely that problem that Series2Folder is intended to address, especially for IceTV, where each recording is a new timer and there's no way of automatically tying the new timer to a folder.

[Grumpy_Geoff]

It may be time for Series2Folder to refuse to operate if recording filenames aren't in a compatible form.

Can Series2Folder be changed to support the "Event name first" value for the setting 'Composition of recording filenames'?
Messy?

[prl]

Probably. And probably.

[Grumpy_Geoff]

Understood, thanks.

[prl]

It may be time for Series2Folder to refuse to operate if recording filenames aren't in a compatible form.

Can Series2Folder be changed to support the "Event name first" value for the setting 'Composition of recording filenames'?
Messy?

I'm currently testing a new version of Series2Folder that:

• Fixes the crash when, for some reason, the configured movie directory doesn't exist.
• Handles "Event name first" recordings that have a .meta file.
• Correctly handles extraction of the show name and timestamp from recording names in all 4 formats if the .meta file doesn't exist
• Prevents recordings that have already been moved to a series folder from being clobbered if a new recording with the same file name is moved there (generates _NNN suffixes for the new files). (The interrupted recording problem.) So if SeriesName/YYYYMMDD HHMM - Service - SeriesName.ts exists, and another YYYYMMDD HHMM - Service - SeriesName.ts is to be moved into SeriesName, then it will be moved there as YYYYMMDD HHMM - Service - SeriesName_001.ts.

"Event name first" wasn't actually very messy: it could use a method needed for the show name and timestamp extraction:

Code: Select all

-        if f.endswith('.ts') and f[0:8].isdigit() and fullpath not in self.isRecording and isfile(fullpath):
+        if self.recNameType(f) is not None and fullpath not in self.isRecording and isfile(fullpath):

recNameType() returns one of ("standard", "event"", short", "long") for the four filename formats, and None if it can't work out what it is. recNameType() is a bit messy, but not too bad (21 LOC).

[Grumpy_Geoff]

Good stuff.
Will the 'prevent clobbering' fix work on non-standard file naming formats (event, short, long) too?

[prl]

Good stuff.
Will the 'prevent clobbering' fix work on non-standard file naming formats (event, short, long) too?

Thanks. Yes, the _NNN suffix bit is in a standard place in the filename, no matter what format is used. It's formatDependentPart[_NNN].extensions. So it's basically take the source filename, strip extensions, strip any existing _NNN suffix, add new suffix, add back extensions, if a file exists in the destination for any of the extensions that make up the source recording, bump the suffix and try again. It tests the .ts extension first, so it will almost always only ever need one file test to see that there's a recording to avoid. Suffix 0 is no suffix rather than _000. It's about 40 LOC all up.

I may see whether I can reuse the techniques for Move, Copy and Move to Trash in the general firmware.

[prl]

I've just installed a beta of the new Series2Folder (the v1.8-to-be) on our in-use T4. If it doesn't cause any problems, I'll probably release it next week.

I can make the beta available here, too. if there's any interest.

There won't be a 1.7 release. 1.7 fixes the crash if Series2Folder runs a scan and the default recording directory doesn't exist.

[prl]

Series2Folder 1.8 can now be downloaded from the first post in the topic.

Release notes v.1.7 (not a separate release, use v1.8 )

• Fix crash when default movie directory doesn't exist.
• Minor code cleanups

Release notes v.1.8

• Handle all recording filename formats (Long filenames, Short filenames, Event first, as well as standard).
• Avoid over-writing existing files in destination directory.

[Grumpy_Geoff]

...

• Handle all recording filename formats (Long filenames, Short filenames, Event first, as well as standard).
• Avoid over-writing existing files in destination directory.

Thanks and thanks.
BTW, I notice there are two 1.6 versions in the topic's first post -
"enigma2-plugin-extensions-series2folder_1.6_all.zip
(11.44 KiB) Not downloaded yet"
"enigma2-plugin-extensions-series2folder_1.6_all.zip
(11.44 KiB) Downloaded 316 times?

[prl]

...

• Handle all recording filename formats (Long filenames, Short filenames, Event first, as well as standard).
• Avoid over-writing existing files in destination directory.

Thanks and thanks.

No problem.

BTW, I notice there are two 1.6 versions in the topic's first post -
"enigma2-plugin-extensions-series2folder_1.6_all.zip
(11.44 KiB) Not downloaded yet"
"enigma2-plugin-extensions-series2folder_1.6_all.zip
(11.44 KiB) Downloaded 316 times?

Indeed, there are.

I'll clean it up when and if "normal service is resumed".

[Grumpy_Geoff]

How easy is this! I originally didn't read enough of the first post to realise the un-zipped version of the plugin was available online. Knowing it now, I can download it direct and then install straight from the HDD -

Code: Select all

root@beyonwizt4:/media/hdd# wget https://bitbucket.org/prl/series2folder/downloads/enigma2-plugin-extensions-series2folder_1.8_all.ipk
Connecting to bitbucket.org (104.192.143.1:443)
Connecting to bbuseruploads.s3.amazonaws.com (52.216.81.144:443)
enigma2-plugin-exten 100% |*******************************************************************************************************************************************************| 12812   0:00:00 ETA
root@beyonwizt4:/media/hdd# ls -al *series2folder*
-rwxr--r--    1 root     root         11552 Apr  4  2016 enigma2-plugin-extensions-series2folder_1.6_all.ipk
-rw-r--r--    1 root     root         12812 Apr 20 08:52 enigma2-plugin-extensions-series2folder_1.8_all.ipk
root@beyonwizt4:/media/hdd#

[prl]

The only reason the forum versions are zipped is that the forum won't let me post .ipk files. The .ipk file contents are compressed anyway.

[Grumpy_Geoff]

Okay, thanks.

Confirming success with moving "Event name first" setting for 'Composition of recording filenames'. S2F has now moved a couple of recordings that were set that way last month

[Grumpy_Geoff]

The only reason the forum versions are zipped is that the forum won't let me post .ipk files. The .ipk file contents are compressed anyway.

Ahh, okay. You could do what adoxa does and stick a forum-accepted fake extension on it (he uses .tgz), and then lazy bones here can simply copy the file URL and download straight from the forum -

Code: Select all

wget -O series2folder_1.8_all.ipk http://www.beyonwiz.com.au/forum/download/file.php?id=0123456789

One less click for me

[Grumpy_Geoff]

Confirming success with "Avoid over-writing existing files in destination directory". I stopped a timer and waited until S2F had moved the recording, then restarted the timer, and got a suffixed recording in the destination -
[Series2Folder] rename /media/hdd/movie/20180420 1405 - ABC ME - Slugterra.ts to /media/hdd/movie/Slugterra/20180420 1405 - ABC ME - Slugterra_001.ts

[prl]

I prefer to use the right extension where possible.

[prl]

Confirming success with "Avoid over-writing existing files in destination directory".

Good to see. Thanks.

[adoxa]

[...] You could do what adoxa does and stick a forum-accepted fake extension on it (he uses .tgz) [...]

I prefer to use the right extension where possible.

Since .ipk files are gzipped tar files, there's an argument that .tgz is the right extension.

[prl]

[...] You could do what adoxa does and stick a forum-accepted fake extension on it (he uses .tgz) [...]

I prefer to use the right extension where possible.

Since .ipk files are gzipped tar files, there's an argument that .tgz is the right extension.

The implementer(s) of Component.Scanner and FileCommander may beg to differ.

And in any case, ".ipk" doesn't necessarily imply that the file format is gzip compressed TAR (for Series2Folder .ipk packages, it's not, it's ar(5) format). Also, ".ipk" implies specific names for the files enclosed, whereas ".tgz" carries no such implication.

[BradJ]

Can S2F be setup to maintain x files within a folder?
e.g. If I have a folder the ‘The Project’, when it moves a new episode into the folder can it delete any older episodes keeping only ‘x’ most recent episodes.
It would save me having to go through all folders when the HDD is almost full cleaning out 10’s of episodes. I’d likely only be interested in catching up on very recent episodes for most shows anyway.

[prl]

No, it can't. Series2Folder never deletes recordings.

[bigalo]

noob question... does this "enigma2-plugin-extensions-series2folder" work on a U4? I've tried twice now to install directly via *.IPK from the root of USB stick to no avail.
My U4 acknowledges the *.IPK file and appears to install it, but I do not see in the list of plugins installed (or available) in the OpenWebif interface.
Furthermore, I have several episodes of a few series recorded now, and I don't seem to be able to access the "...the Move series recordings into folders entry will be in the MENU list in the Movie Player (MEDIA from live TV)."
I am:
1) selecting the MEDIA button on the U4 remote
2) I see the listing of the completed recordings
3) select the MENU button on the U4 remote
4) no "move series recording into folder" option

Can you post a "screen shot" of what the menu is expected to look like?
Any insight would be greatly appreciated.

[MrQuade]

It does work on the U4. Have you restarted the GUI on the U4, or rebooted it?

The plugin doesn't appear in the plugin menu like many of the others plugins do.
If you are in the U4's Media Player, you can highlight any recording, then press the MENU button. The series2folder options should appear in that first popup box (you may need to scroll to the popup box's second screen to see them).

[prl]

If you are in the U4's Media Player, you can highlight any recording, then press the MENU button. The series2folder options should appear in that first popup box (you may need to scroll to the popup box's second screen to see them).

Or media file. But very definitely, if a directory is currently highlighted when you press MENU, none of the Series2Folder entries will appear in the menu. Neither will most of the other functions in the menu.

This is s function of how the menu is constructed, and is not in control of Series2Folder, so even Series2Folder functions that don't operate on a specific recording don't work when a directory is highlighted.

[bigalo]

Thanks for the replies.
Upon attempting to re-install from USB stick, I realize that I did *not* actually "select" the *.APK (ticking the select box)… <DOH!>.
It is all working as advertised now.

[kenmax]

Hi, I am having no luck in getting S2F working. I have installed v1.8 no problems. I press media button, a recording is selected, press menu button, select "Move to Folder" and get a 'series to folder did not find anything to move in media/hdd/movie'
Suggestions?
TIA Ken

[prl]

In the media selection screen, what is "MENU>Series to Folder...>Configure move series recordings to folders>Automatically create folders" set to? It defaults to "2 recordings".

Series2Folder will move series recordings to a folder if there are that many or more episodes of a series in the movie folder, or if a folder for the series already exists.

If you currently don't have any series where you have two recordings or more (for the default setting), MENU>Series to Folder won't move anything.

If the episodes don't have exactly the same recording name, they won't be recognised as part of the same series (that can happen sometimes with FTA recordings where they sometimes add tags like "Special" or "Encore" to a program name).

You may also want to enable "...>Configure move series recordings to folders>Allow Series to Folder to run in the background". It will then run automatically to move recordings to folders (but following the same rules as MENU>Series to Folder).

There are broadly two ways to use Series to Folder:

• Set the number of recordings in Automatically create folders to a reasonable number so that folders will be created automatically when unwatched episodes start to cause clutter.
• Set Automatically create folders to "no autocreate", and use Sel[ected] Series to Folder to create folders for the series that you want Series to Folder to manage. To do that, navigate to any episode of the series and use MENU>Sel Series to Folder to create the folder for the series and move all episodes of it into the folder. This is how I use Series2Folder (with Allow Series to Folder to run in the background enabled).

[MrQuade]

Suggestions?
TIA Ken

The option you selected tells the plugin to search the recordings directory and when it finds two or more (configurable) recordings with the same name, it will create a directory with that name and move the recordings there. It will also look for any single recordings that match an existing directory name and move them in there too.

If you selected that option and didn't have at least two episodes of a series in the recordings folder out didn't have any identically named sub directories, then the plugin won't do anything.

If you want the plugin to make a directory for a single recording, then you need to select the other MENU option "Move selected series to folder".

[kenmax]

Here is what media list shows, settings of StoF and response of StoF.
Do I have to create a folder before StoF can move them?

[MrQuade]

Are those recordings in /media/hdd/movie? (your screenshot is cut off)

[kenmax]

Yes they are

[prl]

Here is what media list shows, settings of StoF and response of StoF.

I've just tried MENU>Series to Folder (in Series2Folder v1.8 ) with the same configuration settings as you, and it worked just fine.

Do I have to create a folder before StoF can move them?

I explained that in detail in my post. With the Series2Folder settings you have, it should automatically create a folder for (at least) Home and Away and Seven News.

Could you try using MENU>Sel Series to Folder on a series that you'd like Series2Folder to put into a folder? That should create a folder just for that program and move all its episodes into that folder.

BTW: do you have any idea why you have multiple recordings of Home and Away for the same start times (or for almost the same start times)? E.g. 5 recordings for it starting 19:29 yesterday 4 Oct?

[Paul_oz53]

Yes they are

I run S2F v1.8 on Home and Away (amongst many others) on two machines with no problems at all. Only thing that temporarily delays transfer is if recordings are active at the time.

[prl]

Yes they are

I run S2F v1.8 on Home and Away (amongst many others) on two machines with no problems at all. Only thing that temporarily delays transfer is if recordings are active at the time.

And for other reasons. By design.

[kenmax]

Still no joy with either selected or all in StoF, still says nothing to do in either. I was running on 17.5.20180417, so i updated to latest beta, no joy, uninstalled StoF and reinstalled, no joy, (ended up with a crash in latest beta, MEDIA would not show recordings)
Reinstalled 17.5.20180417 from USB and went to install StoF (said it couldn't because it was already installed? Should that not have been wiped by USB reinstall?) still won't work! (not sure why I had all those recordings with same time)

[prl]

Reinstalled 17.5.20180417 from USB and went to install StoF (said it couldn't because it was already installed? Should that not have been wiped by USB reinstall?) still won't work! (not sure why I had all those recordings with same time)

I can't see any way that the Series2Folder plugin would be on the system after you did an install from USB. It would help if you would quote the exact text of errors. It helps find where the issue is in the code.

[MrQuade]

Should that not have been wiped by USB reinstall?) still won't work! (not sure why I had all those recordings with same time)

As-per prl's advice, the plugin should most definitely not be installed after a fresh USB firmware installation.

Assuming that the USB firmware install worked properly, I can only think of one thing. Have you field around with the configuration backup settings? Specifically, have you customised which files get backed up when performing a configuration backup?

[prl]

Should that not have been wiped by USB reinstall?) still won't work! (not sure why I had all those recordings with same time)

As-per prl's advice, the plugin should most definitely not be installed after a fresh USB firmware installation.

Assuming that the USB firmware install worked properly, I can only think of one thing. Have you field around with the configuration backup settings? Specifically, have you customised which files get backed up when performing a configuration backup?

Interesting thought, and it might explain how Series2Folder reappeared without being installed after a USB upgrade. However, it would mean that the backup must contain a path that includes /usr/lib/enigma2/python/Plugins/Extensions/Series2Folder.

But even if that were the case I don't think it explains kenmax's original problem.

[kenmax]

I did reinstall backup straight after USB install

Have you field around with the configuration backup settings? Specifically, have you customised which files get backed up when performing a configuration backup?

no I haven't played with backup settings.
I will do a reinstall again and not do reinstall of back up and see how I go.

[kenmax]

All good, did a USB install with 17.5.20180417, did not recover backup, installed StoF v1.6, it worked, installed StoF v1.8, it worked!
recovered backup settings, all good! go figure!
Thanks for help.

[prl]

I'm glad it's been sorted.

It's all been a bit puzzling!

[MIL]

Trying to get head around deploying this plugin properly. There seems to be 3 x "series to folder" options in the "movie list set up". See screen shot. What do I need to select, and what to do next?

Series2folder.PNG (54.01 KiB) Viewed 6355 times

[/c]

[adoxa]

The first moves the currently selected series to a folder; the second moves all series (with two or more episodes) to respective folders; the third configures.

[prl]

with two or more episodes

Two or more by default - the number is configurable.

[MIL]

Thanks all. Appreciated.

I still can't drive this plugin though. Any help would be great. I cant see an option to drop my series into, other than "movie". See screen shots below...

Step 1:

s2f1.PNG (93.42 KiB) Viewed 6345 times

[/c]

Step 2:

s2f2.PNG (290.98 KiB) Viewed 6345 times

[/c]

Step 3:

s2f3.PNG (55.12 KiB) Viewed 6345 times

[adoxa]

The point is it creates the folder. If you select one of the "Four Corners", bring up the menu and choose "Sel Series to Folder", your four "Four Corners" episodes will be moved into the newly created "Four Corners" folder. If you choose "Series to Folder" it will also create and move (of what I can see) "Live: NRL Sunday Footy" and "My Kitchen Rules" to their respective folders.