HDMI recording files being overwritten

[Paul_oz53]

Twice I have experienced files recorded on HDMI-IN being overwritten by a subsequent recording on HDMI-IN.

The sequence is:

Record an event on HDMI-In. The first event is named 2017070n - HDMI IN.
Move it to a folder (mine is called HDMI).
Now record a second event (in /media/hdd/movie/). It also is assigned the name 2017070n - HDMI IN.
At the end of the event, rename this file and move it to folder HDMI. The first file disappears!

Moreover, I have noticed while the second recording is taking place, the first file status marker changes colour to red. I suspect this indicates that it is being over-written as the second file records, despite being in a separate folder to the active recording.

Although the name clash is obvious, the fact I physically relocate the file to another directory does not appear to break the name clash or store a separate unique version of the file. Rather, it appears that it is essential to manually rename the first file to avoid the prospect of it being overwritten.

This behaviour is counter-intuitive and quite annoying - it has twice destroyed long recordings and replaced them with snippets!

I use short filenames as the default setting.

[MrQuade]

Twice I have experienced files recorded on HDMI-IN being overwritten by a subsequent recording on HDMI-IN.

A few things are confusing me about this.
The HDMI-IN recordings all get unique filenames based on their start time (down to the minute).

Are you starting the second HDMI-In recording within a minute of starting the first one?

Second, you refer to recording from HDMI-IN and then moving the file. Is the recording finished when you move the file?

[Grumpy_Geoff]

You'd be better off not moving the recordings, as any subsequent recording's files should get a "_00n" file name suffix when started.

I think that if you're renaming the recordings via the Media player, then you're only changing the recording name (in the .meta file) not the underlying file name.

[prl]

Twice I have experienced files recorded on HDMI-IN being overwritten by a subsequent recording on HDMI-IN.

A few things are confusing me about this.
The HDMI-IN recordings all get unique filenames based on their start time (down to the minute).

Not if

I use short filenames as the default setting.

and the timer is given a name, or an instant recording is made.

In that case the filename format is: YYMMDD timername.*

Similarly if an instant recording (REC or RED from live TV, then select Create instant recording) is made from HDMI with short recording names set. In that case the recording name is YYMMDD HDMI IN.*. That seems to be what Paul_oz53 is using.

Are you starting the second HDMI-In recording within a minute of starting the first one?

That's not the critical time. "The same day" is, with short recording filenames with the same timer name, or for instant recordings.

Second, you refer to recording from HDMI-IN and then moving the file. Is the recording finished when you move the file?

I doubt that that matters.

The problem is that while the code that generates recording names is smart about adding a numbered extension to the name (_001, _002 ...) to avoid conflicts, the code that moves recordings is not so smart. It just moves the recordings and if there was another recording with the same filename in the destination directory, it gets replaced by the new one. There was a similar bug reported by Grumpy_Geoff for the Series2Folder plugin.

The fix for moving recordings in the media player isn't trivial.

I'm not sure how you're renaming the recordings, because as Grumpy_Geoff says, renaming the recording in the media payer selection screen (MEDIA from live TV, then MENU>Rename) doesn't actually change the file names. Renaming the recording files in the File Commander (MENU>Files / Sources from live TV) or in the command-line interface will rename the files.

Some workarounds are:

• Use standard filenames instead of short filenames for recordings. Then recordings have names that include HHMM and you'll only have the issue for recordings started in the same clock minute (the situation MrQuade described).
• Use a normal timer without a timer name to make HDMI recordings instead of an instant recording. That will include HHMM in the recordingname even if you've configured short filenames.
• Use a named normal timer to make HDMI recordings and ensure that a different name is used for each recording that will be made on the same day. This will also help you identify the recordings.
• Only move HDMI recordings to the HDMI directory after all HDMI recordings have been made for the day.

[MrQuade]

Not if

I use short filenames as the default setting.

Ahh yes, I missed that bit at the end.

That's not the critical time. "The same day" is, with short recording filenames with the same timer name, or for instant recordings.

Yes makes sense. Given that I'd missed the earlier point.

Second, you refer to recording from HDMI-IN and then moving the file. Is the recording finished when you move the file?

I doubt that that matters.

What confused me about this was this other statement.

Moreover, I have noticed while the second recording is taking place, the first file status marker changes colour to red. I suspect this indicates that it is being over-written as the second file records, despite being in a separate folder to the active recording.

Is the media player file display so simple that any existing recording with the same filename as a current recording will do this, regardless of the fact that the existing recording is stored in another directory? That sounds like a pretty cheap way to flag active recordings.

[prl]

...
Is the media player file display so simple that any existing recording with the same filename as a current recording will do this, regardless of the fact that the existing recording is stored in another directory? That sounds like a pretty cheap way to flag active recordings.

It is, and it is. have a look at the code that updates and uses MovieList.runningTimers in Components/MovieList.py. I suspect the fact that it's the way it is shows how rarely people use short recording names. I have no idea why it's done that way rather than by using full pathnames as the key in runningTimers.

[Paul_oz53]

Thanks guys for the insights - I had started to reply to MrQuade but you're way ahead of me.

For completeness sake, the recordings were 20+ mins apart and the first was finished when I started the second.

It's clear now that:

• shortfile names are a no-no for multiple recordings,
  instant recording is not safe for use with HDMI IN; and,
  the media player rename function is really an illusion.

Usually I just make a single recording and edit out the ads later. This time I decided to be cute and record the segments between the ads.

I have also been struggling getting the recording to download to the PC. I was wondering if there may have been an issue with HDMI IN recordings not being released (unlocked) properly. However, it seems it might have been the antivirus which was locking the ts file part way through. Can't tell for sure because I changed the antivirus settings to exclude .ts files and then moved the file from the T4 end successfully.

If this recurs I'll report further.

Paul

[prl]

...
I have also been struggling getting the recording to download to the PC. I was wondering if there may have been an issue with HDMI IN recordings not being released (unlocked) properly. ...

The T series doesn't lock recording files in any way.

You might be able to get more useful advice if you are a bit more explicit about the problem than "I have also been struggling...". Like exactly what you are doing and what error messages or other undesirable behaviour you're seeing. My crystal ball burnt out long ago, in the early days of the DP series.

For example, there are at least five different ways I can think of straight off to transfer recordings from a T series PVR to a PC, and within them, several different precise ways of going about each of them.

A new topic with a precise description of the problem would probably be a good way to go, since this really is something different from the original problem in the topic title.

[Paul_oz53]

...
I have also been struggling getting the recording to download to the PC. I was wondering if there may have been an issue with HDMI IN recordings not being released (unlocked) properly. ...

The T series doesn't lock recording files in any way.

You might be able to get more useful advice if you are a bit more explicit about the problem than "I have also been struggling...". Like exactly what you are doing and what error messages or other undesirable behaviour you're seeing. My crystal ball burnt out long ago, in the early days of the DP series.

For example, there are at least five different ways I can think of straight off to transfer recordings from a T series PVR to a PC, and within them, several different precise ways of going about each of them.

A new topic with a precise description of the problem would probably be a good way to go, since this really is something different from the original problem in the topic title.

I should know better than to mention something which I have sorted out for now!

Normally, I sit at the PC screen and use windows explorer to copy files from the T3/T4 mapped movie folders to the PC, using drag and drop. I then edit them if they need it, sometimes convert them to DVD or mp4 or just store them on the PC.

For the HDMI IN recordings I was getting a windows error that the download failed "because another process has locked part of the file". My best guess is that process might have been microsoft security essentials, but I really don't know. All I do know is that attempting to copy from the T4 using the media player copy function failed but 'move' did work. As I said, if this becomes a recurring issue, I'll report back. It doesn't affect FTA recordings to date, only HDMI IN, leaving me to wonder why the difference.

P.

[prl]

... All I do know is that attempting to copy from the T4 using the media player copy function failed but 'move' did work. ...

That's very odd, because a move of recordings from one device to another does a copy then removes the originals.

[Star6key]

...
It's clear now that:
...
instant recording is not safe for use with HDMI IN;
...

This is not correct.
I have recorded many 100's of files using HDMI In using instant recording (I also record back to back on some occasions), and I rename them in the media player. I have never 'lost' a file.
I also use 'short filenames' for the recordings.
The only thing I don't do, is move them to other folders on the T4.

[Paul_oz53]

...
It's clear now that:
...
instant recording is not safe for use with HDMI IN;
...

This is not correct.
I have recorded many 100's of files using HDMI In using instant recording (I also record back to back on some occasions), and I rename them in the media player. I have never 'lost' a file.
I also use 'short filenames' for the recordings.
The only thing I don't do, is move them to other folders on the T4.

Fascinating but the problem I described is real if, like me, you store related files in a sub-folder before processing them. The key difference between us is I move the HDMI IN recording to another folder as they are made but you do not. Moving the file turns out to be unsafe if one does this before all HDMI IN recording is completed.

This is because if you use instant recording, the first HDMI IN filename is not unique. Date - HDMI IN.ts in a folder will be overwritten by another recording which is given the same name in the main recording folder. I have watched this happen more than once. The comments and advice of prl and MrQuade spell out the limitations of the media player code as it interacts with instant recording and offer suggestions as to safer ways of recording HDMI IN.

Using the media player rename function does not overcome this problem because it does not alter the filename but rather, just the tag in the meta data file.

I certainly agree that instant recording can be used with HDMI IN, but with provisos: don't try to move a recorded file until all recording of HDMI IN is finished and be aware that renaming using the media player will not work as expected to avoid the problem!

It is the unstated existence of these provisos which lead me to class instant recording as 'unsafe'. For it to be intrinsically safe, the code would not allow the duplicate filename to ever arise but prl views it as a non-trivial exercise to address the issue. So, users need to be aware of these limitations if they want to avoid this particular 'bug'.

Cheers, Paul

[Star6key]

I would never ever contemplate moving a file that is currently in use. You are just asking for trouble.
I would suggest a simple check from the fw -
File in use?
Yes - Don't allow it to be moved
No - Carry on

[MrQuade]

I would never ever contemplate moving a file that is currently in use. You are just asking for trouble.
I would suggest a simple check from the fw -
File in use?
Yes - Don't allow it to be moved
No - Carry on

He is not moving the files as they are in use. He is saying that he moves them when the recording is complete.

The problem is essentially that the media player will happily overwrite files when moving recordings into a folder that contains existing recordings with the same filename (the filename being independent of the recording name, since renaming a recording does not rename the file, it just edits the metadata)

[scotty]

I would never ever contemplate moving a file that is currently in use. You are just asking for trouble.
I would suggest a simple check from the fw -
File in use?
Yes - Don't allow it to be moved
No - Carry on

I regularly move files while they're still recording without any issues. They continue to record in the new location.

Did it accidentally the first time but since it doesn't cause problems or split files I now find it very handy.

I've been meaning to ask if the system does anything special to achieve it or whether it's a Linux thing.

Scotty

[IanB]

Geez! Such a fuss. To clarify :-

0. Being in use has nothing to do with anything!

1. When any recording is about to start, the code generates a filename. (Various options control then actual name generated.) It then checks to see if that filename already exists. If it does it appends a generation number such that the new filename is always unique. You can make as many unique recordings as you like in any set of target directories.

2. The code that moves and copies recordings does not check if the source filename already exists in the target directory. It just blindly overwrites any recording with the same filename. Yes this is an acknowledged short coming and somebody may fix it one day (it is complicated so don't hold your breath).

[Gully]

And to add to the clarification as it seems to have been lost, the use of short names is a major part of this being a problem. If the standard long names are used it would be really unlikely that 2 recordings would have the same file name.

[prl]

What IanB said. I've already described in some detail just what the cause of the problem is and some ways to avoid it.

The warning about moving actively recording files is because moves within the same volume or device won't affect the recordings, but moving an active recording to another volume or device (e.g network share or USB drive) will break the recording. On a move to another device, the state of the recording at the time of the move will get copied across, then the original recording files will be deleted. The original files will lose their reference from the directory, but will still exist while they are open, so the recording process will continue. Then when the recording finishes and the files are closed, the files will then actually be deleted. If any metadata files are created at the end of the recording, they will end up in the wrong place.

That is an issue, but it's totally unrelated to what Paul_oz53 described in the OP.

...
I regularly move files while they're still recording without any issues. They continue to record in the new location.

If you don't understand the limitations on moving active recordings, then it may well bite you one day

since it doesn't cause problems or split files I now find it very handy.

Just because it hasn't so far doesn't mean that it won't ever. See Bertrand Russell's Chicken.

I've been meaning to ask if the system does anything special to achieve it or whether it's a Linux thing.
...

It's a generic Unix (including Linux) thing and depends on Unix filesystem hard links and the behaviour of open files that have been unlinked from a directory. Moving a file is now available as an indivisible OS operation, but it's always effectively ("==" for equivalence):

mv a/b/c d/e/f == ln a/b/c d/e/f && rm a/b/c

If any processes already have a/b/c open, the open will remain valid across the operations.

Hard links only work within the same filesystem and only in filesystems that support hard links, so that will only work if a/b/c and d/e/f are in the same filesystem/volume and in a filesystem that supports hard links (e.g. not FAT).

In those cases, the move is effectively done as:

mv a/b/c d/e/f == cp a/b/c d/e/f && rm a/b/c

In that case any process that has a/b/c open will continue to be able to access it even though its directory entry has been deleted (and even if that was the last directory entry for the file), but when all opens of the file have closed, the file storage space will be recovered and marked as free space.

This behaviour has been in Unix file systems all the way back to Unix v6 in the mid 70s.

[scotty]

Thanks for the explanation Peter