Skip to content

Commit

Permalink
Update Case_study.md
Browse files Browse the repository at this point in the history
  • Loading branch information
digitensions authored Feb 19, 2024
1 parent 191c5b8 commit 0d1450b
Showing 1 changed file with 37 additions and 16 deletions.
53 changes: 37 additions & 16 deletions Doc/Case_study.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,27 +82,48 @@ Our encoding command:
```
rawcooked -y --all --no-accept-gaps -s 5281680 <path/sequence_name/> -o <path/sequence_name.mkv> >> <path/sequence_name.mkv.txt> 2>&1
```
| Command | Description |
| ---------------------- | ------------------------------------ |
| ```rawcooked``` | Calls the software |
| ```-y``` | Answers 'yes' to software questions |
| ```-all``` | Preservation command with checksums |
| ```--no-accept-gaps``` | Exit with warning if sequence gaps |
| ```-s 5281680``` | Set max attachment size to 5MB |
| ```-o``` | Use output path for FFV1 MKV |
| ```>>``` | Capture console output to text file |
| ```2>&1``` | stderr and stdout messages captured |

This command is generally launched from within a Bash script, and is passed to [GNU Parallel](https://www.gnu.org/software/parallel/) to run multiple encodings in parallel. This software makes it very simple to run multiple encodings, just by writing all the image sequence paths to one text file you can launch a parallel command like this:

| Command | Description |
| ---------------------- | ------------------------------------------ |
| ```rawcooked``` | Calls the software |
| ```-y``` | Answers 'yes' to software questions |
| ```-all``` | Preservation command with checksums |
| ```--no-accept-gaps``` | Exit with warning if sequence gaps found |
| ```-s 5281680``` | Set max attachment size to 5MB |
| ```-o``` | Use output path for FFV1 MKV |
| ```>>``` | Capture console output to text file |
| ```2>&1``` | stderr and stdout messages captured in log |

This command is generally launched from within a Bash script, and is passed to [GNU Parallel](https://www.gnu.org/software/parallel/) to run multiple encodings in parallel. This software makes it very simple to run multiple encodings specified by the ```--job``` flag. By listing all the image sequence paths in one text file you can launch a parallel command like this to run 5 parallel encodings:
```
cat ${sequence_list.txt} | parallel --jobs 10 "rawcooked -y --all --no-accept-gaps -s 5281680 {} -o {}.mkv >> {}.mkv.txt 2>&1"
cat ${sequence_list.txt} | parallel --jobs 5 "rawcooked -y --all --no-accept-gaps -s 5281680 {} -o {}.mkv >> {}.mkv.txt 2>&1"
```

We always capture our console logs for every encoding. The ```2>&1``` ensures any error messages are output alongside the usual standard messages. These are essential for us to review if a problem is found with an encoding. Over time they also provide a very clear record of changes encountered in FFmpeg and RAWcooked software, and data of our own image sequence files. In recent years these logs have been critical in identifying unanticipated edge cases with some DPX encodings, allowing for impact assessment generation. We definitely encourage all RAWcooked users to capture and retain this information as part of their long-term preservation metadata collection.


We always capture our console logs for every encoding. The ```2>&1``` ensures any error messages are output alongside the usual standard console messages for the software. These are essential for us to review if a problem is found with an encoding. Over time they also provide a very clear record of changes encountered in FFmpeg and RAWcooked software, and data of our own image sequence files. These logs have been critical in identifying unanticipated edge cases with some DPX encodings, allowing for impact assessment by Media Area. We definitely encourage all RAWcooked users to capture and retain this information as part of their long-term preservation metadata collection for an image sequence.

---
### <a name="log_assessment">Muxing log assessment</a>

The muxing logs are critical for the automated assessment of the muxing process. Each log consists of four blocks of data:
* The RAWcooked assessment of the sequence
* The FFmpeg encoding data
* The post-muxing RAWcooked assessment of the FFV1 Matroska
* Text review of the success of the muxed sequence

The RAWcooked assessments themselves are lines of repeated data, counting from 0% to 100%. The FFmpeg muxing data contains sequence and FFV1 metadata, along with choices made by the software for the muxing process and logs of the fps for the muxing of the sequence. All this information is really important when there's an issue with the muxing. The final text review is generated by the RAWcooked assessment of the image sequence and the FFV1 Matroska. In this last section you can be given different types of human readable message including:
* Warnings about the image sequence files
* Errors experienced during muxing
* Information about the RAWcooked mux (RAWcooked version, if checksum hashes included)
* Completion success or failure statement

The automation scripts used a the the BFI National Archive largely ignore the warning messages, but look for any messages that have 'Error' in them. Once found this FFV1 Matroska is deleted and the sequence is queued for a repeated encoding attempt. Similarly if the completion statement suggests a failure then the FFV1 is deleted and the sequence is queued for a repeat encoding.

There is one error message that triggers a specific type of re-encoding:
```Error: the reversibility file is becoming big | Error: undecodable file is becoming too big```

For this error we know that we need to re-encode our image sequence with the additional flag ```--output-version 2``` which writes the large reversibility data to the FFV1 Matroska once encoding is completed. FFmpeg has an upper size limit of 1GB for attachments. If there is lots of additional data stored in your DPX file headers then this flag will ensure that your FFV1 Matroska completes fine and the data remains verifiably reversible.

---
### <a name="ffv1_valid">FFV1 Matroska validation</a>

### <a name="ffv1_demux">FFV1 Matroska demux to image sequence</a>
Expand Down

0 comments on commit 0d1450b

Please sign in to comment.