docs: Provide some performance indicators for Snagfactory

Provide some recent Snagfactory benchmark results to give potential users an
idea of expected performances.

Signed-off-by: Romain Gantois <[email protected]>

README.md

@@ -135,6 +135,8 @@ For recovering and flashing large batches of boards efficiently, you may use the
 
 Note that Snagfactory support is only included in the "gui" package variant: `pip install snagboot[gui]`
 
+Some [benchmark results](https://github.com/bootlin/snagboot/blob/main/docs/snagfactory_benchmarks.md) are provided in the Snagfactory docs.
+
 If you encounter issues, please take a look at the
 [troubleshooting](https://github.com/bootlin/snagboot/blob/main/docs/troubleshooting.md) section.
 

docs/snagfactory_benchmarks.md

@@ -0,0 +1,126 @@
+# Snagfactory benchmarks
+
+To give an idea of how Snagfactory can perform in terms of flashing speed and parallelization of tasks, some benchmarks results are provided below. The time taken by Snagfactory to complete a factory flashing task depends on many different factors, including but not limited to:
+
+ - Processing capabilities of the host machine
+ - Current system load
+ - Processing capabilities of the target boards
+ - Speeds of the USB links from host to targets
+ - Total amount of image data to write to each target
+ - USB bus topology
+ - Type of target storage used
+
+The following benchmarks are meant to provide some basic performance indicators for various types of storage devices.
+
+# Measurement protocol:
+
+Host machine CPU: Intel i5-13500
+
+Host machine memory: 8.00 GB
+
+Operating system: Windows 11
+
+Snagboot version: 2.1 with a minor patch to raise loglevels to "debug" and increase snagflash USB transfer timeouts to 8 minutes
+
+1. Snagfactory is used to flash a randomly generated 64MB image to the chosen storage device on the target.
+2. Log timestamps are read from the standard logfiles generated by Snagfactory, and the following durations are computed:
+
+ - Total Snagfactory runtime: from `Start` to `BoardPhase.FLASHER -> BoardPhase.DONE`
+ - Time spent running snagrecover: from `Installing firmware tiboot3` to `Done installing firmware u-boot`
+ - Time spent running snagflash: from `Fastboot object` to last occurrence of `[DEBUG   ] fastboot OKAY`
+ - Time spent erasing flash blocks (for SPI-NAND and SPI-NOR only): from last occurrence of `erasing flash area` to following occurence of `fastboot OKAY`
+ - Time spent transmitting image data over USB: from `flashing file random.img` to following occurrence of `fastboot OKAY`
+ - Time spent writing data to storage medium: from preceding log line to following occurence of `fastboot OKAY`
+
+3. Throughput values are then derived in the following manner:
+
+ - transmission throughput: size of flashed image / time spent transmitting over USB
+ - write throughput: size of flashed image / (time spent writing to storage + time spent erasing storage)
+
+
+For each type of storage, this measurement protocol was performed three times
+with different boards of the same model.
+
+
+A series of additional total runtime measurements were also performed for the
+SPI-NAND case with increasing numbers of boards, to showcase Snagfactory's
+ability to parallelize tasks.
+
+# eMMC
+
+Board: AM62x SKEVM
+
+Storage: Micron MTFC16GAPALBH-IT 16GB eMMC
+
+|total runtime (s)|snagrecover runtime (s)|snagflash runtime (s)|
+|---|---|---|
+|18|9|4|
+|18|9|4|
+|18|9|4|
+
+
+
+NB: The discrepancy between total Snagfactory runtime, and the sum of the other two runtimes is mostly due to the USB reenumeration delay after U-Boot proper is launched on the target
+
+
+
+|transmission throughput (MB/s)| write throughput (MB/s)|
+|---|---|
+|24.701|56.288|
+|24.559|57.971|
+|24.578|56.437|
+
+# SPI-NAND
+
+Board: AM62x Low-power SKEVM
+
+Storage: Winbond W35N01JWTBAG 128MB OSPI-NAND
+
+|total runtime (s)|snagrecover runtime (s)|snagflash runtime (s)|
+|---|---|---|
+|25|10|11|
+|25|9|11|
+|24|9|10|
+
+
+|transmission throughput (MB/s)| write throughput, including erase cycles (MB/s)|
+|---|---|
+|24.568|8.854|
+|24.530|8.443|
+|24.682|9.100|
+
+# OSPI-NOR
+
+Board: AM62x SKEVM
+
+Storage: Infineon S28HS512TGABHM010 64MB OSPI-NOR
+
+|total runtime (s)|snagrecover runtime (s)|snagflash runtime (s)|
+|---|---|---|
+|426|9|411|
+|401|9|387|
+|390|9|375|
+
+
+|transmission throughput (MB/s)| write throughput, including erase cycles (MB/s)|
+|---|---|
+|25.257|0.157|
+|24.625|0.167|
+|24.806|0.172|
+
+# Parallelization
+
+Board: AM62x Low-power SKEVM
+
+Storage: Winbond W35N01JWTBAG 128MB OSPI-NAND
+
+|number of boards|total runtime (s)|
+|---|---|
+|1|25|
+|3|25|
+|4|26|
+|7|27|
+|8|27|
+
+NB: these results were obtained with "native" USB ports.
+