Skip to content

Zstandard v1.5.7

Latest
Latest
Compare
Choose a tag to compare
@Cyan4973 Cyan4973 released this 19 Feb 21:50
· 5 commits to dev since this release
v1.5.7
f8745da
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Zstandard v1.5.7 is a significant release, featuring over 500 commits accumulated over the past year. This update brings enhancements across various domains, including performance, stability, and functionality, and is particularly recommended for 32-bit users due to a bug fix detailed below.

edit: man pages were not properly updated in this package, and are still pointing to v1.5.6. If you want the v1.5.7 version, they are updated in dev branch, and can be cherry-picked using commit 6af3842. Alternatively, they can also be built locally using make manual and make -C programs man.

Performance Enhancements

Enhanced Compression Speed for Small Data Blocks

The compression speed for small data blocks has been notably improved at fast compression levels, thanks to contributions from @TocarIP, further extended in #4165. Below are benchmark results illustrating the performance improvements for level 1 compression on the Silesia corpus, segmented into different block sizes:

Block Size v1.5.6 v1.5.7 Improv.
4 KB 280 MB/s 310 MB/s +10%
8 KB 328 MB/s 364 MB/s +11%
16 KB 360 MB/s 396 MB/s +10%
32 KB 383 MB/s 458 MB/s +20%
64 KB 423 MB/s 474 MB/s +12%
128 KB 460 MB/s 485 MB/s +5%

These improvements are particularly beneficial in data centers and databases, where data is often divided into small segments to enhance seekability. A notable example is RocksDB, which is frequently configured with block sizes around 16 KB. Such systems will experience an immediate performance boost by updating their compression library to v1.5.7.

Additionally, the speed improvements partially extend to dictionary compression, with observed gains ranging from none to +15%.

Substantial --patch-from improvements

The --patch-from functionality of the zstd CLI allows for differential compression of a file given an older version, similar to bsdiff. While compression levels impact processing speed and compression ratio as expected, at high compression levels (18+), speed was previously reduced to uncomfortable levels.

v1.5.7 largely mitigates the speed impact of high compression levels 18+, as illustrated in the following scenario, which compresses linux-6.13.tar as a difference from linux-6.12.tar using 6 threads:

level v1.5.6 v1.5.7 Improv.
18 210 s 42.7 s x4.9
19 273 s 61.8 s x4.4
20 310 s 82.3 s x3.8
21 365 s 120 s x3.1

Additionally, compression ratio of --patch-from is also substantially improved across the board, including higher compression levels:

level v1.5.6 v1.5.7 Improv.
15 6,383,341 6,003,514 6.3%
16 5,909,932 5,421,444 9.0%
17 5,613,879 5,214,734 7.7%
18 5,584,219 5,075,507 10.0%
19 5,322,658 4,724,116 12.6%
20 5,097,236 4,583,435 11.2%
21 4,648,583 4,297,672 8.1%
22 4,560,051 4,225,283 7.9%
--max N/A 3,999,476

Compression ratio improvements for large files

The compression ratio has been enhanced slightly for large data across all compression levels, thanks to a refined approach in selecting block boundaries. This improvement is more pronounced at lower compression levels (≤ 15), where the previous boundary selection method was relatively basic. In contrast, higher compression levels (16+) already employed some analysis, but still benefit from the introduction of the new heuristic, resulting in marginally better compression ratios.

A comparison of the compression ratios between v1.5.6 and v1.5.7 is presented below:

level v1.5.6 v1.5.7 Improv.
1 73,492,125 73,292,801 -199,324
3 66,555,241 66,217,754 -337,487
6 61,614,472 61,350,571 -263,901
8 60,089,890 59,816,878 -273,012
12 58,297,345 58,039,924 -257,421
15 57,290,517 57,031,235 -259,282
16 55,341,618 55,304,857 -36,761
19 52,876,557 52,840,800 -35,757
22 52,326,420 52,289,718 -36,702

Command-Line Interface Updates

Recognizing the prevalence of multi-core systems, zstd now employs multiple threads by default, thanks to #4211 by @daniellerozenblit
In v1.5.7, systems will see default threading scaled according to capability, but remains cautious to mitigate risks of overloading surprises on existing user base, with a maximum of 4 threads by default.
For example, a typical "low-end" system with 4 cores and hyperthreading will employ 2 threads by default. A "higher-end" system with 8 cores and hyperthreading will employ 4 threads by default.

These modifications only impact default settings. As usual, the number of threads that zstd uses can also be set manually, either via command line or via environment variables.

Additionally, a new zstd command --max is available, designed to maximize compression ratios by leveraging all parameters, at the expense of time and memory. It's slower and more resource hungry than --ultra -22, but achieves denser compressed archives if that's the goal. For instance, compressing enwik9 (1,000,000,000 bytes) at --max level on a i7-9700k platform yields:

level cSize cTime cMem
--ultra -22 213,898,083 ~6 mn ~ 3 GB
--max 202,507,076 ~72 mn ~ 18 GB

Bug Fixes

v1.5.7 resolves a long standing and very rare compression issue in 32-bit mode, which can be triggered during long-lasting sessions (same ZSTD_CCtx* being continuously reused). This issue required several conditions to be met and a large amount of data to occur, but at the scale of zstd usage, it has been observed and reported (#4292).
In summary, if you use libzstd in 32-bit mode, this update is highly recommended.

Miscellaneous

The release also brings its lot of various improvements, impacting build scripts (cmake, meson, Apple Framework, Visual Studio), documentation and portability (QNX, FreeBSD, Hurd, icc/icx).
Notably, there are measurable performance gains for binaries produced with Visual Studio, as a cumulative result of several contributions.

Changelog:

  • fix: compression bug in 32-bit mode associated with long-lasting sessions
  • perf: substantial compression speed improvements (up to +30%) on small data, by @TocarIP (#4144) and @Cyan4973 (#4165)
  • perf: improved compression speed (~+5%) for dictionary compression at low levels (#4170)
  • perf: much faster --patch-from at high compression levels (#4276)
  • perf: higher --patch-from compression ratios, notably at high levels (#4288)
  • perf: better speed for binaries on Windows (@pps83) and when compiled with Visual Studio (@MessyHack)
  • perf: slightly improve compression ratio across all levels thanks to better block boundaries (#4136, #4176, #4178)
  • perf: slight compression ratio improvement for dfast, aka levels 3 and 4 (#4171)
  • perf: runtime bmi2 detection now also enabled on x86 32-bit mode (#4251)
  • cli: multi-threading as default CLI setting, by @daniellerozenblit
  • cli: new --max command (#4290)
  • api: new method ZSTD_compressSequencesAndLiterals() (#4217, #4232)
  • api: ZSTD_getFrameHeader() now works on skippable frames (#4228)
  • build: improve msbuild version autodetection, support VS2022, by @ManuelBlanc
  • build: fix meson build by @artem and @Victor-C-Zhang, and on Windows by @bgilbert
  • build: compatibility with Apple Framework, by @Treata11
  • build: improve icc/icx compatibility, by @josepho0918 and @luau-project
  • build: improve compatibility with Android NDK, by Adenilson Cavalcanti
  • portability: linux kernel branch, with improved support for Sequence producers (@embg, @gcabiddu, @Cyan4973)
  • portability: improved qnx compatibility, suggested by @rainbowball
  • portability: improved install script for FreeBSD, by @sunpoet
  • portability: fixed test suite compatibility with gnu hurd, by @diegonc
  • doc: clarify specification, by @elasota
  • misc: improved tests/decodecorpus validation tool (#4102), by antmicro

New Contributors

Full Changelog: v1.5.6...v1.5.7

Contributors

lukaso, proppy, and 44 other contributors
Assets 10
Loading