Skip to main content

Search

Items tagged with: lz4

Repeated the #compression #benchmark with the same file on a beefier machine (AMD Ryzen 9 5950X), results are quite identical, except faster overall.

This plot is also interesting:

- #gzip and #lz4 have fixed (!) and very low RAM usage across levels and compression/decompression
- #xz RAM usage scales with the level from a couple of MBs (0) to nearly a GB (9)
- #zstd RAM usage scales weirdly with level but not as extreme as #xz

#Python #matplotlib

Plot RAM at compression vs RAM at decompression for different compression algorithms. gzip very low RAM, lz4 a little more, xz increasing strongly with the level, zstd not as prominently
#python #compression #zstd #xz #matplotlib #gzip #lz4 #benchmark

First let's look only at the non-fancy options (no --fast or multithreading) and make log-log plots to better see what's happening in the 'clumps' of points. Points of interest for me:

- #gzip has a *really* low memory footprint across all compression levels
- #zstd clearly wins the decompression speed/compression ratio compromise!
- #xz at higher levels is unrivalled in compression ratio
- #lz4 higher levels aren't worth it. #lz4 is also just fast.

plot
plot
plot
plot
#zstd #xz #gzip #lz4

⏱️ Compression vs Decompression speed. Interesting that #zstd has some settings that actually decompress slower than they compress. Might just be my machine, though... #xz is still a turtle 🐢, #gzip is not much better, #lz4 decompresses much faster than is compressed, #zstd again has something for everybody.
Plot of compression vs decompression speed. In general, all compression algorithms decompress faster than they compress, except for zstd which in this case has some settings that decompress slower than they compress.
#zstd #xz #gzip #lz4

My conclusion after all this is that I'll probably use #zstd level 1 (not the default level 3!) for #compression of my #CSV measurement data from now on:

- ultra fast compression and decompression, on par with #lz4
- nearly as good a compression ratio as #gzip level 9
- negligible RAM usage

When I need ultra small files though, e.g. for transfer over a slow connection, I'll keep using #xz level 9.

plot, emoji finger pointing to zstd level 1 being fast and nearly as good a ratio as gzip -9
plot, emoji finger pointing at zstd level 1 having reasonably low RAM usage
plot, emoji finger pointing at zstd level 1 being ultra fast
#csv #compression #zstd #xz #gzip #lz4

⇧