My conclusion after all this is that I'll probably use #zstd level 1 (not the default level 3!) for #compression of my #CSV measurement data from now on:

- ultra fast compression and decompression, on par with #lz4
- nearly as good a compression ratio as #gzip level 9
- negligible RAM usage

When I need ultra small files though, e.g. for transfer over a slow connection, I'll keep using #xz level 9.

🥳 A new version of csv-diff has just been released! 🚀

https://docs.rs/csv-diff/latest/csv_diff/

csv-diff is the fastest CSV-diffing library in the world - written in #Rust

It can compare two 1,000,000 rows x 9 columns CSVs in < 600ms!

Note that this is still a beta release and the library itself is still very young.

#RustLang #Release #CSV #CSVDiff #Performance #DataScience #Data #Diff #Difference #OpenSource #Crate