My conclusion after all this is that I'll probably use #zstd level 1 (not the default level 3!) for #compression of my #CSV measurement data from now on:
- ultra fast compression and decompression, on par with #lz4
- nearly as good a compression ratio as #gzip level 9
- negligible RAM usage
When I need ultra small files though, e.g. for transfer over a slow connection, I'll keep using #xz level 9.
🥳 A new version of csv-diff has just been released! 🚀
docs.rs/csv-diff/latest/csv_di…
csv-diff is the fastest CSV-diffing library in the world - written in #Rust
It can compare two 1,000,000 rows x 9 columns CSVs in < 600ms!
Note that this is still a beta release and the library itself is still very young.
#RustLang #Release #CSV #CSVDiff #Performance #DataScience #Data #Diff #Difference #OpenSource #Crate
