Version 0.8.0 of R’s fst package has been released to CRAN. The package is now multi-threaded allowing for even faster serialization of data frames to disk. This post covers some of the enhancements that have been made to fst in the latest release and measures the performance against the serialization ... [Read more...]

The fst package uses LZ4 and ZSTD to compress columnar data. In the latest release, methods compress_fst and decompress_fst were added which allow for direct (multi-threaded) access to these excellent compressors. LZ4 and ZSTD LZ4 is one of the fastest compressors around, and like all LZ77-type compressors, ... [Read more...]

The fst package uses the xxHash algorithm for internal hashing of (meta-)data. With method hash_fst the user now has direct access to this extremely fast hashing algorithm. Hashing Hashing is used to map data of any size to data of a (usually much smaller) fixed size. That’s ... [Read more...]