Yes, it is a combination of delta coding, zigzag, bitpacking and outliner detection.
It only works well for integer compression. For text-based data, results are not use-full.
SIMD and decompression speed is an important aspect. All forecast APIs use the compressed files as well. Previously I was using mmap'ed float16 grids, which were faster, but took significantly more space.
It only works well for integer compression. For text-based data, results are not use-full.
SIMD and decompression speed is an important aspect. All forecast APIs use the compressed files as well. Previously I was using mmap'ed float16 grids, which were faster, but took significantly more space.