Fast Base64 stream encoder/decoder
This is an implementation of a base64 stream encoding/decoding library in C99 with SIMD (AVX2, AVX512, NEON, AArch64/NEON, SSSE3, SSE4.1, SSE4.2, AVX) and OpenMP acceleration. It also contains wrapper functions to encode/decode simple length-delimited strings. This library aims to be:
- FAST;
- easy to use;
- elegant.
On x86, the library does runtime feature detection. The first time it's called, the library will determine the appropriate encoding/decoding routines for the machine. It then remembers them for the lifetime of the program. If your processor supports AVX2, SSSE3, SSE4.1, SSE4.2 or AVX instructions, the library will pick an optimized codec that lets it encode/decode 12 or 24 bytes at a time, which gives a speedup of four or more times compared to the "plain" bytewise codec.
AVX512 support is only for encoding at present, utilizing the AVX512 VL and VBMI instructions. Decoding part reused AVX2 implementations. For CPUs later than Cannonlake (manufactured in 2018) supports these instructions.
NEON support is hardcoded to on or off at compile time, because portable runtime feature detection is unavailable on ARM.
Even if your processor does not support SIMD instructions, this is a very fast library. The fallback routine can process 32 or 64 bits of input in one round, depending on your processor's word width, which still makes it significantly faster than naive bytewise implementations. On some 64-bit machines, the 64-bit routines even outperform the SSSE3 ones.
To the author's knowledge, at the time of original release, this was the only Base64 library to offer SIMD acceleration. The author wrote an article explaining one possible SIMD approach to encoding/decoding Base64. The article can help figure out what the code is doing, and why.
Notable features:
- Really fast on x86 and ARM systems by using SIMD vector processing;
- Can use OpenMP for even more parallel speedups;
- Really fast on other 32 or 64-bit platforms through optimized routines;
- Reads/writes blocks of streaming data;
- Does not dynamically allocate memory;
- Valid C99 that compiles with pedantic options on;
- Re-entrant and threadsafe;
- Unit tested;
- Uses Duff's Device.
Acknowledgements
The original AVX2, NEON and Aarch64/NEON codecs were generously contributed by Inkymail, who, in their fork, also implemented some additional features. Their work is slowly being backported into this project.
The SSSE3 and AVX2 codecs were substantially improved by using some very clever optimizations described by Wojciech Muła in a series of articles. His own code is here.
The AVX512 encoder is based on code from Wojciech Muła's base64simd library.
The OpenMP implementation was added by Ferry Toth (@htot) from Exalon Delft.
Building
The lib directory contains the code for the actual library. Typing make in the toplevel directory will build lib/libbase64.o and bin/base64. The first is a single, self-contained object file that you can link into your own project. The second is a standalone test binary that works similarly to the base64 system utility.