Feature request: Support for CDB64?

[lemondevxyz]

CDB has a file size limit of 4 GB which is a part of the file specification. There is a library that makes some of the 32 bit integers, the ones responsible for the 4 GB limit, into 64 bit integers thereby erasing the file limit.

Here's a link to the cdb64 library.

The use case for this feature request is multiple hosts. Imagine you have multiple hosts that use high quality images (4 MB per image) and the 4 GB is quickly reached when applied to multiple hosts.

I can implement this feature request myself.

[cipriancraciun]

I know about the various 64 bit CDB alternatives (including the one you've mentioned), however (at least at the time I've done my initial assesment of them) none of them supported mmap-ed databases. In fact neither does the "plain" CDB one (at least the Go port), which I had to patch to make it support mmap (the fork is here https://github.com/cipriancraciun/go-cdb-lib).

In fact, see this issue raised on the project that you've mentioned, which actually points to my initial issue on the "plain" CDB Go library:

• Add support for mmap-backed CDB reader chrislusf/cdb64#7 (still open)
• Add support for mmap-backed CDB reader (yielding a x35 improvement) colinmarc/cdb#11 (still open)

---

Why is mmap important?

Kawipiko's main focus is raw performance. Go has a garbage-collected run-time. These two don't play nice with one another.

Using read syscalls requires (new) buffers, which then are handed to the HTTP library as the response, which are then "dropped", and thus transformed into "garbage" that the Go runtime has to "collect". This heavily impacts performance. (Buffer pools are a solution, but both the CDB library and the HTTP library should support them, which unfortunately none of them do.)

Another minor observation is that using mmap eliminates the need of syscalls, thus improving the performance a bit.

As a side-note, Kawipiko doesn't use Go's HTTP implementation (for HTTP/1.1) but instead it uses https://github.com/valyala/fasthttp which is carefully implemented to eliminate as much as possible the need for the Go runtime to execute garbage collection. (I do use Go's HTTP implementation for HTTP/2, and that is why the performance for HTTP/1.1 is far better than HTTP/2.)

Getting back to mmap, because the database is mmap-ed, there is no extra buffer allocation, no extra garbage-collection, and thus the performance is improved.

(In fact, I spent perhaps more time on profiling Kawipiko to make sure the Go garbage collector doesn't kick-in than on actually writing Kawipiko.)

---

Thus to summarize, using a 64 bit variant of CDB is doable, but only if one forks and patches one of the existing libraries to support mmap.

(I'll open a discussion thread about a different approach.)

[cipriancraciun]

@lemondevxyz also see discussion #14 which might be an alternative.

[bsergean]

Hi there, my company just released a library with a golang binding which is similar in spirit to CBD, and with 64 bits support.

https://github.com/AppLovin/AxonCache

I'd really like to benchmark it against cbd library, so I guess I would need to use your patch/fork (it uses mmap as well under the hood, and is a bit of a hash-table).