www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Persistent key-value-store for D?

reply krylon <anonymous example.com> writes:
Hello everyone,

After several unsuccessful attempts over the last couple of 
years, I made a new attempt at learning D last weekend and this 
time ... something clicked. So first of all, let me say that I 
like it very much so far!

Over the years, I have fallen into the habit when learning a new 
programming language to rewrite a certain program I first started 
working on about 12 years ago. It's not so much that this program 
is super-useful (it really isn't), but that it is a good way to 
learn how a given programming language handles things like 
concurrency, data structures, networking, and user interfaces.

One part of this program requires a DBM-like database library, 
preferably one that handles concurrent read-write-access as 
transparently as possible. In prior incarnations of this program 
I have used Berkeley DB and Tokyocabinet.

I looked at the DUB package registry and asked Google quite a bit 
now, but I did not found such a package for D. So my first 
question is - did I not look hard enough? I found a 
reimplentation of QDBM [1] (the spiritual ancestor of 
Tokyocabinet), but it does not seem to handle concurrency at all. 
Are there other options along those lines? (If there was one that 
also provides transactions, that would be awesome!)

If I understand what I have read so far correctly, it is possible 
to access libraries written in C or C++ from D - in that case, I 
could just use Tokyocabinet directly, but I have not found any 
pointers on how to do this. Is this a feasible option, and if so, 
where can I find documentation on how to do this?

[1] http://forum.dlang.org/thread/mjinam$31cp$2 digitalmars.com

Thank you very much for any insights you feel like sharing with 
me,
Benjamin
Apr 26
next sibling parent reply bachmeier <no spam.net> writes:
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:

 If I understand what I have read so far correctly, it is 
 possible to access libraries written in C or C++ from D - in 
 that case, I could just use Tokyocabinet directly, but I have 
 not found any pointers on how to do this. Is this a feasible 
 option, and if so, where can I find documentation on how to do 
 this?
Welcome to the D community. I've never heard of Tokyocabinet, but I did a search and the header files look like they'd be a straightforward translation to D, for instance http://bazaar.launchpad.net/~tokyocabinet/tokyocabinet/c99-posix-trunk/view/head:/tcutil.h You can call C libraries directly. I'd suggest trying dstep on the header files as a first step. https://github.com/jacob-carlborg/dstep These resources might help: https://wiki.dlang.org/Bind_D_to_C http://dlang.org/spec/interfaceToC.html You only need to write a binding for functions that you actually want to call, which might explain why nobody has made bindings available.
Apr 26
parent krylon <anonymous example.com> writes:
On Wednesday, 26 April 2017 at 19:07:22 UTC, bachmeier wrote:
 On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:

 [...]
Welcome to the D community. [...] You can call C libraries directly. I'd suggest trying dstep on the header files as a first step. https://github.com/jacob-carlborg/dstep These resources might help: https://wiki.dlang.org/Bind_D_to_C http://dlang.org/spec/interfaceToC.html You only need to write a binding for functions that you actually want to call, which might explain why nobody has made bindings available.
Thank you! I will look into that!
Apr 27
prev sibling next sibling parent reply Matthias Klumpp <mak debian.org> writes:
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:
 [...]
 If I understand what I have read so far correctly, it is 
 possible to access libraries written in C or C++ from D - in 
 that case, I could just use Tokyocabinet directly, but I have 
 not found any pointers on how to do this. Is this a feasible 
 option, and if so, where can I find documentation on how to do 
 this?
I can recommend using LMDB[1] which likely does all you want. It also has D bindings[2]. We use it in the AppStream generator with multithreaded parallelization and it is working great so far (although for the usecase of asgen it's not an ideal database choice, but one which makes deployments of the generator really simple). I explored other solutions including Tokyokabinet and LevelDB and LMDB was by far the fastest. Cheers, Matthias [1]: https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database [2]: https://github.com/ximion/appstream-generator/blob/master/src/asgen/bindings/lmdb.d
Apr 26
parent krylon <anonymous example.com> writes:
On Thursday, 27 April 2017 at 01:56:12 UTC, Matthias Klumpp wrote:
 On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:
 [...]
 If I understand what I have read so far correctly, it is 
 possible to access libraries written in C or C++ from D - in 
 that case, I could just use Tokyocabinet directly, but I have 
 not found any pointers on how to do this. Is this a feasible 
 option, and if so, where can I find documentation on how to do 
 this?
I can recommend using LMDB[1] which likely does all you want. It also has D bindings[2]. [...] Cheers, Matthias [1]: https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database [2]: https://github.com/ximion/appstream-generator/blob/master/src/asgen/bindings/lmdb.d
Thank you very much, that sounds exactly like what I was looking for!
Apr 27
prev sibling parent reply yawniek <yawniek srtnwz.com> writes:
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:
 I looked at the DUB package registry and asked Google quite a 
 bit now, but I did not found such a package for D. So my first 
 question is - did I not look hard enough? I found a 
 reimplentation of QDBM [1] (the spiritual ancestor of 
 Tokyocabinet), but it does not seem to handle concurrency at 
 all. Are there other options along those lines? (If there was 
 one that also provides transactions, that would be awesome!)

 If I understand what I have read so far correctly, it is 
 possible to access libraries written in C or C++ from D - in 
 that case, I could just use Tokyocabinet directly, but I have 
 not found any pointers on how to do this. Is this a feasible 
 option, and if so, where can I find documentation on how to do 
 this?
i recommend leveldb http://code.dlang.org/packages/d-leveldb its easy to use and mostly faster than tokyocabinet ( only very specifically tuned tokyo btrees outperform leveldb) i used above library with great success. it also shows you how to do c bindings.
Apr 28
parent krylon <anonymous example.com> writes:
On Friday, 28 April 2017 at 16:01:58 UTC, yawniek wrote:
 On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote:
 I looked at the DUB package registry and asked Google quite a 
 bit now, but I did not found such a package for D. So my first 
 question is - did I not look hard enough? I found a 
 reimplentation of QDBM [1] (the spiritual ancestor of 
 Tokyocabinet), but it does not seem to handle concurrency at 
 all. Are there other options along those lines? (If there was 
 one that also provides transactions, that would be awesome!)

 If I understand what I have read so far correctly, it is 
 possible to access libraries written in C or C++ from D - in 
 that case, I could just use Tokyocabinet directly, but I have 
 not found any pointers on how to do this. Is this a feasible 
 option, and if so, where can I find documentation on how to do 
 this?
i recommend leveldb http://code.dlang.org/packages/d-leveldb its easy to use and mostly faster than tokyocabinet ( only very specifically tuned tokyo btrees outperform leveldb) i used above library with great success. it also shows you how to do c bindings.
Thank you for the suggestion! For the moment I am using LMDB, as suggested in another reply, and it works well. LevelDB seems to have a cleaner interface, though, at least in D. I'll keep that in mind for future projects.
Apr 28