www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - SQLite 3.6.23.1 wrapper + connector

reply awishformore <awishformore gmail.com> writes:
Hello there.

I've converted the .h file of the latest SQLite version to .d and I 
thought I'd let the world know (as suggested on IRC ;). Maybe it will 
save someone some work.

I've also written a nice connector class that wraps all the C functions 
for convenient use in your application. It's very basic, but I will 
probably add more features (like automatic preparation of statements and 
automatic caching of several prepared statements) later.

For the time being both files are included in the download: 
http://nexunity.com/sqlite4d.rar

I'm pretty new to this kind of stuff, so what I did to get it to work 
was compiling the latest SQLite dll from source with dmc and then link 
the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

I'm sure there is a better way like compiling it as static lib (dmc 
complained about no entry point) or having some kind of other file to 
link into your app in order for it to compile and then use the dll. I 
however couldn't figure it out and it works for me. Don't hesitate to 
teach me nonetheless.

Any kind of feedback is always appreciated.

Greetings, Max.
Jul 18 2010
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from awishformore (awishformore gmail.com)'s article
 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.
Awesome. D1, D2 or both? I've wanted a simple database for some time now, but been too lazy to write bindings or find one one myself, which encourages me to roll my own ad-hoc formats instead. Given that sqlite is in the public domain, maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so that people can use it w/o creating dependency hell in their projects.
Jul 18 2010
next sibling parent reply BLS <windevguy hotmail.de> writes:
On 19/07/2010 00:57, dsimcha wrote:
   Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so
 that people can use it w/o creating dependency hell in their projects.
Agreed, even a very popular and commercial multi OS RAD Tool called "Real Basic" comes bundled with SQLite. Let's use this excellent tool instead of the thick openrj stuff, we had before. -bjoern
Jul 18 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 06:22 PM, BLS wrote:
 On 19/07/2010 00:57, dsimcha wrote:
 Given that sqlite is in the public domain,
 maybe Phobos should eventually include SQLite + a nice D-ish wrapper
 for it, so
 that people can use it w/o creating dependency hell in their projects.
Agreed, even a very popular and commercial multi OS RAD Tool called "Real Basic" comes bundled with SQLite. Let's use this excellent tool instead of the thick openrj stuff, we had before. -bjoern
I thought Qt does, too. Python definitely does, and I've actually been using it a lot lately. One thing I noticed when switching my code from postgresql to sqlite is that their respective python libraries use different parameter styles. Otherwise, it was completely painless. I'd love to see sqlite distributed with D, but not before a good DB API specification (not to disparage awishformore's work, of course).
Jul 18 2010
prev sibling next sibling parent awishformore <awishformore gmail.com> writes:
On 19/07/2010 00:57, dsimcha wrote:
 == Quote from awishformore (awishformore gmail.com)'s article
 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.
Awesome. D1, D2 or both? I've wanted a simple database for some time now, but been too lazy to write bindings or find one one myself, which encourages me to roll my own ad-hoc formats instead. Given that sqlite is in the public domain, maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so that people can use it w/o creating dependency hell in their projects.
Hello. Ah yeah, I guess I should have mentioned. I'm only working with D2 at the moment, so it is D2 right now (because of the use of string). However, it should be straight forward to replace all string occurrences with char[] to get it to work on D1. I'm pretty sure that's the only issue stopping it from working on D1. Also, when using my wrapper, please note a few specifics with the bind method: - to bind a zeroblob to a prepared statement, pass an int array with a single element with the size of the blob as value. Couldn't come up with a better way to make that function available through the same method. - to bind NULL just pass null ;) - currently, it lets you pass anything and will try to treat it as sqlite3_value struct reference, so be careful Greetings, Max.
Jul 18 2010
prev sibling parent Charles Hixson <charleshixsn earthlink.net> writes:
On 07/18/2010 03:57 PM, dsimcha wrote:
 == Quote from awishformore (awishformore gmail.com)'s article
 Hello there.
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
... for me. Don't hesitate to
 teach me nonetheless.
 Any kind of feedback is always appreciated.
 Greetings, Max.
Awesome. D1, D2 or both? I've wanted a simple database for some time now, but been too lazy to write bindings or find one one myself, which encourages me to roll my own ad-hoc formats instead. Given that sqlite is in the public domain, maybe Phobos should eventually include SQLite + a nice D-ish wrapper for it, so that people can use it w/o creating dependency hell in their projects.
Personally, I think it would be wonderful if Phobos included an SQLite class, or even just a B+Tree class. (Often I don't want the entire weight of SQL, and just use it because it's so difficult to get access to a B+Tree.)
Jul 19 2010
prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 05:44 PM, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
btw, would it be possible to distribute as something other than a rar? I wanted to have a peek at what you've done, but apparently nothing on my [linux] system can read that file.
Jul 18 2010
parent reply Jonathan M Davis <jmdavisprog gmail.com> writes:
On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:
 On 07/18/2010 05:44 PM, awishformore wrote:
 Hello there.
 
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
btw, would it be possible to distribute as something other than a rar? I wanted to have a peek at what you've done, but apparently nothing on my [linux] system can read that file.
While I hate rar, every distribution than I have used has had unrar. There's a decent chance that it isn't installed by default though. - Jonathan M Davis
Jul 18 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 09:20 PM, Jonathan M Davis wrote:
 On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:
 btw, would it be possible to distribute as something other than a rar? I
 wanted to have a peek at what you've done, but apparently nothing on my
 [linux] system can read that file.
While I hate rar, every distribution than I have used has had unrar. There's a decent chance that it isn't installed by default though. - Jonathan M Davis
Hmm. I suppose I could have spent more than 2 seconds trying to open it. You're right on both counts. As for the code, I'm looking at SQLiteConnector.ptrtostr. Does reserve really work that way? Just curious - I wouldn't know one way or the other, although I would probably use something more like string ptrtostr(char* ptr, int size = 0){ return ptr ? ptr[0 .. size].idup : null; }
Jul 18 2010
parent reply awishformore <awishformore gmail.com> writes:
On 19/07/2010 04:59, Ellery Newcomer wrote:
 On 07/18/2010 09:20 PM, Jonathan M Davis wrote:
 On Sunday 18 July 2010 19:10:35 Ellery Newcomer wrote:
 btw, would it be possible to distribute as something other than a rar? I
 wanted to have a peek at what you've done, but apparently nothing on my
 [linux] system can read that file.
While I hate rar, every distribution than I have used has had unrar. There's a decent chance that it isn't installed by default though. - Jonathan M Davis
Hmm. I suppose I could have spent more than 2 seconds trying to open it. You're right on both counts. As for the code, I'm looking at SQLiteConnector.ptrtostr. Does reserve really work that way? Just curious - I wouldn't know one way or the other, although I would probably use something more like string ptrtostr(char* ptr, int size = 0){ return ptr ? ptr[0 .. size].idup : null; }
I think an empty array/string is safer to return than null. And yes, reserve works that way, it will reserve the space needed for the (character) array to expand. The strings are null terminated in SQLite and you won't always know the size; I had previously used the same method in another place of my app. I however also wasn't aware that you could still access pointers like arrays in D, so that would indeed be a better way to do it when you always know the size. You could then even go as far as not iduping, but instead casting it to a string to avoid unnecessary allocations. Greetings, Max.
Jul 18 2010
parent Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/18/2010 11:20 PM, awishformore wrote:
 I think an empty array/string is safer to return than null.
It doesn't matter. D treats 'null' arrays exactly the same as empty arrays. Just think of a 'null' array as {length=0, ptr=null}
Jul 18 2010
prev sibling next sibling parent reply Johannes Pfau <spam example.com> writes:
On 19.07.2010 00:44, awishformore wrote:
 Hello there.
 
 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.
 
 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.
 
 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar
 
 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).
 
 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.
 
 Any kind of feedback is always appreciated.
 
 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?) -- Johannes Pfau
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Jul 19 2010
parent reply "Rory McGuire" <rmcguire neonova.co.za> writes:
On Mon, 19 Jul 2010 20:39:35 +0200, awishformore <awishformore gmail.com>  
wrote:

 Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements  
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Did you test with a string that was not in the code itself, e.g. from a config file? String literals are null terminated so you wouldn't have had an issue if all your strings were literals. Utf8 doesn't contain the string length, so you will run in to problems eventually. You have to use toStringz or your own null terminator. Unless of course you know that the function will always be taking string literals. But even then leaving something like that up to the programmer to remember is not exactly fool proof. Enjoy. ~Rory
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 19.07.2010 21:34, schrieb Rory McGuire:
 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Did you test with a string that was not in the code itself, e.g. from a config file? String literals are null terminated so you wouldn't have had an issue if all your strings were literals. Utf8 doesn't contain the string length, so you will run in to problems eventually. You have to use toStringz or your own null terminator. Unless of course you know that the function will always be taking string literals. But even then leaving something like that up to the programmer to remember is not exactly fool proof. Enjoy. ~Rory
Hey again and thanks for the hint. I tried finding something on the DM page about string literals being null terminated and while the section about string literals didn't even mention it, it was said some place else. That explains why using string literals works even though I expected it to fail. It's indeed good to know and adding std.string.toStringz is probably a good idea ;). Thanks. Greetings, Max.
Jul 19 2010
parent reply "Rory McGuire" <rmcguire neonova.co.za> writes:
On Tue, 20 Jul 2010 01:37:10 +0200, awishformore <awishformore gmail.com>  
wrote:

 Am 19.07.2010 21:34, schrieb Rory McGuire:
 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C  
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then  
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Did you test with a string that was not in the code itself, e.g. from a config file? String literals are null terminated so you wouldn't have had an issue if all your strings were literals. Utf8 doesn't contain the string length, so you will run in to problems eventually. You have to use toStringz or your own null terminator. Unless of course you know that the function will always be taking string literals. But even then leaving something like that up to the programmer to remember is not exactly fool proof. Enjoy. ~Rory
Hey again and thanks for the hint. I tried finding something on the DM page about string literals being null terminated and while the section about string literals didn't even mention it, it was said some place else. That explains why using string literals works even though I expected it to fail. It's indeed good to know and adding std.string.toStringz is probably a good idea ;). Thanks. Greetings, Max.
sure, I must admit it is annoying when the same code can do different things just because of where the data came from. It would be easier to notice the bug if d never added a null on literals, but then there would also be a lot more usages of toStringz. I think if you want to test it you can do: auto s = "blah"; open(s[0..$].dup.ptr); // duplicating it should put it somewhere else // just slicing will not test
Jul 19 2010
parent reply awishformore <awishformore gmail.com> writes:
Am 20.07.2010 08:56, schrieb Rory McGuire:
 On Tue, 20 Jul 2010 01:37:10 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 21:34, schrieb Rory McGuire:
 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it will
 save someone some work.

 I've also written a nice connector class that wraps all the C
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to work
 was compiling the latest SQLite dll from source with dmc and then
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file to
 link into your app in order for it to compile and then use the dll. I
 however couldn't figure it out and it works for me. Don't hesitate to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Did you test with a string that was not in the code itself, e.g. from a config file? String literals are null terminated so you wouldn't have had an issue if all your strings were literals. Utf8 doesn't contain the string length, so you will run in to problems eventually. You have to use toStringz or your own null terminator. Unless of course you know that the function will always be taking string literals. But even then leaving something like that up to the programmer to remember is not exactly fool proof. Enjoy. ~Rory
Hey again and thanks for the hint. I tried finding something on the DM page about string literals being null terminated and while the section about string literals didn't even mention it, it was said some place else. That explains why using string literals works even though I expected it to fail. It's indeed good to know and adding std.string.toStringz is probably a good idea ;). Thanks. Greetings, Max.
sure, I must admit it is annoying when the same code can do different things just because of where the data came from. It would be easier to notice the bug if d never added a null on literals, but then there would also be a lot more usages of toStringz. I think if you want to test it you can do: auto s = "blah"; open(s[0..$].dup.ptr); // duplicating it should put it somewhere else // just slicing will not test
When thinking about it, it makes sense to have string literals null terminated in order to have C functions work with them. However, I wonder about some stuff, for instance: string s = "string"; // is s == "string\0" now? char[] c = cast(char[])s; // is c[6] == '\0' now? char* p = s.ptr; // is *(p+6) == '\0' now? I think use of the zero terminator should be consistent. Either make every string (and char[] for that matter) zero terminated in the underlying memory for backwards compatibility with C or leave it to the user in all cases. /Max
Jul 20 2010
next sibling parent "Rory McGuire" <rmcguire neonova.co.za> writes:
On Tue, 20 Jul 2010 14:23:07 +0200, awishformore <awishformore gmail.com>  
wrote:

 Am 20.07.2010 08:56, schrieb Rory McGuire:
 On Tue, 20 Jul 2010 01:37:10 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 21:34, schrieb Rory McGuire:
 On Mon, 19 Jul 2010 20:39:35 +0200, awishformore
 <awishformore gmail.com> wrote:

 Am 19.07.2010 17:50, schrieb Johannes Pfau:
 On 19.07.2010 00:44, awishformore wrote:
 Hello there.

 I've converted the .h file of the latest SQLite version to .d and I
 thought I'd let the world know (as suggested on IRC ;). Maybe it  
 will
 save someone some work.

 I've also written a nice connector class that wraps all the C
 functions
 for convenient use in your application. It's very basic, but I will
 probably add more features (like automatic preparation of  
 statements
 and
 automatic caching of several prepared statements) later.

 For the time being both files are included in the download:
 http://nexunity.com/sqlite4d.rar

 I'm pretty new to this kind of stuff, so what I did to get it to  
 work
 was compiling the latest SQLite dll from source with dmc and then
 link
 the .obj file into my app ( http://nexunity.com/sqlite3.obj ).

 I'm sure there is a better way like compiling it as static lib (dmc
 complained about no entry point) or having some kind of other file  
 to
 link into your app in order for it to compile and then use the  
 dll. I
 however couldn't figure it out and it works for me. Don't hesitate  
 to
 teach me nonetheless.

 Any kind of feedback is always appreciated.

 Greetings, Max.
Nice work! I'm by no means an expert in d, but I wonder whether the following in the connector could cause troubles: -------------------------------------- int open(string db_name) { return sqlite3_open_v2(cast(char*)db_name, -------------------------------------- I always thought just casting to the pointer type isn't really safe. I guess you should use std.strings toStringz. (Off topic: std.strings toStringz uses the GC to store the c string. Now if I passed the returned pointer to a C function, but didn't store a reference in the D program, couldn't that c string get collected while the c library would expect it to be still there?)
Hello there. I'm not an expert for anything related to programming, but I'm pretty sure the string/char[]/char* isn't needed by SQLite after the function returns, as you will always be using the sqlite4 object pointer. As for the issue about null-terminated C strings, I was initially expecting this to not work, but tried anyway. I guess this has to do with the fact that SQLite interprets the char* as UTF-8, which is what D uses. With only a few lines of code, UTF-16 would also work with the wrapper by the way, but I won't ever use it, so I didn't bother. I added quite a few unit tests to the file in order to verify stuff like this and all of them seem to pass without the slightest issue and there are no memory leaks either, so I'm assuming that at least the basic functionality of my connector class is absolutely safe to use. The only thing I changed except for the unit tests is the ptrtostr/ptrtoblob to access the pointer like an array instead of a while loop and the else -> else static if(T==sqlite3_value*) to ensure a valid parameter is always passed. Greetings, Max.
Did you test with a string that was not in the code itself, e.g. from a config file? String literals are null terminated so you wouldn't have had an issue if all your strings were literals. Utf8 doesn't contain the string length, so you will run in to problems eventually. You have to use toStringz or your own null terminator. Unless of course you know that the function will always be taking string literals. But even then leaving something like that up to the programmer to remember is not exactly fool proof. Enjoy. ~Rory
Hey again and thanks for the hint. I tried finding something on the DM page about string literals being null terminated and while the section about string literals didn't even mention it, it was said some place else. That explains why using string literals works even though I expected it to fail. It's indeed good to know and adding std.string.toStringz is probably a good idea ;). Thanks. Greetings, Max.
sure, I must admit it is annoying when the same code can do different things just because of where the data came from. It would be easier to notice the bug if d never added a null on literals, but then there would also be a lot more usages of toStringz. I think if you want to test it you can do: auto s = "blah"; open(s[0..$].dup.ptr); // duplicating it should put it somewhere else // just slicing will not test
When thinking about it, it makes sense to have string literals null terminated in order to have C functions work with them. However, I wonder about some stuff, for instance: string s = "string"; // is s == "string\0" now? char[] c = cast(char[])s; // is c[6] == '\0' now? char* p = s.ptr; // is *(p+6) == '\0' now? I think use of the zero terminator should be consistent. Either make every string (and char[] for that matter) zero terminated in the underlying memory for backwards compatibility with C or leave it to the user in all cases. /Max
perhaps the NULL is there because its there in the executable file? NULL is also often after a dynamic array simply because of d always initializing memory, and when you get an allocation often a larger amount is allocated which remains NULL.
Jul 20 2010
prev sibling parent Pelle <pelle.mansson gmail.com> writes:
On 07/20/2010 02:23 PM, awishformore wrote:
 When thinking about it, it makes sense to have string literals null
 terminated in order to have C functions work with them. However, I
 wonder about some stuff, for instance:

 string s = "string";
 // is s == "string\0" now?
No, but the byte after the string (in the executable) is 0.
 char[] c = cast(char[])s;
 // is c[6] == '\0' now?
No, c.length == 6.
 char* p = s.ptr;
 // is *(p+6) == '\0' now?
Yes. Since it came from a literal.
 I think use of the zero terminator should be consistent. Either make
 every string (and char[] for that matter) zero terminated in the
 underlying memory for backwards compatibility with C or leave it to the
 user in all cases.

 /Max
The current situation allows for this: printf("Hello!\n"); but not this: string s = "Hello!\n"; printf(s); Since the compiler knows about literals being null terminated, but not just any string. I think it's a good compromise.
Jul 22 2010
prev sibling next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
awishformore Wrote:

 Ah yeah, I guess I should have mentioned. I'm only working with D2 at 
 the moment, so it is D2 right now (because of the use of string). 
 However, it should be straight forward to replace all string occurrences 
 with char[] to get it to work on D1. I'm pretty sure that's the only 
 issue stopping it from working on D1.
D1 has 'string' defined as char[] so there is no need to replace it, and it should not be replaced.
Jul 21 2010
prev sibling next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
awishformore Wrote:

 You could then even go as far as not iduping, but instead casting it to 
 a string to avoid unnecessary allocations.
In D2 you would use std.contracts.assumeUnique() for this. http://digitalmars.com/d/2.0/phobos/std_contracts.html#assumeUnique
Jul 21 2010
prev sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Johannes Pfau Wrote:

 I always thought just casting to the pointer type isn't really safe. I
 guess you should use std.strings toStringz.
 
 (Off topic: std.strings toStringz uses the GC to store the c string. Now
 if I passed the returned pointer to a C function, but didn't store a
 reference in the D program, couldn't that c string get collected while
 the c library would expect it to be still there?)
 -- 
 Johannes Pfau
I had my own question about toStringz, yours was one of them, the function returns a const(char)*, but C doesn't have such a notion so how is this enforced?
Jul 21 2010