www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - 64 bit types and C libraries

reply Regan Heath <regan netmail.co.nz> writes:
Hi all,

I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks
like a mammoth task but I figure I'll just chip away at it a piece at a time.  

This seems like the right time to ask if anyone else is having a go at it, or
wants to give me a hand.  I've actually just got a job (yay me) but wont start
for a bit (11th June) so I have some spare time.  Once I start work I wont have
as much time so it would be good to find someone else who can help.

In the meantime I have reached a point where a type PQ_64BIT is used.  Looking
at the headers it appears this type is defined differently for each
system/compiler based on whether they have a 64 bit type, so on windows using
M$ compilers etc it is __int64 for example.  In the rest of cases it is a
custom BIGNUM struct.

My question is this.. what the heck do I put in my D port when I see this type
used?

In D, long is a 64 bit type, but... I'm not writing new code I am writing a
wrapper so I need to put the actual type the C library is using (was compiled
using), otherwise any struct using PQ_64BIT might be the wrong size and
disastrous things might happen.

Am I understanding this correctly?

It seems that I can probably get away with swapping PQ_64BIT for long in the
short term, as I suspect my library was compiled with a 64bit type, not the
BIGNUM struct...

Thoughts?
May 22 2007
next sibling parent torhu <fake address.dude> writes:
Regan Heath wrote:
 Hi all,
 
 I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks
like a mammoth task but I figure I'll just chip away at it a piece at a time.  
 
 This seems like the right time to ask if anyone else is having a go at it, or
wants to give me a hand.  I've actually just got a job (yay me) but wont start
for a bit (11th June) so I have some spare time.  Once I start work I wont have
as much time so it would be good to find someone else who can help.
 
 In the meantime I have reached a point where a type PQ_64BIT is used.  Looking
at the headers it appears this type is defined differently for each
system/compiler based on whether they have a 64 bit type, so on windows using
M$ compilers etc it is __int64 for example.  In the rest of cases it is a
custom BIGNUM struct.
 
 My question is this.. what the heck do I put in my D port when I see this type
used?
 
 In D, long is a 64 bit type, but... I'm not writing new code I am writing a
wrapper so I need to put the actual type the C library is using (was compiled
using), otherwise any struct using PQ_64BIT might be the wrong size and
disastrous things might happen.
 
 Am I understanding this correctly?
 
 It seems that I can probably get away with swapping PQ_64BIT for long in the
short term, as I suspect my library was compiled with a 64bit type, not the
BIGNUM struct...
 
 Thoughts?
 
 

How's the BIGNUM struct defined?
May 22 2007
prev sibling next sibling parent Mandel <mandel no.no> writes:
Regan Heath Wrote:

 This seems like the right time to ask if anyone else is having a go at it, or
wants to give me a hand.

In my opinion it's good for simple questions, this NG is very good for the harder ones.
 In the meantime I have reached a point where a type PQ_64BIT is used.  Looking
at the headers it appears this type is defined differently for each
system/compiler based on whether they have a 64 bit type, so on windows using
M$ compilers etc it is __int64 for example.  In the rest of cases it is a
custom BIGNUM struct.
 My question is this.. what the heck do I put in my D port when I see this type
used?

The NEWS file of the openssl source package mentions that BIGNUM is optimized for special tasks, probably optimizations the D compiler couldn't figure out. Imho it's a good idea to use long/long for now and to postpone this optimization for such (more rare) systems. Maybe the D compilers can do this on it's own, I did not took a look a the BIGNUM yet. good luck!
May 22 2007
prev sibling next sibling parent Sean Kelly <sean f4.ca> writes:
It sounds like PQ_64BIT is intended to always be 64 bits in length, so 
it's probably safe to just use a long.


Sean
May 22 2007
prev sibling next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Regan Heath wrote:
 Hi all,
 
 I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks
like a mammoth task but I figure I'll just chip away at it a piece at a time.  
 
 This seems like the right time to ask if anyone else is having a go at it, or
wants to give me a hand.  I've actually just got a job (yay me) but wont start
for a bit (11th June) so I have some spare time.  Once I start work I wont have
as much time so it would be good to find someone else who can help.
 
 In the meantime I have reached a point where a type PQ_64BIT is used.  Looking
at the headers it appears this type is defined differently for each
system/compiler based on whether they have a 64 bit type, so on windows using
M$ compilers etc it is __int64 for example.  In the rest of cases it is a
custom BIGNUM struct.
 
 My question is this.. what the heck do I put in my D port when I see this type
used?
 
 In D, long is a 64 bit type, but... I'm not writing new code I am writing a
wrapper so I need to put the actual type the C library is using (was compiled
using), otherwise any struct using PQ_64BIT might be the wrong size and
disastrous things might happen.
 
 Am I understanding this correctly?
 
 It seems that I can probably get away with swapping PQ_64BIT for long in the
short term, as I suspect my library was compiled with a 64bit type, not the
BIGNUM struct...
 
 Thoughts?

Having a very quick poke around the OpenSSL docs, it looks like PQ_64BIT is designed to be used as an opaque type. The problem, of course, is that you don't know how the library was compiled. It looks like the only safe way to do it is to mimic the C headers. Something like this: version( SIXTY_FOUR_BIT ) { typedef ulong PQ_64BIT; // Note that I think OpenSSL uses ulongs! void pq_64bit_init(ref PQ_64BIT x) {} void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {x += w} } else { typedef BIGNUM PQ_64BIT; void pq_64bit_init(ref PQ_64BIT x) { BN_INIT(&x); } void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) { BN_add_word(&x, &w); } } And so on. Ugly as hell, but that's how OpenSSL is doing things. Hope that's of some help :P -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } http://xkcd.com/ v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
May 22 2007
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
torhu wrote:
 How's the BIGNUM struct defined?

As far as I can tell, the OpenSSL docs don't specify; they state that BIGNUM should be treated as an opaque type, and that you must never attempt to directly access its members. That practically screams of "BIGNUM's implementation is platform-dependant" :P Mandel wrote:
 I think you can go with long/ulong. Since the size is fixed for D, the
 compiler does the job emulating a 64 integer on systems where int is
 smaller 32bits.

It doesn't matter what D is capable of; if OpenSSL has been compiled in 32-bit mode, then it's using BIGNUM internally. If you try passing it a long, then you can very well end up with serious errors (see below). Sean Kelly wrote:
 It sounds like PQ_64BIT is intended to always be 64 bits in length, so
 it's probably safe to just use a long.

 Sean

Actually, PQ_64BIT can be aliased to BIGNUM which can be an *arbitrary* number of bits; it's a dynamically allocated structure. However, if his version is compiled with 64-bit integer support, and he doesn't mind not supporting BIGNUM versions, he could just stick to ulong. Maybe just stub out the other side of the version path with some static assert(false)'s. :) Ok; bed time. -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } http://xkcd.com/ v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
May 22 2007
parent Regan Heath <regan netmail.co.nz> writes:
Daniel Keep Wrote:
 torhu wrote:
 How's the BIGNUM struct defined?

As far as I can tell, the OpenSSL docs don't specify; they state that BIGNUM should be treated as an opaque type, and that you must never attempt to directly access its members. That practically screams of "BIGNUM's implementation is platform-dependant" :P

Yep, it looks like this: struct bignum_st { BN_ULONG *d; /* Pointer to an array of 'BN_BITS2' bit chunks. */ int top; /* Index of last used d +1. */ /* The next are internal book keeping for bn_expand. */ int dmax; /* Size of the d array. */ int neg; /* one if the number is negative */ int flags; }; which answers the question "is it 64 bits", the answer is "nope".
 Actually, PQ_64BIT can be aliased to BIGNUM which can be an *arbitrary*
 number of bits; it's a dynamically allocated structure.

Yep, that's the problem as I see it.
 However, if his version is compiled with 64-bit integer support, and he
 doesn't mind not supporting BIGNUM versions, he could just stick to
 ulong.  Maybe just stub out the other side of the version path with some
 static assert(false)'s. :)

This is what I did, to progress with other parts in the meantime. However, I think I may need to re-think my strategy. I started with ssl.h, the top level C include file. Created an ssl.d copying/converting everything, then compiled with a main.d which imported openssl.ssl; This kicked up all the undefined symbols (handled by includes in the original source) I tracked the first of these and converted that file, however, each new source includes more undefined symbols and I end up in the murky depths of openssl. The problem with my method is that of course I am going to end up converting the whole thing one file at a time. It just strikes me that I should be able to get away with converting less, just the stuff a library end-user would actually use. Maybe I should start by identifying that.. hmm. Regan
May 22 2007
prev sibling parent Regan Heath <regan netmail.co.nz> writes:
Daniel Keep Wrote:
 Regan Heath wrote:
 Hi all,
 
 I am throwing my hand at writing D ports of the OpenSSL libraries.  It looks
like a mammoth task but I figure I'll just chip away at it a piece at a time.  
 
 This seems like the right time to ask if anyone else is having a go at it, or
wants to give me a hand.  I've actually just got a job (yay me) but wont start
for a bit (11th June) so I have some spare time.  Once I start work I wont have
as much time so it would be good to find someone else who can help.
 
 In the meantime I have reached a point where a type PQ_64BIT is used.  Looking
at the headers it appears this type is defined differently for each
system/compiler based on whether they have a 64 bit type, so on windows using
M$ compilers etc it is __int64 for example.  In the rest of cases it is a
custom BIGNUM struct.
 
 My question is this.. what the heck do I put in my D port when I see this type
used?
 
 In D, long is a 64 bit type, but... I'm not writing new code I am writing a
wrapper so I need to put the actual type the C library is using (was compiled
using), otherwise any struct using PQ_64BIT might be the wrong size and
disastrous things might happen.
 
 Am I understanding this correctly?
 
 It seems that I can probably get away with swapping PQ_64BIT for long in the
short term, as I suspect my library was compiled with a 64bit type, not the
BIGNUM struct...
 
 Thoughts?

Having a very quick poke around the OpenSSL docs, it looks like PQ_64BIT is designed to be used as an opaque type. The problem, of course, is that you don't know how the library was compiled. It looks like the only safe way to do it is to mimic the C headers. Something like this: version( SIXTY_FOUR_BIT ) { typedef ulong PQ_64BIT; // Note that I think OpenSSL uses ulongs! void pq_64bit_init(ref PQ_64BIT x) {} void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) {x += w} } else { typedef BIGNUM PQ_64BIT; void pq_64bit_init(ref PQ_64BIT x) { BN_INIT(&x); } void pq_64bit_add_word(ref PQ_64BIT x, ref PQ_64BIT w) { BN_add_word(&x, &w); } } And so on. Ugly as hell, but that's how OpenSSL is doing things. Hope that's of some help :P

I think you're right. Ick. Regan
May 22 2007
prev sibling parent reply Daniel919 <Daniel919 web.de> writes:
Hi, some time ago I already used bcd 
(http://www.dsource.org/projects/bcd/) to successfully convert the 
openssl headers to d.

It's working. The only problem is that bcd doesn't convert the macros 
from the .h files, so that I had to manually resolve them:
in C: SSL_CTX_set_options(ctx, SSL_OP_ALL);
in D: SSL_CTX_ctrl((ctx),SSL_CTRL_OPTIONS,(SSL_OP_ALL),null);

If you didn't know bcd then you should check it out, since
it might save you a lot of work. I know the openssl headers are very 
extensive.

Best regards,
Daniel
May 22 2007
parent reply Regan Heath <regan netmail.co.nz> writes:
Daniel919 Wrote:
 Hi, some time ago I already used bcd 
 (http://www.dsource.org/projects/bcd/) to successfully convert the 
 openssl headers to d.
 
 It's working. The only problem is that bcd doesn't convert the macros 
 from the .h files, so that I had to manually resolve them:
 in C: SSL_CTX_set_options(ctx, SSL_OP_ALL);
 in D: SSL_CTX_ctrl((ctx),SSL_CTRL_OPTIONS,(SSL_OP_ALL),null);
 
 If you didn't know bcd then you should check it out, since
 it might save you a lot of work. I know the openssl headers are very 
 extensive.

Good point, this might be a better starting point. Yep, the openssl headers are chock full of macros. How long did it take you to resolve/replace them all with function calls? What version of openssl did you convert/wrap? Perhaps it would be easier to bring those up to date? Regan
May 22 2007
parent reply Daniel919 <Daniel919 web.de> writes:
 How long did it take you to resolve/replace them all with function calls?

it would be a really complete port ;) I only expanded the ones I needed.
 What version of openssl did you convert/wrap?  Perhaps it would be 

As far as I can remember it was 0.9.8d But bcd should be fine with all versions. I put my project online: http://mitglied.lycos.de/daniel919/ssl_ftpd_dummy.rar It's a simple ssl ftpd. Maybe you will find something that could be helpful. I had to manually make some changes to the d files generated by bcd at that time. Maybe in the meantime bcd will not require some of these changes anymore. And really, a complete port of openssl for D would be great.
May 22 2007
parent Regan Heath <regan netmail.co.nz> writes:
Daniel919 Wrote:
  > How long did it take you to resolve/replace them all with function calls?
 Well, I didn't replace all the macros in the .h files. If I had done so,
 it would be a really complete port ;)
 I only expanded the ones I needed.

Ahh, ok.
  > What version of openssl did you convert/wrap?  Perhaps it would be 
 easier to bring those up to date?
 As far as I can remember it was 0.9.8d

I have 0.9.8e so it should be pretty similar.
 But bcd should be fine with all versions.

True.
 I put my project online: 
 http://mitglied.lycos.de/daniel919/ssl_ftpd_dummy.rar
 It's a simple ssl ftpd.

Cool.
 Maybe you will find something that could be helpful. I had to manually 
 make some changes to the d files generated by bcd at that time.
 Maybe in the meantime bcd will not require some of these changes anymore.
 
 And really, a complete port of openssl for D would be great.

Indeed it would. I had been meaning to get onto it before I went overseas for 6 months and now that I have some time I figured I would give it a crack. Thanks for your tips. Regan
May 22 2007