www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - A question about modules

reply Myron Alexander <someone somewhere.com> writes:
Hello.

My understanding of D is still maturing so I may make some glaringly 
incorrect statements. Please correct me if I am wrong. I may also ramble 
a bit but then I have not slept since midday yesterday. (BTW I did 
search on the newsgroups, found nothing on this topic so I hope I am not 
opening any old wounds).

I am trying to package my library for use in another project of mine. My 
problem is that I make extensive use of templates in the library so I 
cannot just create a static library with some prototype .di files. I am 
used to the Java jar files and would like to know if it is possible to 
do something like that for D (even if it is only a zipped up source 
library).

Right now I am copying my library source tree into the project, not a 
great solution. I was also looking at how to hide the template 
implementation which got me thinking.

I would also like to know if there are any technical reasons that D 
could not adopt a binary format module mechanism. In Java, you import 
the compiled class file, not the source. I cannot think of a reason why 
D should not be able to do the same. The library would not be a C 
compatible library but then this is the language D.

As it stands, if you want C compatibility, you have to code C specific 
interfaces so nothing would change on that front, except that you would 
have to tell the compiler that you are generating a C compatible 
library. Thus we have a compiler that spits out D modules by default and 
can produce C libraries on demand (not advocating automatic conversion, 
the developer would have to do the interface dirty work as it currently 
stands).

I hope I made some sense.

What are you thoughts on this matter?

Regards,

Myron Alexander
dprogramming...myron...alexander...com
replace the first ... with  , remove the second, and replace the third 
with ".".
Jun 12 2007
next sibling parent reply orgoton <orgoton mindless.com> writes:
I suppose what you mean is to hide the source of you library, and/or not to
have the source available at compile time. The solution is simple, pre-compile
it.
In short, compile your library into a .lib file and ask DMD to generate a
interface file (.di). Then, on your modules import the .di file and add:

pragma(lib, "libfile.lib"); //libfile.lib must be inside the dirs searched by
DMD

or add the -Ilibfile.lib to your DMD command. The downside is that you either
compile the lib in release/debug mode and that's it. If you want to change the
build mode, you'll have to recompile the library. But that's not a real problem.
Jun 12 2007
parent reply Myron Alexander <someone somewhere.com> writes:
orgoton wrote:
 I suppose what you mean is to hide the source of you library, and/or not to
have the source available at compile time. The solution is simple, pre-compile
it.
 In short, compile your library into a .lib file and ask DMD to generate a
interface file (.di). Then, on your modules import the .di file and add:
 
 pragma(lib, "libfile.lib"); //libfile.lib must be inside the dirs searched by
DMD
 
 or add the -Ilibfile.lib to your DMD command. The downside is that you either
compile the lib in release/debug mode and that's it. If you want to change the
build mode, you'll have to recompile the library. But that's not a real problem.

Orgoton, I am aware of the .di files (although I did not see that dmd can generate them for you so thanks for the info) but what I want to do is eliminate the need for a separate .lib and .di files. The way Java works is that you do not have to include a textual header with the .class files, the .class files act as the header and obj in one file. I believe that it is possible to do it for D, but it means some work on the front-end and back-end so the linker will have to have an additional output mode for D specific modules. You would not be able link a D module with a C library directly, the linker would have to extract the executable object from the binary module and then link it all together into an exe. Since I have not written a compiler in nearly 2 decades, I will not say I know it is possible; I hope someone in the know could inform me on the viability of this suggestion. If there are no technical reasons preventing the implementation of such a mechanism, and it is just a matter of Walter not having the time, then so be it; but I would like to know, just in case something could be worked out. Best Regards, Myron.
Jun 12 2007
next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Myron Alexander wrote:
 I am aware of the .di files (although I did not see that dmd can 
 generate them for you so thanks for the info) but what I want to do is 
 eliminate the need for a separate .lib and .di files. The way Java works 
  is that you do not have to include a textual header with the .class 
 files, the .class files act as the header and obj in one file.

Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.
Jun 12 2007
parent reply Myron Alexander <someone somewhere.com> writes:
Lutger wrote:
 Myron Alexander wrote:
 I am aware of the .di files (although I did not see that dmd can 
 generate them for you so thanks for the info) but what I want to do is 
 eliminate the need for a separate .lib and .di files. The way Java 
 works  is that you do not have to include a textual header with the 
 .class files, the .class files act as the header and obj in one file.

Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.

The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.
Jun 12 2007
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Myron,

 The .dm file contains templates and other compile time goodies but
 without showing them in plain text.

IIRC Ada does somthing along this line, but I have never used Ada so I don't know how well this applys.
Jun 12 2007
parent Myron Alexander <someone somewhere.com> writes:
BCS wrote:
 Reply to Myron,
 
 The .dm file contains templates and other compile time goodies but
 without showing them in plain text.

IIRC Ada does somthing along this line, but I have never used Ada so I don't know how well this applys.

I think Delphi units work this way. I am not a Delphi programmer but I heard that .dcu compiled units are binary modules. Delphi also has something called a package which is a collection of units. Regards, Myron.
Jun 12 2007
prev sibling next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
That's a great idea!

But as far as java decompilation goes... never used jad, but cavaj generates a
picture-perfect representation of the code except for local variable/parameter
names. If it had those in the bytecode, I'd use it as a code
formatter/beautifier.

Myron Alexander Wrote:

 Lutger wrote:
 Myron Alexander wrote:
 I am aware of the .di files (although I did not see that dmd can 
 generate them for you so thanks for the info) but what I want to do is 
 eliminate the need for a separate .lib and .di files. The way Java 
 works  is that you do not have to include a textual header with the 
 .class files, the .class files act as the header and obj in one file.

Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.

The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.

Jun 12 2007
prev sibling parent Ary Manzana <ary esperanto.org.ar> writes:
Nice idea!

I also had this idea some weeks ago (but, I never told it)... And, of 
course, the name was also "dar". It would realy be cool. Especially 
since in Descent I could replace all ocurrences of the word "jar" with 
"dar" and voilà! :-P

Myron Alexander escribió:
 Lutger wrote:
 Myron Alexander wrote:
 I am aware of the .di files (although I did not see that dmd can 
 generate them for you so thanks for the info) but what I want to do 
 is eliminate the need for a separate .lib and .di files. The way Java 
 works  is that you do not have to include a textual header with the 
 .class files, the .class files act as the header and obj in one file.

Perhaps this is a stupid question, but what is the benefit of this? Templates are handled ok with .lib and .di files. If the problem is ease of use of packaging, then this is taken care of by tools like dsss.

The problem is having to manage separate header and obj files. This was always a pain in C and it is a pain in D, less of a pain but one none the less (especially with bug 1133). The other problem with .di files is that the template code is in plain sight - not so good for closed source / commercial libraries. Imagine this scenario with the binary modules: Starting with sources, you compile the library and generate a .dar file (D archive). The .dar file is very much like an uber jar file (with all the meta-data issues sorted out). The .dar file is an archive that contains .dm files or D module files. The D module file acts like the .di file when sources are compiled against it, and it acts like the .obj file in the link phase. Although it acts like the .di, it is not source, it is some syntax tree or optimized representation so it actually speeds up compilation time. The .dm file contains templates and other compile time goodies but without showing them in plain text. Because of how it works, with the right tools, you could convert the .di portion back into readable code, ala java jad, but it would not be as readable as the .di, just like jad. This is not an attempt to lock away the interface code, just not make it so obvious. The main benefit to .dar files is that you have a one file library that you drop into a directory and compile/link against simplifying distribution. Unlike jar files, these are not class loaded modules, they are linked into the binary just like .libs. I hope I made sense, my brain is shutting down and I am having some difficulty concentrating. If you have worked with jar files you will understand the convenience of not having to maintain separate include directories and a lib dir. Best Regards, Myron.

Jun 12 2007
prev sibling parent reply BCS <ao pathlink.com> writes:
Reply to Myron,

 what I want to do is
 eliminate the need for a separate .lib and .di files. The way Java
 works is that you do not have to include a textual header with 
 the .class files, the .class files act as the header and obj in one file.

 If there are no technical reasons preventing the implementation of
 such a mechanism, and it is just a matter of Walter not having the
 time, then so be it; but I would like to know, just in case something
 could be worked out.
 
 Best Regards,
 
 Myron.
 

I think it might be possible to write a program that would extract the symbol names from an object file, un-mangle them and build a .di file from this information. If this could be done, then this could be linked into the compiler and then pass this .di file off to the code that parses it. (or the .di generator could be re worked so that it inserts the symbol directly into the symbol table)
Jun 12 2007
parent Myron Alexander <someone somewhere.com> writes:
BCS wrote:
 I think it might be possible to write a program that would extract the 
 symbol names from an object file, un-mangle them and build a .di file 
 from this information. If this could be done, then this could be linked 
 into the compiler and then pass this .di file off to the code that 
 parses it. (or the .di generator could be re worked so that it inserts 
 the symbol directly into the symbol table)
 
 

This will not work with compile time features such as templates. See my answer to Lutger. Best Regards, Myron.
Jun 12 2007
prev sibling parent reply =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= writes:
Myron Alexander wrote:
I am copying my library source tree into the project, not a
 great solution.

If you are using Subversion for both projects you can setup one folder to automatically import the other project (Even on different repositories) using an external property. That is what I usually do.
Jun 12 2007
parent reply Myron Alexander <someone somewhere.com> writes:
Julio César Carrascal Urquijo wrote:
 Myron Alexander wrote:
 I am copying my library source tree into the project, not a
 great solution.

If you are using Subversion for both projects you can setup one folder to automatically import the other project (Even on different repositories) using an external property. That is what I usually do.

I didn't know you could do this. Although this will work for my project, it does not solve the library distribution problem. We end up having the same problem that the C repositories have. I'm hoping to solve that problem once and for all. Perhaps for now, we could come up with a convention such as this: All external libraries exist under an ext directory. The ext directory has the following structure: ext/<libname>/di/<toplevel>/... ext/<libname>/<libname>.lib Then with the rebuild/bud/dmd tool or whatever, you could just have an option such as -import=<libname> and it would add the .di to the include path, and the .lib to the link path. This is a workable solution until we get binary modules. If that works, we could go to the next level where we zip up the library. Same directory structure, just zipped up and call it .dlz for D library zipped. Then the rebuild/bud or library tool could unzip the contents into a temp/cache directory and add those directories/.libs to the paths. It does not have to be built into bud/rebuild, it could be a separate library tool that creates a few environment variables to add to the bud/rebuild/dmd command line. What do you think? Regards, Myron.
Jun 13 2007
parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Myron Alexander wrote:
 Perhaps for now, we could come up with a convention such as this:
 
 All external libraries exist under an ext directory. The ext directory 
 has the following structure:
 
 ext/<libname>/di/<toplevel>/...
 ext/<libname>/<libname>.lib
 
 Then with the rebuild/bud/dmd tool or whatever, you could just have an 
 option such as -import=<libname> and it would add the .di to the include 
 path, and the .lib to the link path. This is a workable solution until 
 we get binary modules.
 
 If that works, we could go to the next level where we zip up the 
 library. Same directory structure, just zipped up and call it .dlz for D 
 library zipped. Then the rebuild/bud or library tool could unzip the 
 contents into a temp/cache directory and add those directories/.libs to 
 the paths. It does not have to be built into bud/rebuild, it could be a 
 separate library tool that creates a few environment variables to add to 
 the bud/rebuild/dmd command line.
 
 What do you think?
 
 Regards,
 
 Myron.

dsss is meant for this and I still think it fulfills it's purpose nicely. Rather than creating another convention, it is perhaps more effective to discuss what you find lacking in dsss and how to improve on that. Keep in mind that dsss already does what you mentioned and a lot more too: install over the net (with automatic dependency installs), documentation generation with candydoc, pre- and post build steps... Only thing it doesn't do is hiding .di files for closed source distribution, but that will require another solution altogether. I'm not used to Java style distribution, so I may miss something worthwhile here, but I find the D way not so problematic. It is certainly a lot better than C, and C++ too. This is largely because D has a true module system instead of text replacement, and tools like dsss, rebuild and bud are very good.
Jun 13 2007
parent reply Myron Alexander <someone somewhere.com> writes:
Lutger wrote:
 dsss is meant for this and I still think it fulfills it's purpose 
 nicely. Rather than creating another convention, it is perhaps more 
 effective to discuss what you find lacking in dsss and how to improve on 
 that.
 
 Keep in mind that dsss already does what you mentioned and a lot more 
 too: install over the net (with automatic dependency installs), 
 documentation generation with candydoc, pre- and post build steps... 
 Only thing it doesn't do is hiding .di files for closed source 
 distribution, but that will require another solution altogether.
 
 I'm not used to Java style distribution, so I may miss something 
 worthwhile here, but I find the D way not so problematic. It is 
 certainly a lot better than C, and C++ too. This is largely because D 
 has a true module system instead of text replacement, and tools like 
 dsss, rebuild and bud are very good.

Lutger, I have taken another look at DSSS. When I first looked at it, I was sleep deprived so I did miss some things. Now I see that it does something like I suggested and is better in the usual use-case but it does not cover all the use-cases. The problem with DSSS is that it ties you to the DSSS repository. Apart from the fact that few libraries use it, it is intended, and designed, around release libraries. What about debug libraries? Also, how does it handle versioning, specifically unreleased experimental versions that are distributed to a few alpha/beta testers? What about libraries that have special build conditions? Some example build conditions (outside of portability issues): trial versions, lite versions (only core features), modular vs monolithic versions, and so on. I'm not saying that DSSS cannot handle those cases, I have yet to use it properly so I reserve my judgement, but, from the documentation those are not the typical use-case scenarios. My suggestion complements, rather than replaces, DSSS and could provide that middle ground; might even make Gregor's life easier. If I misunderstand something about DSSS, please correct me. Best Regards, Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 13 2007
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Hi Myron,

You can use DSSS without putting the library in the net repository. DSSS also
has support for version statements, etc., and you can always have multiple
copies of a library in DSSS, for different purposes.

As far as your other suggestion goes, have you checked out DDL
(http://www.dsource.org/projects/ddl )? It's designed primarily around dynamic
libraries, but the format certainly contains enough information for static
linking. It would be great if the same format could be used for both static and
dynamic linking, though exactly how this would work I can't imagine. The
project looks dead, but I think pragma said he plans to get back to it once
tango gets to 1.0.

Myron Alexander Wrote:

 Lutger wrote:
 dsss is meant for this and I still think it fulfills it's purpose 
 nicely. Rather than creating another convention, it is perhaps more 
 effective to discuss what you find lacking in dsss and how to improve on 
 that.
 
 Keep in mind that dsss already does what you mentioned and a lot more 
 too: install over the net (with automatic dependency installs), 
 documentation generation with candydoc, pre- and post build steps... 
 Only thing it doesn't do is hiding .di files for closed source 
 distribution, but that will require another solution altogether.
 
 I'm not used to Java style distribution, so I may miss something 
 worthwhile here, but I find the D way not so problematic. It is 
 certainly a lot better than C, and C++ too. This is largely because D 
 has a true module system instead of text replacement, and tools like 
 dsss, rebuild and bud are very good.

Lutger, I have taken another look at DSSS. When I first looked at it, I was sleep deprived so I did miss some things. Now I see that it does something like I suggested and is better in the usual use-case but it does not cover all the use-cases. The problem with DSSS is that it ties you to the DSSS repository. Apart from the fact that few libraries use it, it is intended, and designed, around release libraries. What about debug libraries? Also, how does it handle versioning, specifically unreleased experimental versions that are distributed to a few alpha/beta testers? What about libraries that have special build conditions? Some example build conditions (outside of portability issues): trial versions, lite versions (only core features), modular vs monolithic versions, and so on. I'm not saying that DSSS cannot handle those cases, I have yet to use it properly so I reserve my judgement, but, from the documentation those are not the typical use-case scenarios. My suggestion complements, rather than replaces, DSSS and could provide that middle ground; might even make Gregor's life easier. If I misunderstand something about DSSS, please correct me. Best Regards, Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".

Jun 13 2007
next sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Robert Fraser wrote:
 As far as your other suggestion goes, have you checked out DDL
(http://www.dsource.org/projects/ddl )? It's designed primarily around dynamic
libraries, but the format certainly contains enough information for static
linking. It would be great if the same format could be used for both static and
dynamic linking, though exactly how this would work I can't imagine. The
project looks dead, but I think pragma said he plans to get back to it once
tango gets to 1.0.

Actually, I believe the way DDL works is that you compile as if you're going to link statically to it, and then it doesn't actually link until run-time. So linking to files extracted from it should work just fine. Put the contents of a .di in a metadata field and you have a file containing both a binary module and the header to link to it. The build utility in use would just have to extract them at the appropriate time and use them. (It doesn't look like DDL provides a stand-alone extraction utility but I think one could easily be created, either by linking to DDL or by writing it from scratch[1]) That would only work for contained binaries the current toolchain supports though, not arbitrary ones (as IIRC DDL itself supports). [1] The file format for .ddl files is at http://dsource.org/projects/ddl/wiki/Reference/ABI/DDL and looks easy enough to parse.
Jun 13 2007
prev sibling parent Myron Alexander <someone somewhere.com> writes:
Robert Fraser wrote:
 You can use DSSS without putting the library in the net repository.
 DSSS also has support for version statements, etc., and you can
 always have multiple copies of a library in DSSS, for different
 purposes.

I have split my library into two libraries, dbapi and sqlite. The sqlite library is the glue code that used to be in the "dbapi/c" directory but I decided that it should be split from the dbapi code to make it easier to support different sqlite library versions. The dbapi depends on the sqlite library. The sqlite library is made up of two files: sqlite3.lib and sqlite3.di. By hand, I create the library as follows: implib /s out\sqlite3.lib sqlitedll\sqlite3.dll dmd -c -o- -H -Hdout\di\sqlite src\sqlite\sqlite3.d Now I want to translate it to DSSS and allow DSSS to store the result in the repository. It took a while, but I finally figured out how to do it. Here is my config file:
 name    = sqlite3-glue-library-3.3.17
 version = 3.3.17

 [+sqlite]
 prebuild    = implib /s out\SDD-sqlite-3.3.17.lib sqlitedll\sqlite3.dll
 postbuild   = dmd -c -o- -H -Hdout\di\sqlite src\sqlite\sqlite3.d
 postinstall = install out/SDD-sqlite-3.3.17.lib  $LIB_PREFIX; \
               install out/di/sqlite/sqlite3.di $INCLUDE_PREFIX/sqlite

Now that I have used DSSS, I can safely say that my original comments about DSSS versioning weakness stands. It just copies all the includes into a global include directory, and all the libs into a global lib directory. The name is just for a package manifest. Here is what I said:
 What about debug libraries? Also, how does it handle versioning, specifically
unreleased experimental versions that are distributed to a few alpha/beta
testers? What about libraries that have special build conditions? Some example
build conditions (outside of portability issues): trial versions, lite versions
(only core features), modular vs monolithic versions, and so on.

Now let say that, for some reason, I have to make a tested version of dbapi linked to sqlite3.dll version 3.3.10. This could be due to some corporate that is using that version, happy with it, and won't change it. I create a copy of my 3.3.17 library and hand edit the .d file based on diffs between the .h versions (as I have been doing when upping the library version). The module name and relative location does not change, how do I install the .di file for 3.3.10 without overwriting 3.3.17? The library is no problem as I will change the name to be SDD-sqlite-3.3.10.lib which will sit happily with SDD-sqlite-3.3.17.lib. I hope I have shown that DSSS does not solve all problems, yet. What Gregor is trying to achieve is not trivial, but not impossible.
 
 As far as your other suggestion goes, have you checked out DDL
 (http://www.dsource.org/projects/ddl )? It's designed primarily
 around dynamic libraries, but the format certainly contains enough
 information for static linking. It would be great if the same format
 could be used for both static and dynamic linking, though exactly how
 this would work I can't imagine. The project looks dead, but I think
 pragma said he plans to get back to it once tango gets to 1.0.
 

Something to check out once the project gets going again. Myron. dprogramming...myron...alexander...com replace the first ... with , remove the second, and replace the third with ".".
Jun 13 2007