www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DSSS and multiple intermingled environments

reply Gregor Richards <Richards codu.org> writes:
DSSS seems to be getting a lot of flak recently because it's not 
particularly happy being used with multiple versions of DMD, GDC, etc 
all layered together.

This is hardly fair to DSSS. Detecting such situations is an 
arbitrary-complexity problem, and DSSS can never really know what has 
been changed under it. It can make some reasonable guesses, but since 
IMHO those would usually been wrong, I've erred on the side of 
ignorance. That is, if you're going to change DSSS' underlying 
compilation environment, it anticipates that you will consider the 
consequences and `dsss distclean` first.

I don't know whether this situation will improve. To be honest, I hardly 
consider the fact that DSSS is not psychic a bug. As I already said, 
it's not something that can be fixed magically, it's actually quite 
complex. If you're going to be forcing DSSS to jump through multiple 
environments, just clean up after yourself.

Do a `dsss distclean` before building with a different compiler.

  - Gregor Richards
Aug 28 2007
next sibling parent reply Johan Granberg <lijat.meREM OVEgmail.com> writes:
Gregor Richards wrote:


 DSSS seems to be getting a lot of flak recently because it's not
 particularly happy being used with multiple versions of DMD, GDC, etc
 all layered together.
 
 This is hardly fair to DSSS. Detecting such situations is an
 arbitrary-complexity problem, and DSSS can never really know what has
 been changed under it. It can make some reasonable guesses, but since
 IMHO those would usually been wrong, I've erred on the side of
 ignorance. That is, if you're going to change DSSS' underlying
 compilation environment, it anticipates that you will consider the
 consequences and `dsss distclean` first.


I don't think distclean should never be a solution. Wouldn't it be better if
there was a way to specify separate locations for binaries for the
different compiler/stdlib combinations in the dsss.conf file?
(gdc-phobos-bin,dmd-tango-bin,etc...)

Personally i consider the need to remove object files before a build a bug
in the build system and try to write makefiles (or other commandfiles) to
not require any make clean or similar.


 I don't know whether this situation will improve. To be honest, I hardly
 consider the fact that DSSS is not psychic a bug. As I already said,
 it's not something that can be fixed magically, it's actually quite
 complex. If you're going to be forcing DSSS to jump through multiple
 environments, just clean up after yourself.
 
 Do a `dsss distclean` before building with a different compiler.
 
   - Gregor Richards


ps. while we already are talking dsss deficiencies will it be possible to
run your own repositories (think of how git works) and specify in the
dsss.conf that this lib is from here and that lib from there etc ... ?
Aug 28 2007
parent reply Gregor Richards <Richards codu.org> writes:
Johan Granberg wrote:

 Gregor Richards wrote:
 

 DSSS seems to be getting a lot of flak recently because it's not
 particularly happy being used with multiple versions of DMD, GDC, etc
 all layered together.

 This is hardly fair to DSSS. Detecting such situations is an
 arbitrary-complexity problem, and DSSS can never really know what has
 been changed under it. It can make some reasonable guesses, but since
 IMHO those would usually been wrong, I've erred on the side of
 ignorance. That is, if you're going to change DSSS' underlying
 compilation environment, it anticipates that you will consider the
 consequences and `dsss distclean` first.


 
 I don't think distclean should [n]ever be a solution.


Neither do I. But detecting environment changes is not always possible. 
It would be easy enough to detect changed flags at build time and then 
clean up old things before building, but that's a nonsolution after 
installing it. DSSS was not designed to rectify all the problems with 
combining DMD and GDC, Phobos and Tango.


 Wouldn't it be better if
 there was a way to specify separate locations for binaries for the
 different compiler/stdlib combinations in the dsss.conf file?
 (gdc-phobos-bin,dmd-tango-bin,etc...)


No, that would not be better. That would mean that one person could 
write a dsss.conf file for their library that would work with building 
on multiple compilers, and the next person would write one that didn't. 
Builds should not be this inconsistent. Besides that, it would make 
dsss.conf files unduly complex. Besides that, it builds flexibility 
limits into DSSS, making it assume that there will only ever be two 
compilers.


 
 Personally i consider the need to remove object files before a build a bug
 in the build system  and try to write makefiles (or other commandfiles) to
 not require any make clean or similar.


Unless your makefiles also solve the halting problem, they cannot detect 
all such scenarios. Try making, then doing `make 
CC=incompatible_c_compiler`. The autotools "solve" this very much like 
DSSS does - if you want to suddenly use a different compiler, you have 
to clean up and start again. Try reconfiguring with a different compiler 
and then just typing `make` - if you're really lucky, it'll clean up for 
you. If you're not, it'll fail miserably. This only works for autotools 
because it has a configure "lock-in" phase that DSSS does not have - it 
can detect most environmental changes because it locks you into an 
environment.

As I've said over and over again: Detecting when the environment has 
changed in an incompatible way is an arbitrary-complexity problem.


 

 I don't know whether this situation will improve. To be honest, I hardly
 consider the fact that DSSS is not psychic a bug. As I already said,
 it's not something that can be fixed magically, it's actually quite
 complex. If you're going to be forcing DSSS to jump through multiple
 environments, just clean up after yourself.

 Do a `dsss distclean` before building with a different compiler.

   - Gregor Richards


 
 ps. while we already are talking dsss deficiencies will it be possible to
 run your own repositories (think of how git works) and specify in the
 dsss.conf that this lib is from here and that lib from there etc ... ?


The dsss.conf would be a wholly-inappropriate place to specify this. As 
per having multiple upstream sources, that's a possibility that will 
enter into it as I add the variety of changes to the source system that 
are coming in with DSource integration.

  - Gregor Richards
Aug 28 2007
parent reply Johan Granberg <lijat.meREM OVEgmail.com> writes:
Gregor Richards wrote:


 Johan Granberg wrote:

 Gregor Richards wrote:
 

 DSSS seems to be getting a lot of flak recently because it's not
 particularly happy being used with multiple versions of DMD, GDC, etc
 all layered together.

 This is hardly fair to DSSS. Detecting such situations is an
 arbitrary-complexity problem, and DSSS can never really know what has
 been changed under it. It can make some reasonable guesses, but since
 IMHO those would usually been wrong, I've erred on the side of
 ignorance. That is, if you're going to change DSSS' underlying
 compilation environment, it anticipates that you will consider the
 consequences and `dsss distclean` first.


 
 I don't think distclean should [n]ever be a solution.


 
 Neither do I. But detecting environment changes is not always possible.
 It would be easy enough to detect changed flags at build time and then
 clean up old things before building, but that's a nonsolution after
 installing it. DSSS was not designed to rectify all the problems with
 combining DMD and GDC, Phobos and Tango.
 

 Wouldn't it be better if
 there was a way to specify separate locations for binaries for the
 different compiler/stdlib combinations in the dsss.conf file?
 (gdc-phobos-bin,dmd-tango-bin,etc...)


 
 No, that would not be better. That would mean that one person could
 write a dsss.conf file for their library that would work with building
 on multiple compilers, and the next person would write one that didn't.
 Builds should not be this inconsistent. Besides that, it would make
 dsss.conf files unduly complex. Besides that, it builds flexibility
 limits into DSSS, making it assume that there will only ever be two
 compilers.
 

 
 Personally i consider the need to remove object files before a build a
 bug
 in the build system  and try to write makefiles (or other commandfiles)
 to not require any make clean or similar.


 
 Unless your makefiles also solve the halting problem, they cannot detect
 all such scenarios. Try making, then doing `make
 CC=incompatible_c_compiler`. The autotools "solve" this very much like
 DSSS does - if you want to suddenly use a different compiler, you have
 to clean up and start again. Try reconfiguring with a different compiler
 and then just typing `make` - if you're really lucky, it'll clean up for
 you. If you're not, it'll fail miserably. This only works for autotools
 because it has a configure "lock-in" phase that DSSS does not have - it
 can detect most environmental changes because it locks you into an
 environment.
 
 As I've said over and over again: Detecting when the environment has
 changed in an incompatible way is an arbitrary-complexity problem.


In what way? if the binaries lives in a project subdirectory named after the
compiler and standard library like cc-stdlib-bin and dsss always uses this
to place binaries. That way when the compiler or standard library was
swapped dsss would use a different directory and no conflicts would occur.

It is possible that I missunderstand the issue but in that case whats wrong
with the above system?


 

 I don't know whether this situation will improve. To be honest, I hardly
 consider the fact that DSSS is not psychic a bug. As I already said,
 it's not something that can be fixed magically, it's actually quite
 complex. If you're going to be forcing DSSS to jump through multiple
 environments, just clean up after yourself.

 Do a `dsss distclean` before building with a different compiler.

   - Gregor Richards


 
 ps. while we already are talking dsss deficiencies will it be possible to
 run your own repositories (think of how git works) and specify in the
 dsss.conf that this lib is from here and that lib from there etc ... ?


 
 The dsss.conf would be a wholly-inappropriate place to specify this. As
 per having multiple upstream sources, that's a possibility that will
 enter into it as I add the variety of changes to the source system that
 are coming in with DSource integration.
 
   - Gregor Richards


Where else if not dsss.conf? The way I envisioned it was a line per hos used
something like this.
with host1.addres1.com
with host2.addres2.org
possibly with specifiers for which library to get from where.

Personally this is an extremely important feature to have. It would mean
that I don't need to trust some external central point of failure to host
my project.
Aug 28 2007
parent reply Gregor Richards <Richards codu.org> writes:
Johan Granberg wrote:

 Gregor Richards wrote:

 Unless your makefiles also solve the halting problem, they cannot detect
 all such scenarios. Try making, then doing `make
 CC=incompatible_c_compiler`. The autotools "solve" this very much like
 DSSS does - if you want to suddenly use a different compiler, you have
 to clean up and start again. Try reconfiguring with a different compiler
 and then just typing `make` - if you're really lucky, it'll clean up for
 you. If you're not, it'll fail miserably. This only works for autotools
 because it has a configure "lock-in" phase that DSSS does not have - it
 can detect most environmental changes because it locks you into an
 environment.

 As I've said over and over again: Detecting when the environment has
 changed in an incompatible way is an arbitrary-complexity problem.


 
 In what way? if the binaries lives in a project subdirectory named after the
 compiler and standard library like cc-stdlib-bin and dsss always uses this
 to place binaries. That way when the compiler or standard library was
 swapped dsss would use a different directory and no conflicts would occur.


Well, first, DSSS doesn't really know what standard library is being 
used, since it's not that easy to detect whether Tango is installed. It 
should always be configured to know this, but is at least as often 
configured improperly as properly, since it can't detect that automatically.

I suppose I could reliably get which compiler is being used straight out 
of the configuration, and put all my object files into the appropriate 
place per that. That's a pretty good strategy actually. 
http://www.dsource.org/projects/dsss/ticket/128

However, that still doesn't solve anything once you've installed. People 
want to install everything into DSSS' default prefix, which means 
intermingling DMD-compiled and GDC-compiled libraries and the .di files 
associated with them.

I'm starting to mentally formulate a solution for the .di file problem. 
I don't know whether it will work, it needs more thought. My mindset is 
actually shown perfectly by Stéphan's post in this thread - I don't want 
DSSS to depend on things being incompatible. I'll take a few steps to 
make things happy together, but I don't want DSSS to start failing 
when/if things work together properly.


 
 It is possible that I missunderstand the issue but in that case whats wrong
 with the above system?
 

 I don't know whether this situation will improve. To be honest, I hardly
 consider the fact that DSSS is not psychic a bug. As I already said,
 it's not something that can be fixed magically, it's actually quite
 complex. If you're going to be forcing DSSS to jump through multiple
 environments, just clean up after yourself.

 Do a `dsss distclean` before building with a different compiler.

   - Gregor Richards


 ps. while we already are talking dsss deficiencies will it be possible to
 run your own repositories (think of how git works) and specify in the
 dsss.conf that this lib is from here and that lib from there etc ... ?


 The dsss.conf would be a wholly-inappropriate place to specify this. As
 per having multiple upstream sources, that's a possibility that will
 enter into it as I add the variety of changes to the source system that
 are coming in with DSource integration.

   - Gregor Richards


 
 Where else if not dsss.conf? The way I envisioned it was a line per hos used
 something like this.
 with host1.addres1.com
 with host2.addres2.org
 possibly with specifiers for which library to get from where.
 
 Personally this is an extremely important feature to have. It would mean
 that I don't need to trust some external central point of failure to host
 my project.


I have avoided encoding dependencies into dsss.conf at all. When the 
video of my presentation at dconf comes out, watch it - my ideal is that 
all forms of dependencies are encoded into imports. This keeps dsss.conf 
files simple and unilateral.

Also, you don't need to (nor even can you) host it on some DSSS-specific 
host. DSSS' net system just has a list of upstream sources. It needs 
this "central point of failure" because that's where it aggregates 
information about module-library relationships. That aggregation is why 
you can do `dsss net deps` without specifying library dependencies - 
they're specified by your imports.

So, the solution would be to just have an alternate source list. This is 
like yum, apt-get, etc (except that those usually can't live separately 
from the actual source).

  - Gregor Richards
Aug 28 2007
parent reply BCS <ao pathlink.com> writes:
Reply to Gregor,


 Also, you don't need to (nor even can you) host it on some
 DSSS-specific host. DSSS' net system just has a list of upstream
 sources. It needs this "central point of failure" because that's where
 it aggregates information about module-library relationships. That
 aggregation is why you can do `dsss net deps` without specifying
 library dependencies - they're specified by your imports.
 
 So, the solution would be to just have an alternate source list. This
 is like yum, apt-get, etc (except that those usually can't live
 separately from the actual source).


So what would it take to let the user play with the list of servers that 
'dsss net ...' starts looking at?


 
 - Gregor Richards
Aug 28 2007
parent reply Gregor Richards <Richards codu.org> writes:
BCS wrote:

 Reply to Gregor,
 

 Also, you don't need to (nor even can you) host it on some
 DSSS-specific host. DSSS' net system just has a list of upstream
 sources. It needs this "central point of failure" because that's where
 it aggregates information about module-library relationships. That
 aggregation is why you can do `dsss net deps` without specifying
 library dependencies - they're specified by your imports.

 So, the solution would be to just have an alternate source list. This
 is like yum, apt-get, etc (except that those usually can't live
 separately from the actual source).


 
 So what would it take to let the user play with the list of servers that 
 'dsss net ...' starts looking at?
 

 - Gregor Richards


 
 


I'm not sure which of two questions you're asking here (which isn't 
surprising since the net architecture is a bit complicated), so I'll 
just answer both:


So what would it take to let the user play with the default list of 
sources from which `dsss net` will download libraries?

Just register at http://dsss.codu.org/ and tell me what project you need 
access to.


So what would it take to let the user use a different list of sources 
from which `dsss net` will download libraries?

Currently you can only specify one at a time, but the --source option 
exists for this purpose.

  - Gregor Richards
Aug 28 2007
parent reply BCS <ao pathlink.com> writes:
Reply to Gregor,


 I'm not sure which of two questions you're asking here (which isn't
 surprising since the net architecture is a bit complicated), so I'll
 just answer both:
 


this question


 
 So what would it take to let the user use a different list of sources
 from which `dsss net` will download libraries?
 
 Currently you can only specify one at a time, but the --source option
 exists for this purpose.


Cool, what would it take to set up my own list server? If it is easy, It 
would let me play with DSSS locally. (My main dev system has, by design, 
no net access)
 

 - Gregor Richards
Aug 28 2007
parent reply Gregor Richards <Richards codu.org> writes:
BCS wrote:

 Reply to Gregor,
 

 I'm not sure which of two questions you're asking here (which isn't
 surprising since the net architecture is a bit complicated), so I'll
 just answer both:


 
 this question
 

 So what would it take to let the user use a different list of sources
 from which `dsss net` will download libraries?

 Currently you can only specify one at a time, but the --source option
 exists for this purpose.


 
 Cool, what would it take to set up my own list server? If it is easy, It 
 would let me play with DSSS locally. (My main dev system has, by design, 
 no net access)
 

 - Gregor Richards


 
 


In docs/README.technical, read the section on 'net installation'. It may 
or may not be sufficient (nobody has ever really used it as a reference 
:) ), so suggestions for improvements would be appreciated.

  - Gregor Richards
Aug 28 2007
parent reply BCS <ao pathlink.com> writes:
Reply to Gregor,


 BCS wrote:
 

 Reply to Gregor,
 

 I'm not sure which of two questions you're asking here (which isn't
 surprising since the net architecture is a bit complicated), so I'll
 just answer both:
 


 this question
 

 So what would it take to let the user use a different list of
 sources from which `dsss net` will download libraries?
 
 Currently you can only specify one at a time, but the --source
 option exists for this purpose.
 


 Cool, what would it take to set up my own list server? If it is easy,
 It would let me play with DSSS locally. (My main dev system has, by
 design, no net access)
 

 - Gregor Richards
 




 In docs/README.technical, read the section on 'net installation'. It
 may or may not be sufficient (nobody has ever really used it as a
 reference :) ), so suggestions for improvements would be appreciated.
 
 - Gregor Richards
 


So If I generate the listed files and stuff them on an HTTP server, I can 
get dsss to play with it?
How is the pkgs.list file generated? 
Do you have any tools for making life easy for someone trying to set up a 
system?
Aug 28 2007
parent Gregor Richards <Richards codu.org> writes:
BCS wrote:

 Reply to Gregor,
 

 BCS wrote:


 Reply to Gregor,


 I'm not sure which of two questions you're asking here (which isn't
 surprising since the net architecture is a bit complicated), so I'll
 just answer both:


 this question


 So what would it take to let the user use a different list of
 sources from which `dsss net` will download libraries?

 Currently you can only specify one at a time, but the --source
 option exists for this purpose.


 Cool, what would it take to set up my own list server? If it is easy,
 It would let me play with DSSS locally. (My main dev system has, by
 design, no net access)


 - Gregor Richards




 In docs/README.technical, read the section on 'net installation'. It
 may or may not be sufficient (nobody has ever really used it as a
 reference :) ), so suggestions for improvements would be appreciated.

 - Gregor Richards


 
 So If I generate the listed files and stuff them on an HTTP server, I 
 can get dsss to play with it?
 How is the pkgs.list file generated? Do you have any tools for making 
 life easy for someone trying to set up a system?
 
 


As mentioned in the last section of that file, pkgs.list is generated by 
pkgslist.d. I suppose I should mention in there that the file pkgslist.d 
is at
http://svn.dsource.org/projects/dsss/sources/pkgslist.d (which is in the 
default sources archive). Mayhaps that should move somewhere a bit more 
accessible too ...

Oh, right, answering the question (will get step documented better 
later). Within the directory with your sources.list, run pkgslist. It 
will download all the sources, trace imports, and create pkgs.list.

  - Gregor Richards
Aug 28 2007
prev sibling next sibling parent =?ISO-8859-1?Q?St=E9phan_Kochen?= <stephan kochen.nl> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gregor Richards schreef:

 DSSS seems to be getting a lot of flak recently because it's not
 particularly happy being used with multiple versions of DMD, GDC, etc
 all layered together.


I see this as a more complex problem than just a problem with DSSS. But
maybe I'm making assumptions here that are incorrect, so feel free to
correct me.

If I'm not mistaken, the (new) goal with standard libraries is to have
them live peacefully next to each other with a minimal layer that
provides the necessary language support (dynamic arrays, garbage
collection, etc.).

Furthermore, D has a documented ABI, and I assume that this is supposed
to be a standard ABI (at some point). This means that libraries compiled
using different compilers will be compatible. (like C)

Lots of this is still far away (if at all planned), but this means that
the only problem left for DSSS is to simply select a compiler. And
that's an option that is already available from what I've gathered.

The two problems mentioned really have been D annoyances all along. It
seems to me that the flak you were getting was just another rant about
these two problems, the _real_ problems. DSSS ended up unjustly mingled
in with these rants.

You are already walking the right path with DSSS, in my opinion.

- -- Stéphan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG1FXecFUq0gzqDwQRAlmUAKCheLxBIfqWrF7XTbzAro9Q1oMkDQCfcK3+
rJOxpQ9pUBsoPpHDsJfvDjU=
=DWyk
-----END PGP SIGNATURE-----
Aug 28 2007
prev sibling parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Gregor Richards wrote:

 DSSS seems to be getting a lot of flak recently because it's not 
 particularly happy being used with multiple versions of DMD, GDC, etc 
 all layered together.
 
 This is hardly fair to DSSS. Detecting such situations is an 
 arbitrary-complexity problem, and DSSS can never really know what has 
 been changed under it. It can make some reasonable guesses, but since 
 IMHO those would usually been wrong, I've erred on the side of 
 ignorance. That is, if you're going to change DSSS' underlying 
 compilation environment, it anticipates that you will consider the 
 consequences and `dsss distclean` first.


I don't expect DSSS to detect all situations.  But I think it would make 
sense to store build products created using 'some-profile' under 
directories specific to 'some-profile' or filenames specific to 
'some-profile'.  That's pretty much the approach boostjam takes.  It 
creates lots of very hugely nested directory trees and libraries with 
very long names, but it seems to work well.

As for the di files, I think you can make it work better with 
multi-flavor builds by importing the pragma link bit rather than 
embedding it directly in the di.

So you replace this bit at the end of each installed .di:

version (build) {
     pragma(link, "SDD-somelib");
}

with something like

import current.package.name.somelib-linkstub;

And put somelib-linkstub.d in a build-specific include directory, 
containing the build-specific pragma above.

Or maybe easier would be to just make pragma link smart enough to infer 
the prefix (that's a rebuild feature, right? I guess then the problem is 
that rebuild doesn't know about DSSS's conventions.)


 I don't know whether this situation will improve. To be honest, I hardly 
 consider the fact that DSSS is not psychic a bug. As I already said, 
 it's not something that can be fixed magically, it's actually quite 
 complex. If you're going to be forcing DSSS to jump through multiple 
 environments, just clean up after yourself.
 
 Do a `dsss distclean` before building with a different compiler.


That's not a very good solution to the problem of maintaining 
multi-flavored builds.  Currently if someone wants to make a library 
available to people in more than one config, testing the different 
configs means they have to basically re-install from scratch every 
library that their library depends on before rebuilding their own 
library and test apps [1].  DSSS is set up to handle a DMD/GDC 
distinction in the libraries with that type code in the names.  That's 
good. But the version-specific pragma(link,...) in the di files prevents 
that from being very helpful.  And DSSS net's penchant for destroying 
builds of a different flavor than the current one is also a problem.

Ok, so there are really two issues here - one is how dsss installs stuff 
(the pragmas added to .di's and blowing away different lib versions). 
And the other is how it builds stuff locally.  The install problem I 
would call major.  The fact that local build products all go in the same 
dsss_imports and dsss_objs directories is a less serious issue, but it's 
still an issue when it comes to scalability.  If I need to go between 
dmd and gdc builds -- or even release and debug builds -- of 
some-huge-project, it could take a lot of time to switch if I'm required 
to distclean every time.  Perhaps for the local build products there 
could be a way for the user to specify project 'configs' that would 
control the subdirectory into which build products are placed.

--bb

[1] I guess an alternative would be to maintain 2 (or more) completely 
separate dsss installs that you switch between by manipulating your PATH 
variable.
Aug 28 2007