www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How to correctly deal with dmd.conf with multiple dmd installations -

reply aliak <something something.com> writes:
Alo,

I'm wondering what’s the deal with dmd.conf and what’s the 
correct way to handle it with dmd installations.

Scenario: `brew install dmd` puts dmd in `/usr/local/bin/dmd` 
(symlink) and sets the `SYSCONFDIR` in its build config so 
`dmd.conf` is in `/usr/local/etc/dmd.conf`.

AFAICT: that seems correct. Is it?

The problem comes with tools like digger. Which seems to be part 
of DMD’s PR process as D bot seems to love it :) so I guess it's 
officially supported? - plus it seems super convenient for dmd 
development.

But anyway, it’s broken on the mac if you use brew (and brew is 
[probably?] defacto). There is a workaround when running `digger 
build` but it’s not nice, and you’d never be able to guess it, 
and it’s error prone [0].

When you do `digger install` it seems to not “install” a 
`dmd.conf` but it does install the `dmd` binary in 
`/usr/local/bin/dmd` - but that wasn’t built with `SYSCONFDIR` so 
it doesn’t find `/usr/local/etc/dmd.conf` either, but even if it 
did, that’s the wrong `dmd.conf` (i.e. not the one that the build 
was based on - so would it even point to the right env?).

So basically, is dmd.conf supposed to go along with an ad hoc 
installation of dmd?

And, is dmd.conf necessary or can I safely ignore its existence? 
(it seems like maybe dub depends on it though so maybe not).

Cheers,
- Ali

[0]: https://github.com/CyberShadow/Digger/issues/74
Sep 25 2018
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, September 25, 2018 4:34:17 AM MDT aliak via Digitalmars-d wrote:

 Alo,

 I'm wondering what’s the deal with dmd.conf and what’s the
 correct way to handle it with dmd installations.

 Scenario: `brew install dmd` puts dmd in `/usr/local/bin/dmd`
 (symlink) and sets the `SYSCONFDIR` in its build config so
 `dmd.conf` is in `/usr/local/etc/dmd.conf`.

 AFAICT: that seems correct. Is it?

 The problem comes with tools like digger. Which seems to be part
 of DMD’s PR process as D bot seems to love it :) so I guess it's
 officially supported? - plus it seems super convenient for dmd
 development.

 But anyway, it’s broken on the mac if you use brew (and brew is
 [probably?] defacto). There is a workaround when running `digger
 build` but it’s not nice, and you’d never be able to guess it,
 and it’s error prone [0].

 When you do `digger install` it seems to not “install” a
 `dmd.conf` but it does install the `dmd` binary in
 `/usr/local/bin/dmd` - but that wasn’t built with `SYSCONFDIR` so
 it doesn’t find `/usr/local/etc/dmd.conf` either, but even if it
 did, that’s the wrong `dmd.conf` (i.e. not the one that the build
 was based on - so would it even point to the right env?).

 So basically, is dmd.conf supposed to go along with an ad hoc
 installation of dmd?

 And, is dmd.conf necessary or can I safely ignore its existence?
 (it seems like maybe dub depends on it though so maybe not).

 Cheers,
 - Ali

 [0]: https://github.com/CyberShadow/Digger/issues/74


Yes. You want dmd.conf, or certain paths won't be set correctly (like where
to find Phobos). Personally, I'd strongly suggest against having multiple
copies of dmd installed at the same time. It just sounds like a recipe for
disaster. Regardless, here's the description for how dmd finds dmd.conf:

https://dlang.org/dmd-linux.html#dmd-conf

- Jonathan M Davis
Sep 25 2018
parent reply Jacob Carlborg <doob me.com> writes:
On 2018-09-25 13:55, Jonathan M Davis wrote:


 Yes. You want dmd.conf, or certain paths won't be set correctly (like where
 to find Phobos). Personally, I'd strongly suggest against having multiple
 copies of dmd installed at the same time. It just sounds like a recipe for
 disaster. Regardless, here's the description for how dmd finds dmd.conf:
 
 https://dlang.org/dmd-linux.html#dmd-conf


Having multiple copies of DMD (or rather multiple versions) works 
perfectly fine. That's the entire point of DVM [1]. But I've never used 
a global dmd.conf file either.

[1] https://github.com/jacob-carlborg/dvm

-- 
/Jacob Carlborg
Sep 25 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Sep 25, 2018 at 09:00:57PM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 2018-09-25 13:55, Jonathan M Davis wrote:
 

 Yes. You want dmd.conf, or certain paths won't be set correctly
 (like where to find Phobos). Personally, I'd strongly suggest
 against having multiple copies of dmd installed at the same time. It
 just sounds like a recipe for disaster. Regardless, here's the
 description for how dmd finds dmd.conf:
 
 https://dlang.org/dmd-linux.html#dmd-conf


 
 Having multiple copies of DMD (or rather multiple versions) works
 perfectly fine. That's the entire point of DVM [1]. But I've never
 used a global dmd.conf file either.


[...]

Unlike many other similar software, dmd works remarkably well when
installed with multiple copies of itself.  Provided you setup a suitable
dmd.conf in the same directory as the executable, and pointing to the
appropriate versions of druntime/phobos.  (The latter two are more
liable to cause headaches if you point dmd to the wrong versions
thereof. When in doubt, run dmd -v to see exactly which dmd.conf and
which druntime/phobos are being used.)

If you're using multiple versions of dmd, though, I'd highly recommend
*not* having a /etc/dmd.conf (nor a $HOME/dmd.conf), because all
versions of dmd will look there, and all except one will pick up the
wrong settings. Save yourself the ensuing headache.


T

-- 
If the comments and the code disagree, it's likely that *both* are wrong. --
Christopher
Sep 25 2018
prev sibling next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 09/25/2018 06:34 AM, aliak wrote:

 Alo,
 
 I'm wondering what’s the deal with dmd.conf and what’s the correct way 
 to handle it with dmd installations.
 


Basically, you want an appropriate, matching `dmd.conf` together with 
(ie, "in the same directory as") each `dmd` executable. No other 
`dmd.conf` should ever exist in any place where it could potentially be 
picked up by a D compiler.[1]

Anything other than that just breeds problems and totally screws up 
normally-perfectly-safe things like upgrading the compiler or having 
multiple compilers installed. Using that rule, I've NEVER had any 
problems from having many different compilers installed.

I'm not familiar with how digger or brew work, so I can't really comment 
on whether they do things safely or not.

[1] There USED to arguably be a (sorta-)good use for having a common 
`dmd.conf` in a non-compiler-specific location: That was in the D1 days 
when there was no DUB, and the built-in `dmd.conf` was trivial and 
rarely changed. In those days, `dmd.conf` could be used as a really 
hacky substitute for a package manager. But there were still 
downsides/dangers to that then, and even more now, so it's really not a 
recommended approach anymore.
Sep 25 2018
prev sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Tuesday, 25 September 2018 at 10:34:17 UTC, aliak wrote:

 When you do `digger install` it seems to not “install” a 
 `dmd.conf` but it does install the `dmd` binary in 
 `/usr/local/bin/dmd` - but that wasn’t built with `SYSCONFDIR` 
 so it doesn’t find `/usr/local/etc/dmd.conf` either, but even 
 if it did, that’s the wrong `dmd.conf` (i.e. not the one that 
 the build was based on - so would it even point to the right 
 env?).


The solution I'm thinking of for Digger is to attempt to detect 
the SYSCONFDIR of the current `dmd` binary in `PATH`, and use 
that by default when Digger builds D. Doing so would break 
Digger's "maximally reproducible builds" policy a little, so we'd 
need to be loud about it.

An alternative would be to complain loudly when attempting to 
build or (especially) install a compiler with a mismatching 
SYSCONFDIR, but still proceed with the default.
Sep 25 2018
parent reply aliak <something something.com> writes:
On Wednesday, 26 September 2018 at 01:17:24 UTC, Vladimir 
Panteleev wrote:

 On Tuesday, 25 September 2018 at 10:34:17 UTC, aliak wrote:

 When you do `digger install` it seems to not “install” a 
 `dmd.conf` but it does install the `dmd` binary in 
 `/usr/local/bin/dmd` - but that wasn’t built with `SYSCONFDIR` 
 so it doesn’t find `/usr/local/etc/dmd.conf` either, but even 
 if it did, that’s the wrong `dmd.conf` (i.e. not the one that 
 the build was based on - so would it even point to the right 
 env?).


 The solution I'm thinking of for Digger is to attempt to detect 
 the SYSCONFDIR of the current `dmd` binary in `PATH`, and use 
 that by default when Digger builds D. Doing so would break 
 Digger's "maximally reproducible builds" policy a little, so 
 we'd need to be loud about it.


Can you explain a bit maybe how it'd break the maximally 
reproducible builds with an example? I believe you might've 
mentioned that in the issue linked but I didn't quite get it.


 An alternative would be to complain loudly when attempting to 
 build or (especially) install a compiler with a mismatching 
 SYSCONFDIR, but still proceed with the default.


So let's say dmd in PATH is olddmd and the one we want to build 
with digger is newdmd.

For the first option, are you suggesting:
* digger build newdmd
   1. confFileName = parseFileNameFrom(olddmd -v | grep Config)
   2. build dmd with SYSCONFDIR=pluckPathFrom(confFileName)

If yes, then that means the newdmd would use olddmd's phobos/lib 
right?

Another idea: create a $HOME/.digger dir at build time. Then 
digger build could set SYSCONFDIR=$HOME/.digger. And newdmd in 
work/result/bin/will be fine because there's a dmd.conf near the 
exe that gets priority.

digger install (assuming `which dmd` = /usr/local/bin/dmd)
- if exists(/usr/bin/local/olddmd)
   move olddmd -> $HOME/.digger/uninstall/ (with whatever else you 
need)
- if exists($HOME/dmd.conf)
   move it to $HOME/.digger/uninstall/ (with metadata saying where 
it came form)
- copy newdmd to /usr/bin/local/
- copy work/result/import to $HOME/.digger/import
- copy work/result/lib to $HOME/.digger/lib
- write a new dmd.conf to $HOME/.digger with contents:
"""
[Environment]
DFLAGS=-I$HOME/.digger/import -L-L$HOME/.digger/lib
"""

/usr/local/bin/newdmd will pick up the dmd.conf that was just 
written and point to the correct includes and lib because 
SYSCONFDIR was set at build time, and $HOME/dmd.conf was moved 
during installation (from the link Jonathan posted it seems like 
that would be necessary because it's given priority over 
SYSCONFDIR).

An uninstall will move $HOME/.digger/uninstall/olddmd and 
dmd.conf back to their places and remove contents of $HOME/.digger

In my head at least that works and keeps the current behavior as 
is. I'm not sure about Windows though because I'm not too 
familiar with paths, permissions, etc.

PS: I would be happy to help to get osx + brew working with your 
"maximal reproducibility" given time and guidance.

Cheers,
- Ali
Sep 27 2018
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Thursday, 27 September 2018 at 19:53:32 UTC, aliak wrote:

 Can you explain a bit maybe how it'd break the maximally 
 reproducible builds with an example? I believe you might've 
 mentioned that in the issue linked but I didn't quite get it.


Well, essentially Digger tries to minimize the number of things 
from its environment that affect the resulting DMD binary, so 
that "digger build <something>" on one machine has the maximal 
chance of producing a DMD binary that is or behaves exactly like 
the result of "digger build <something>" on another machine. 
Fully reproducible builds provide some guarantees which are very 
useful in cases like "this program crashes when built and run on 
machine A but not B", in which case it would allow eliminating 
the possibility of A having a bad compiler (but we aren't there 
with FULLY reproducible builds yet).

Now, some variables like platform or system libraries are 
inevitable, but others, like environment variables or certain 
dependencies, can be controlled. In this case, if we opt in to 
autodetection, a DMD built with the default options on one 
machine with brew won't be the same as a DMD built on a machine 
without brew.


 So let's say dmd in PATH is olddmd and the one we want to build 
 with digger is newdmd.

 For the first option, are you suggesting:
 * digger build newdmd
   1. confFileName = parseFileNameFrom(olddmd -v | grep Config)
   2. build dmd with SYSCONFDIR=pluckPathFrom(confFileName)

 If yes, then that means the newdmd would use olddmd's 
 phobos/lib right?


Well, "digger install" would upgrade Phobos too, but otherwise 
that's the general idea.


 Another idea: create a $HOME/.digger dir at build time. Then 
 digger build could set SYSCONFDIR=$HOME/.digger. And newdmd in 
 work/result/bin/will be fine because there's a dmd.conf near 
 the exe that gets priority.


That's fine until it gets installed systemwide, then problems 
begin.


 digger install (assuming `which dmd` = /usr/local/bin/dmd)
 - if exists(/usr/bin/local/olddmd)
   move olddmd -> $HOME/.digger/uninstall/ (with whatever else 
 you need)
 - if exists($HOME/dmd.conf)
   move it to $HOME/.digger/uninstall/ (with metadata saying 
 where it came form)
 - copy newdmd to /usr/bin/local/
 - copy work/result/import to $HOME/.digger/import
 - copy work/result/lib to $HOME/.digger/lib
 - write a new dmd.conf to $HOME/.digger with contents:
 """
 [Environment]
 DFLAGS=-I$HOME/.digger/import -L-L$HOME/.digger/lib
 """


What's the advantage of this over just adding 
digger/work/result/bin to PATH?
Sep 27 2018