www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Orbit - Package Manager - Specification/ideas

reply Jacob Carlborg <doob me.com> writes:
I've written a more formal specification of my ideas for a package 
manager for D.

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Note that I am exploring the possibility of using D as the language for 
all the files mentioned in the link above.

The current status is that building packages and installing them works, 
but quite limited. No dependency tracking or central repository so far.

Please comment and suggest.

-- 
/Jacob Carlborg
Jul 13 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Just curious, why do people prefer using extensionless files? I mean,
I can easily open them up in an editor, but I can't set up syntax
highlighting without knowing what type the file is.
Jul 13 2011
parent reply "Nick Sabalausky" <a a.a> writes:
"Andrej Mitrovic" <andrej.mitrovich gmail.com> wrote in message 
news:mailman.1613.1310586759.14074.digitalmars-d puremagic.com...
 Just curious, why do people prefer using extensionless files? I mean,
 I can easily open them up in an editor, but I can't set up syntax
 highlighting without knowing what type the file is.

Programmer's Notepad 2 lets me map an extention *or* a filename to a syntax highlighting profile. But I agree, it's still easier to just use an existing extension. I suspect it's partly a just buildsystem tradition, and partly because it's not very common to use/need a buildscript with a name other than the standard one. ------------------------------- Not sent from an iPhone.
Jul 13 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-07-13 22:14, Nick Sabalausky wrote:
 "Andrej Mitrovic"<andrej.mitrovich gmail.com>  wrote in message
 news:mailman.1613.1310586759.14074.digitalmars-d puremagic.com...
 Just curious, why do people prefer using extensionless files? I mean,
 I can easily open them up in an editor, but I can't set up syntax
 highlighting without knowing what type the file is.

Programmer's Notepad 2 lets me map an extention *or* a filename to a syntax highlighting profile. But I agree, it's still easier to just use an existing extension. I suspect it's partly a just buildsystem tradition, and partly because it's not very common to use/need a buildscript with a name other than the standard one. ------------------------------- Not sent from an iPhone.

Something like that. -- /Jacob Carlborg
Jul 14 2011
prev sibling next sibling parent reply jdrewsen <jdrewsen nospam.com> writes:
Den 13-07-2011 21:19, Jacob Carlborg skrev:
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language for
 all the files mentioned in the link above.

 The current status is that building packages and installing them works,
 but quite limited. No dependency tracking or central repository so far.

 Please comment and suggest.

Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory? Orbfile section: The "orb" command that accepts git/hg/svn repositories should also allow for a tag/commit parameter I think. The "orb" commands second parameter could also be a list of serveral repositories to try in order for fallback. I guess a user configuration file in ~/.orb could contain "source" commands as well. Orb package section: I think the versioning scheme should be set in stone actually. Most other packaging systems does that. It makes your life much easier. Central repository section: Please let us settle for one format for the metadata.xxx file. My vote is for json or yaml. XML is too verbose for my taste. I also think that it should be compressed e.g. metadata.json.bzip since it will quickly grow quite large and the packaging system has to be fast. Maybe add the build revision on /orb/<package>-<version>_<build> since it is quite common reupload the same package version with at simple build fix. Additionally the architecture should be added to the name: /orb/<package>-<version>_<build>-<arch> Now it is just like how debian files look like :) Maybe put the file in an arch subdir /orb/<arch>/<package>-<version>_<build> I'm really in favor of doing this in D instead of ruby though. /Jonas
Jul 13 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-07-13 22:20, jdrewsen wrote:
 Den 13-07-2011 21:19, Jacob Carlborg skrev:
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language for
 all the files mentioned in the link above.

 The current status is that building packages and installing them works,
 but quite limited. No dependency tracking or central repository so far.

 Please comment and suggest.

Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory?

I'm not sure if this is possible or not (specially on multiple platforms) but I was thinking something like this: Say that you have several versions of the dwt package installed and you want to use a particular version, you can run the command like this: "orb use dwt --version 0.3.5" Then when you link with the "dwt" library it will link with the 0.3.5 version.
 Orbfile section:

 The "orb" command that accepts git/hg/svn repositories should also allow
 for a tag/commit parameter I think.

Absolutely.
 The "orb" commands second parameter could also be a list of serveral
 repositories to try in order for fallback.

That might be a good idea.
 I guess a user configuration file in ~/.orb could contain "source"
 commands as well.

Sounds reasonable.
 Orb package section:

 I think the versioning scheme should be set in stone actually. Most
 other packaging systems does that. It makes your life much easier.

So you mean I shouldn't allow custom version strings?
 Central repository section:

 Please let us settle for one format for the metadata.xxx file. My vote
 is for json or yaml. XML is too verbose for my taste. I also think that
 it should be compressed e.g. metadata.json.bzip since it will quickly
 grow quite large and the packaging system has to be fast.

I haven't actually thought about what format to use. I just listed a few reasonable formats. Settling for one format is probably a good thing. Probably json since both Phobos and Tango has json modules and it's available in most languages.
 Maybe add the build revision on /orb/<package>-<version>_<build>
 since it is quite common reupload the same package version with at
 simple build fix.

If you read the version scheme I was thinking about having the last number as a build number: major.minor.build * Increment "build" when an implementation detail change * Increment "minor" when a non-braking API change is made, i.e. adding a new function * Increment "major" when a braking API change is made, i.e. removing a function Maybe there should be a patch level of some king as well and alpha, beta and release candidates as well.
 Additionally the architecture should be added to the name:
 /orb/<package>-<version>_<build>-<arch>
 Now it is just like how debian files look like :)

Hm, this will take some though. I don't like the idea that the developer has to create multiple packages for a single package.
 Maybe put the file in an arch subdir
 /orb/<arch>/<package>-<version>_<build>

That is probably better. I haven't actually thought that much about binary packages. I was mostly thinking about source packages that need to be built when installing them. Source packages don't have these problems.
 I'm really in favor of doing this in D instead of ruby though.

 /Jonas

I guess most people are. -- /Jacob Carlborg
Jul 14 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-14 11:07, Johannes Pfau wrote:
 jdrewsen wrote:
 Den 13-07-2011 21:19, Jacob Carlborg skrev:
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language
 for all the files mentioned in the link above.

 The current status is that building packages and installing them
 works, but quite limited. No dependency tracking or central
 repository so far.

 Please comment and suggest.

Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory? Orbfile section: The "orb" command that accepts git/hg/svn repositories should also allow for a tag/commit parameter I think. The "orb" commands second parameter could also be a list of serveral repositories to try in order for fallback. I guess a user configuration file in ~/.orb could contain "source" commands as well. Orb package section: I think the versioning scheme should be set in stone actually. Most other packaging systems does that. It makes your life much easier. Central repository section: Please let us settle for one format for the metadata.xxx file. My vote is for json or yaml. XML is too verbose for my taste. I also think that it should be compressed e.g. metadata.json.bzip since it will quickly grow quite large and the packaging system has to be fast. Maybe add the build revision on /orb/<package>-<version>_<build> since it is quite common reupload the same package version with at simple build fix. Additionally the architecture should be added to the name: /orb/<package>-<version>_<build>-<arch> Now it is just like how debian files look like :) Maybe put the file in an arch subdir /orb/<arch>/<package>-<version>_<build> I'm really in favor of doing this in D instead of ruby though. /Jonas

And in Package SubTypes: No documentation subtype for now?

I though they were quite self explanatory. But I can add documentation for them if necessary.
 Do Library and Dynamic library always contain the .d/.di headers?

Library (which is static library) needs to contain headers yes. Dynamic library will probably contain headers as well.
 In orbspec:
 The callbacks are run when the package is being build, correct? (The
 install callback could also be called when the package is being
 installed, but I think we don't need this functionality)

I see now that I have to give this some thought. The "build" callback could be called both when all files are built into the package and when the package later is built to be installed.
 So those are the hooks to use any custom build system?

Yes and no. There is also the "build" field which allow you yo call a custom shell script or custom build tool. Something like this: build :shell, "my_custom_build_script.sh"
 Are Check Platform and Set Platform really needed in the orbspec? I
 think this can be left to drake or other build systems, can't we keep
 orbspecs platform agnostic?

Absolutely, the orbspecs are as platform agnostic as possible. But I see no harm in having the possibility to check the platform. I'm pretty sure at least someone will find a need for it.
 I'd make build_dependencies and dependencies a required field.
 (Although empty is ok. Can we distinguish between not set and empty?)

I see no point in setting the build_dependencies and dependencies fields if the package doesn't have any dependencies. nil/null would be not set and an empty array (in this case) would be empty. -- /Jacob Carlborg
Jul 15 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-07-15 11:21, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-14 11:07, Johannes Pfau wrote:
 And in Package SubTypes:
 No documentation subtype for now?

I though they were quite self explanatory. But I can add documentation for them if necessary.

Sorry, my question wasn't clear: I meant where will 'api documentation' for libraries go? In a special 'documentation' subtype or in the library packages?

Aha. In the library package. Or is there a need for a package containing just API documentation?
 I see no point in setting the build_dependencies and dependencies
 fields if the package doesn't have any dependencies. nil/null would be
 not set and an empty array (in this case) would be empty.

Well, having to set it explicitly makes it less likely to forget those fields, but that's the only reason.

Ok, I don't think it would be necessary. -- /Jacob Carlborg
Jul 15 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
jdrewsen wrote:
Den 13-07-2011 21:19, Jacob Carlborg skrev:
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language
 for all the files mentioned in the link above.

 The current status is that building packages and installing them
 works, but quite limited. No dependency tracking or central
 repository so far.

 Please comment and suggest.

Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory? Orbfile section: The "orb" command that accepts git/hg/svn repositories should also allow for a tag/commit parameter I think. The "orb" commands second parameter could also be a list of serveral repositories to try in order for fallback. I guess a user configuration file in ~/.orb could contain "source" commands as well. Orb package section: I think the versioning scheme should be set in stone actually. Most other packaging systems does that. It makes your life much easier. Central repository section: Please let us settle for one format for the metadata.xxx file. My vote is for json or yaml. XML is too verbose for my taste. I also think that it should be compressed e.g. metadata.json.bzip since it will quickly grow quite large and the packaging system has to be fast. Maybe add the build revision on /orb/<package>-<version>_<build> since it is quite common reupload the same package version with at simple build fix. Additionally the architecture should be added to the name: /orb/<package>-<version>_<build>-<arch> Now it is just like how debian files look like :) Maybe put the file in an arch subdir /orb/<arch>/<package>-<version>_<build> I'm really in favor of doing this in D instead of ruby though. /Jonas

And in Package SubTypes: No documentation subtype for now? Do Library and Dynamic library always contain the .d/.di headers? In orbspec: The callbacks are run when the package is being build, correct? (The install callback could also be called when the package is being installed, but I think we don't need this functionality) So those are the hooks to use any custom build system? Are Check Platform and Set Platform really needed in the orbspec? I think this can be left to drake or other build systems, can't we keep orbspecs platform agnostic? I'd make build_dependencies and dependencies a required field. (Although empty is ok. Can we distinguish between not set and empty?) -- Johannes Pfau
Jul 14 2011
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ivkrdj$ci4$1 digitalmars.com...
 I've written a more formal specification of my ideas for a package manager 
 for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language for 
 all the files mentioned in the link above.

 The current status is that building packages and installing them works, 
 but quite limited. No dependency tracking or central repository so far.

 Please comment and suggest.

Good start :) Here are my random thoughts on it (sorry if you've already answered some of them, I haven't read the rest of this thread yet): - I find the "orb" vs "orbit" distinction a little confusing and unnecessary. Why not just call it all "orb"? (Or call everything "orbit"?) - I think the files "orbfile", "{name}.orbspec" and "metadata" (the one inside the packages) should all just be one file. Frankly, the existence of all three of them confuses the hell out of me. - Are the subtypes a "pick one" or "pick all that are included" deal? I think that latter would make more sense. - Instead of "~> 0.3.4", what about "> 0.3+.4+"? Or ""~> 0.3+.4+"? Or something vaguely like that. That would be more flexible. - What happens if someone tries to upload a newer version with the same old version number? Or if they forcefully do it? - Are version numbers allowed to have more or less than three parts? I think they should. Do version comparisons still work on version numbers with an arbitrary number of parts? Again, I think they should. - Is version "2.10" after "2.1" or are they the same? What about "2.01" vs "2.1"? I would vote for "2.10 > 2.1 == 2.01", because I see version "parts" as distinct numbers separated by a period, rather than fractional numbers. - It should allow boolean operators and parens for the version selections. For instance: "(>= 2.1 && <= 2.6 && != 2.4) || >= 3.4" (Ie, "Any version from 2.1 through 2.6, but 2.4 has critical bugs, and 3.4+ contains a 2.x compatibility layer.") - There should be a "list" command and a "list {package name}" command to see what's installed. Maybe even "list {package name} {version expression}"? And maybe something too see what's available but not yet installed? They should all list the versions in guaranteed-sorted order so you can see which are the newest and oldest installed versions by looking at the first and last. - At some point, 7z should be supported (and tarballs, of course). - How should platforms be handled WRT packages? Ie, Do all platforms need to be in the same orb package? Do they all need to be in separate packages? Either way? If they're not all required to be in the same package, how does orb find the package that had the right platform? - Is it really necessary to have separate "build_dependencies" and "runtime_dependencies"? And why have both "runtime_dependencies" and "orbs" instead of just picking one name and sticking with it? - Would it be a good idea to have and additional field "extra" for non-standard expandability? So people could add extra fields they felt would be useful, like "extra.foo" and "extra.bar", etc. And popular ones could eventually be formally added to the specification as just simply "foo" and "bar". - What's the point of the fields "files", "libraries" and "executables"? Seems like extra work for no real benefit. - This supports having multiple versions of the same package installed at the same time, right? If not it should. - I see there's an "upgrade" callback, but I didn't see an "upgrade" command. Is upgrading in or out? I think that there should be an "upgrade" command that upgrades the installed versions of packages as far as it can *without* breaking any other installed packages that depend on it. Ex, if Foo requires Bar v2.6 or earlier, and SuperFoo requires Bar v2.7 or earlier, and Bar v2.3 is installed, but the latest Bar is v2.9, then "upgrade bar" would upgrade Bar v2.3 to v2.6 and display a message that says "Bar upgraded from 2.3 to 2.6, but the newest is 2.9, run "orb install Bar" to install the newest Bar, too." (Or maybe it should install both 2.6 and 2.7? Or one/both of those and 2.9?) For upgrading, we should also think about how to do upgrades without clobbering any of it's settings. - For POSTing a package to a repository, how does authentication work? All repos don't have to provide unrestricted upload access do they? - I'm not sure I understand how the "source" command works. Can it be provided more than once? And then it just picks the first one that actually has the package? - The "central repositories" don't necessarily sound all that central, so they probably should just be called "repositories". - What about default repositories? It should support that. (Kinda makes sense, otherwise how would "orb install xxx" know where to look?) And there should be simple commands to add/update/remove/list (and reorder?) the default repositories. If a package A specifies a dependency B and a repository for that dependency B, then which one has priority for downloading B: The default repositories or the repository specified by package A? - Here's a problem with using an actual programming language for the orbfile/name.orbspec/metadata file: Suppose Orb version X uses Ruby version Y. Then, Orb X+1 comes out and which has Ruby upgraded to Y+1. Now, someone creates PackageA with an orbfile/name.orbspec/whatever that relies on Ruby Y+1. Someone else still has Orb version X and tries to get PackageA. Kaboom! Therefore, the orbfile/name.orbspec/whatever needs to specify which version of Orb (or Ruby) it requires. But now we have a chicken-and-the-egg problem: How can Orb X figure out that PackageA requires Orb X+1 if Orb X can't properly read PackageA's orbfile/name.orbspec/whatever? - If I install D library "libfoo", then I should be able write myapp.d with "include foo.blah;" and then do "dmd myapp.d" *without* manually specifying -Ipath_to_libfoo. It should just work. How will Orb handle that? And how will that interact with DVM? Ie, if I do "dvm use 2.051", then "orb install libfoo", then "dvm use 2.054", then I should still have access to libfoo without needing to specify -Ipath_to_libfoo. - Where does everything get installed? - In many ways this sounds a lot like a generalized DVM. Maybe Orb should eventually take over DVM's duties by making a DMD orb package.
Jul 15 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-15 09:14, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ivkrdj$ci4$1 digitalmars.com...
 I've written a more formal specification of my ideas for a package manager
 for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language for
 all the files mentioned in the link above.

 The current status is that building packages and installing them works,
 but quite limited. No dependency tracking or central repository so far.

 Please comment and suggest.

Good start :) Here are my random thoughts on it (sorry if you've already answered some of them, I haven't read the rest of this thread yet): - I find the "orb" vs "orbit" distinction a little confusing and unnecessary. Why not just call it all "orb"? (Or call everything "orbit"?)

This is how I'm thinking: Orbit is the name of the package manager, "orb" is what you type on the command line when interacting with the package manager. A package is also called an "orb".
 - I think the files "orbfile", "{name}.orbspec" and "metadata" (the one
 inside the packages) should all just be one file. Frankly, the existence of
 all three of them confuses the hell out of me.

Ok, this is how it works: "orbfile" is a completely optional file listing all packages a project depends on, it shouldn't contain anything else. It allows you to run "orb install" in the project directory to install all needed packages. Just because a project uses packages doesn't mean it self needs to be a package. The orbspec file requires more than just listing dependencies. I guess the tool could look for an .orbspec file and install all its dependencies. But what happends if it finds several .orbspec files in a directory. This is just like the Gemfile for those how have used bundler: http://gembundler.com/ "orbspec" is a specification of how a package looks like and what it contains. It's intended for creating a package out of a project. "metadata" is basically the orbspec copied into the archive. I was first thinking about "compiling" down the Ruby code into YAML or JSON but for now the Ruby code is included in the archive.
 - Are the subtypes a "pick one" or "pick all that are included" deal? I
 think that latter would make more sense.

I'm thinking this can be inferred from other fields like "executables" and "libraries". A package could contain several types.
 - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>  0.3+.4+"? Or
 something vaguely like that. That would be more flexible.

So you mean I can have a version like this: "> 0.3.4+" meaning any version from "0.3.4" to "0.3.9"? It might be a good idea.
 - What happens if someone tries to upload a newer version with the same old
 version number? Or if they forcefully do it?

I haven't thought about that. It probably shouldn't be possible.
 - Are version numbers allowed to have more or less than three parts? I think
 they should. Do version comparisons still work on version numbers with an
 arbitrary number of parts? Again, I think they should.

I was hoping to only have version numbers with three parts. If fewer parts are used it would probably easiest to infer a 0 for the missing parts, i.e. "1" == "1.0.0". Is there a need for more parts than three? The whole idea of having three version parts is to be able to use "~> 0.3.4". But if "> 0.3.4+" would be allowed then arbitrary number of parts could be allowed.
 - Is version "2.10" after "2.1" or are they the same? What about "2.01" vs
 "2.1"? I would vote for "2.10>  2.1 == 2.01", because I see version "parts"
 as distinct numbers separated by a period, rather than fractional numbers.

I haven't thought about that. I see version parts as distinct numbers as well.
 - It should allow boolean operators and parens for the version selections.
 For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>= 3.4" (Ie, "Any version
 from 2.1 through 2.6, but 2.4 has critical bugs, and 3.4+ contains a 2.x
 compatibility layer.")

Hehe. Now this is getting quite complicate, but it would be nice to have yes. Not something I will aim for in the first release.
 - There should be a "list" command and a "list {package name}" command to
 see what's installed. Maybe even "list {package name} {version expression}"?
 And maybe something too see what's available but not yet installed? They
 should all list the versions in guaranteed-sorted order so you can see which
 are the newest and oldest installed versions by looking at the first and
 last.

Absolutely, I've completely forgot about the "list" command.
 - At some point, 7z should be supported (and tarballs, of course).

If someone is willing to create D module of create bindings for any available libraries. Any libraries the forces a specific license is out of the question.
 - How should platforms be handled WRT packages? Ie, Do all platforms need to
 be in the same orb package? Do they all need to be in separate packages?
 Either way? If they're not all required to be in the same package, how does
 orb find the package that had the right platform?

I good questions. I haven't given binary packages that much thought. I was first going for a source only package manager that requires all packages to be built before installed. I see three options: * One package for all platforms * Include the platform in the package name and in the orbspec * Have a sub path (on the server) for every platform, i.e.: dorbit.org/orbs/linux/dwt-1.3.2.orb.zip
 - Is it really necessary to have separate "build_dependencies" and
 "runtime_dependencies"? And why have both "runtime_dependencies" and "orbs"
 instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. Therefore it would be unnecessary to do a permanent installation on those dependencies. The user could get an option to either permanently installed these dependencies or to temporarily install them. The other way around would be possible as well. A package can depend on a dynamically linked library and use it only through function pointers. Then the package would only have a runtime dependency on the library, i.e. it wouldn't be needed when building.
 - Would it be a good idea to have and additional field "extra" for
 non-standard expandability? So people could add extra fields they felt would
 be useful, like "extra.foo" and "extra.bar", etc. And popular ones could
 eventually be formally added to the specification as just simply "foo" and
 "bar".

I guess there could be an "extra" field accepting a hash. How would the field be used, by other tools?
 - What's the point of the fields "files", "libraries" and "executables"?
 Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. "libraries" and "executables" are all libraries and executables that should be installed (regardless if they are pre-compiled or built during installation).
 - This supports having multiple versions of the same package installed at
 the same time, right? If not it should.

Yes, that's the whole point of having version, as I see it.
 - I see there's an "upgrade" callback, but I didn't see an "upgrade"
 command. Is upgrading in or out? I think that there should be an "upgrade"
 command that upgrades the installed versions of packages as far as it can
 *without* breaking any other installed packages that depend on it. Ex, if
 Foo requires Bar v2.6 or earlier, and SuperFoo requires Bar v2.7 or earlier,
 and Bar v2.3 is installed, but the latest Bar is v2.9, then "upgrade bar"
 would upgrade Bar v2.3 to v2.6 and display a message that says "Bar upgraded
 from 2.3 to 2.6, but the newest is 2.9, run "orb install Bar" to install the
 newest Bar, too." (Or maybe it should install both 2.6 and 2.7? Or one/both
 of those and 2.9?) For upgrading, we should also think about how to do
 upgrades without clobbering any of it's settings.

I guess I didn't think this through. I think it will require some thought.
 - For POSTing a package to a repository, how does authentication work? All
 repos don't have to provide unrestricted upload access do they?

I haven't thought about this more than there will be some kind of authentication. Probably HTTP basic authentication.
 - I'm not sure I understand how the "source" command works. Can it be
 provided more than once? And then it just picks the first one that actually
 has the package?

The "source" command specifies a path to a repository where to fetch packages from. I haven't thought about if the it can be provided more than once. It might be a good idea.
 - The "central repositories" don't necessarily sound all that central, so
 they probably should just be called "repositories".

Ok.
 - What about default repositories? It should support that. (Kinda makes
 sense, otherwise how would "orb install xxx" know where to look?) And there
 should be simple commands to add/update/remove/list (and reorder?) the
 default repositories. If a package A specifies a dependency B and a
 repository for that dependency B, then which one has priority for
 downloading B: The default repositories or the repository specified by
 package A?

Yes there will be a default repository. As someone else suggested there could be an "orbfile" in the users home directory that can contain default settings like for "source". Or a more general config file for orbit.
 - Here's a problem with using an actual programming language for the
 orbfile/name.orbspec/metadata file: Suppose Orb version X uses Ruby version
 Y. Then, Orb X+1 comes out and which has Ruby upgraded to Y+1. Now, someone
 creates PackageA with an orbfile/name.orbspec/whatever that relies on Ruby
 Y+1. Someone else still has Orb version X and tries to get PackageA. Kaboom!

Yeah, that is a problem. But I wonder how much this will be a problem in practice. I don't think this will so big problem in practice using Ruby as the language. On the other hand, using D, will be a big problem. D breaks something in every release.
 Therefore, the orbfile/name.orbspec/whatever needs to specify which version
 of Orb (or Ruby) it requires. But now we have a chicken-and-the-egg problem:
 How can Orb X figure out that PackageA requires Orb X+1 if Orb X can't
 properly read PackageA's orbfile/name.orbspec/whatever?

That is a problem. If the metadata is "compiled" YAML/JSON then we can get around this.
 - If I install D library "libfoo", then I should be able write myapp.d with
 "include foo.blah;" and then do "dmd myapp.d" *without* manually
 specifying -Ipath_to_libfoo. It should just work. How will Orb handle that?

It can't. The solution to this is a build tool, as I see it. The build tool knows about the package manager and let you specify dependencies on package. Think about Drake, it could look like this: target("myapp.d", { orb("libfoo"); // automatically links with "libfoo" and includes its header path. }); -Ipath_to_libfoo needs somehow be passed to the compiler, and linking with the library as well. Maybe it would be possible to manipulate the dmd.conf/sc.ini but this seems very complicated.
 And how will that interact with DVM? Ie, if I do "dvm use 2.051", then "orb
 install libfoo", then "dvm use 2.054", then I should still have access to
 libfoo without needing to specify -Ipath_to_libfoo.

If you do "dvm use 2.051", then "orb install libfoo" then "libfoo" will only be installed for dmd 2.051, that's the whole point. You would have to run "orb install libfoo" again after switching compiler. Orbit could of course share the same package if possible.
 - Where does everything get installed?

For now, on Posix, in /usr/local/orbit. If used through DVM it will be installed in somewhere in ~/.dvm.
 - In many ways this sounds a lot like a generalized DVM. Maybe Orb should
 eventually take over DVM's duties by making a DMD orb package.

No, I don't think so. DVM is quite specialized in what it does. Manipulating the PATH variable (or the registy on Windows) to be able to do what it does. I don't what to mix DVM and Orbit. -- /Jacob Carlborg
Jul 15 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-07-15 19:11, Johannes Pfau wrote:
 The lzma SDK is public domain (lzma is the compression algorithm used
 in 7z), xzutils has a library called liblzma which is based on the lzma
 sdk, easier to use and also public domain. Tar should be easy enough to
 implement in d. Seems like there are more or less recent D
 bindings for liblzma:
 http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=117344
 But someone still has to combine those parts together. LZMA bindings
 would also fit nicely into phobos, as we already have zlib in there.

When someone actually does this I can use lzma/7z instead of/in addition to zip. But I have no interest myself in creating bindings or building a D module. -- /Jacob Carlborg
Jul 15 2011
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ivpkdp$hq4$1 digitalmars.com...
 On 2011-07-15 09:14, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ivkrdj$ci4$1 digitalmars.com...
 I've written a more formal specification of my ideas for a package 
 manager
 for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language for
 all the files mentioned in the link above.

 The current status is that building packages and installing them works,
 but quite limited. No dependency tracking or central repository so far.

 Please comment and suggest.

Good start :) Here are my random thoughts on it (sorry if you've already answered some of them, I haven't read the rest of this thread yet): - I find the "orb" vs "orbit" distinction a little confusing and unnecessary. Why not just call it all "orb"? (Or call everything "orbit"?)

This is how I'm thinking: Orbit is the name of the package manager, "orb" is what you type on the command line when interacting with the package manager. A package is also called an "orb".

But why should the name of the project and the name of the tool be different? It would be less confusing to have a packager manager named Orb that gets invoked via "orb". (Or a package manager named Orbit that gets invoked via "orbit".
 - I think the files "orbfile", "{name}.orbspec" and "metadata" (the one
 inside the packages) should all just be one file. Frankly, the existence 
 of
 all three of them confuses the hell out of me.

Ok, this is how it works: "orbfile" is a completely optional file listing all packages a project depends on, it shouldn't contain anything else. It allows you to run "orb install" in the project directory to install all needed packages. Just because a project uses packages doesn't mean it self needs to be a package. The orbspec file requires more than just listing dependencies. I guess the tool could look for an .orbspec file and install all its dependencies. But what happends if it finds several .orbspec files in a directory. This is just like the Gemfile for those how have used bundler: http://gembundler.com/ "orbspec" is a specification of how a package looks like and what it contains. It's intended for creating a package out of a project. "metadata" is basically the orbspec copied into the archive. I was first thinking about "compiling" down the Ruby code into YAML or JSON but for now the Ruby code is included in the archive.

I think the "metadata" would make sence if, and *only* if, it were "compiled" down to a pure-data format. As for orbfile vs orbspec: I still think these could and should be merged. I don't need a separate buildscript for different actions. *Everything* the buildsystem does is configured in one centralized script (and whatever other files it may or may not choose to delegate out to). I see no reason why the same can't or shouldn't be done with package management. If your "orbblah" file doesn't contain the parts needed for orb to auto-create a package, then you just can't have orb auto-create your package. Simple. Also, I don't see why the name of "orbspec" needs to be prefixed with the package's name. Just like an orbfile, or makefile, or rakefile, etc., you already know what package it's for from what directory/package it's inside of.
 - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>  0.3+.4+"? Or
 something vaguely like that. That would be more flexible.

So you mean I can have a version like this: "> 0.3.4+" meaning any version from "0.3.4" to "0.3.9"? It might be a good idea.

Well, any version that's >= 0.3.4 and < 0.4.0
 - What happens if someone tries to upload a newer version with the same 
 old
 version number? Or if they forcefully do it?

I haven't thought about that. It probably shouldn't be possible.

That would make sense. But, it could always be forcefully done by doing it manually without going through orb. Don't know if that's worth worrying about though.
 - Are version numbers allowed to have more or less than three parts? I 
 think
 they should. Do version comparisons still work on version numbers with an
 arbitrary number of parts? Again, I think they should.

I was hoping to only have version numbers with three parts. If fewer parts are used it would probably easiest to infer a 0 for the missing parts, i.e. "1" == "1.0.0".

I agree with inferring 0 for any parts not provided.
 Is there a need for more parts than three? The whole idea of having three 
 version parts is to be able to use "~>  0.3.4". But if "> 0.3.4+" would be 
 allowed then arbitrary number of parts could be allowed.

I think it should be up to the project developer to choose how many parts if appropriate for their project. And I don't see any problems with doing that.
 - It should allow boolean operators and parens for the version 
 selections.
 For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>= 3.4" (Ie, "Any version
 from 2.1 through 2.6, but 2.4 has critical bugs, and 3.4+ contains a 2.x
 compatibility layer.")

Hehe. Now this is getting quite complicate, but it would be nice to have yes. Not something I will aim for in the first release.

Fair enough. :)
 - How should platforms be handled WRT packages? Ie, Do all platforms need 
 to
 be in the same orb package? Do they all need to be in separate packages?
 Either way? If they're not all required to be in the same package, how 
 does
 orb find the package that had the right platform?

I good questions. I haven't given binary packages that much thought. I was first going for a source only package manager that requires all packages to be built before installed.

I think that might be a bit limiting. And keep in mind, just because something is source-only doesn't mean it'll actually work on more than one platform.
 I see three options:

 * One package for all platforms
 * Include the platform in the package name and in the orbspec
 * Have a sub path (on the server) for every platform, i.e.:

 dorbit.org/orbs/linux/dwt-1.3.2.orb.zip

I think the first option makes the most sense, and the second option sounds like it would be a sensible workaround when someone really wants things separate. I'm not sure I like the third option because that wouldn't work for repositories that aren't explicity chosen unless you made it a standard naming system, but then that creates extra requirements for the server, and then how do you know whether to GET from the platform-specific directory or the platform-independent/cross-platform directory, etc. And what about having a windows package and a combination OSX/linux package? It's workable, but I'm not sure it's worth it in light of the first two options. Although, on second thought, if a good system for the third option could be worked out, then maybe that would be best. (But maybe just not for the initial release?)
 - Is it really necessary to have separate "build_dependencies" and
 "runtime_dependencies"? And why have both "runtime_dependencies" and 
 "orbs"
 instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. Therefore it would be unnecessary to do a permanent installation on those dependencies. The user could get an option to either permanently installed these dependencies or to temporarily install them.

So you mean a statically-linked dependency then? "Runtime dependency" makes me think it's needed at runtime and thus *would* need to be installed. Actually, what you're describing sounds just like a build dependency to me. Is there really all that much benefit to doing a temporary install of things temporarally needed anyway? And is it that common of a need? I'm wondering if it's worth the extra complexity.
 The other way around would be possible as well. A package can depend on a 
 dynamically linked library and use it only through function pointers. Then 
 the package would only have a runtime dependency on the library, i.e. it 
 wouldn't be needed when building.

Hmm I think I still don't understand the whole runtime dep vs build dep matter.
 - Would it be a good idea to have and additional field "extra" for
 non-standard expandability? So people could add extra fields they felt 
 would
 be useful, like "extra.foo" and "extra.bar", etc. And popular ones could
 eventually be formally added to the specification as just simply "foo" 
 and
 "bar".

I guess there could be an "extra" field accepting a hash. How would the field be used, by other tools?

That would be up to them. The idea is just that the package metadata is extendable for whatever uses may arise.
 - What's the point of the fields "files", "libraries" and "executables"?
 Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. "libraries" and "executables" are all libraries and executables that should be installed (regardless if they are pre-compiled or built during installation).

So it's purely for creating a package? If not, then I'm afraid I still don't understand. If so, then I still don't see the use of "libraries" and "executables". Also I think they should all be optional (if they aren't already) because all that info is already going to exist in the buildscript, and so a lot of people (like me) are going to want to generate the package via their build system rather than duplicating all that work and information just so that the package manager can do it "automatically".
 - For POSTing a package to a repository, how does authentication work? 
 All
 repos don't have to provide unrestricted upload access do they?

I haven't thought about this more than there will be some kind of authentication. Probably HTTP basic authentication.

Other things to think about are: How does the user actually specify the login info? Wouldn't want them in the orbfile/orbspec/metadata/etc.
 - If I install D library "libfoo", then I should be able write myapp.d 
 with
 "include foo.blah;" and then do "dmd myapp.d" *without* manually
 specifying -Ipath_to_libfoo. It should just work. How will Orb handle 
 that?

It can't. The solution to this is a build tool, as I see it.

Ouch. I have to say, I don't like that at all. The way I see it, this is one of the two primary responsibilities of a package manager for a language (the other being automatic dependency handling). Without it, we're not much ahead of where we are right now.
 The build tool knows about the package manager and let you specify 
 dependencies on package. Think about Drake, it could look like this:

 target("myapp.d", {
     orb("libfoo"); // automatically links with "libfoo" and includes its 
 header path.
 });

That doesn't work: If orb is managing the packages, then how is dake/drake/etc supposed to know what path to include for -I? Orb knows where it's sticking packages. The build systems don't know that. This also reminds me of another issue: For D libraries, the orbfile/orbspec/metadata/whatever needs to give the relative path for the base include directory.
 -Ipath_to_libfoo needs somehow be passed to the compiler, and linking with 
 the library as well. Maybe it would be possible to manipulate the 
 dmd.conf/sc.ini but this seems very complicated.

I think what we need is to have DVM handle dmd.conf/sc.ini separately from the compiler version (which kind of goes along with DVM Issue #9). Orb and DVM need to work together to manage dmd.conf/sc.ini. It's complicated, but I think it's necessary. This makes me think of another, too: How does the buildscript know what version of a lib orb has chosen? Since orb packages (rightfully) are allowed to specify a range of versions instead of just one specific version, I think that will be needed.
 And how will that interact with DVM? Ie, if I do "dvm use 2.051", then 
 "orb
 install libfoo", then "dvm use 2.054", then I should still have access to
 libfoo without needing to specify -Ipath_to_libfoo.

If you do "dvm use 2.051", then "orb install libfoo" then "libfoo" will only be installed for dmd 2.051, that's the whole point. You would have to run "orb install libfoo" again after switching compiler.

I think that's very, very bad. I should be able to compile something with a different version of DMD without reinstalling everything. "dvm use xxx" should *just work*, just as it already does now.
 - In many ways this sounds a lot like a generalized DVM. Maybe Orb should
 eventually take over DVM's duties by making a DMD orb package.

No, I don't think so. DVM is quite specialized in what it does. Manipulating the PATH variable (or the registy on Windows) to be able to do what it does. I don't what to mix DVM and Orbit.

Orb manages different versions of arbitrary packages. DVM manages different versions of one specific package. If they're so different despite all that, then either Orbit isn't generalized enough, or DVM is too specialized.
Jul 17 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-18 00:26, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 But why should the name of the project and the name of the tool be
 different? It would be less confusing to have a packager manager named Orb
 that gets invoked via "orb". (Or a package manager named Orbit that gets
 invoked via "orbit".

I don't know. It's also easy to separate the library code in one package named "orbit" and the application code in on package named "orb".
 I think the "metadata" would make sence if, and *only* if, it were
 "compiled" down to a pure-data format.

Ok, fair enough. That was the initial idea, to have the "metadata" file in a pure-data format.
 As for orbfile vs orbspec: I still think these could and should be merged. I
 don't need a separate buildscript for different actions. *Everything* the
 buildsystem does is configured in one centralized script (and whatever other
 files it may or may not choose to delegate out to). I see no reason why the
 same can't or shouldn't be done with package management.

I guess I can do that.
 If your "orbblah" file doesn't contain the parts needed for orb to
 auto-create a package, then you just can't have orb auto-create your
 package. Simple.

Ok.
 Also, I don't see why the name of "orbspec" needs to be prefixed with the
 package's name. Just like an orbfile, or makefile, or rakefile, etc., you
 already know what package it's for from what directory/package it's inside
 of.

The package needs a name. One option is to specify that in the orbspec file. Another option is to specify it in the name of the orbspec file. I just thought that it was convenient. Is it a good idea to take the name directory where the orbspec is located? Note that many of my choices are based on RubyGems.
 Well, any version that's>= 0.3.4 and<  0.4.0

Ok.
 That would make sense. But, it could always be forcefully done by doing it
 manually without going through orb. Don't know if that's worth worrying
 about though.

Do you mean if you have direct access to the server where the repository is located? I don't think that's something worth worrying about. Isn't that the same with all servers you don't own yourself? If you have a website on a hosted server they can, if they want to, just remove it.
 I agree with inferring 0 for any parts not provided.

Good.
 I think it should be up to the project developer to choose how many parts if
 appropriate for their project. And I don't see any problems with doing that.

Ok. But then I think we most allow: "> 0.3.4+". Or else you cannot know if a given version will break the API.
 I think that might be a bit limiting. And keep in mind, just because
 something is source-only doesn't mean it'll actually work on more than one
 platform.

As you can read in the specification I do intend to have binary packages. But as I said, that was not my original thought and that's why I haven't given it much thought. Of course it won't work on all platform just because it's source only but if there are source only packages then the problem below doesn't exist.
 I see three options:

 * One package for all platforms
 * Include the platform in the package name and in the orbspec
 * Have a sub path (on the server) for every platform, i.e.:

 dorbit.org/orbs/linux/dwt-1.3.2.orb.zip

I think the first option makes the most sense, and the second option sounds like it would be a sensible workaround when someone really wants things separate.

There's always this issue with downloading files you don't need.
 I'm not sure I like the third option because that wouldn't work for
 repositories that aren't explicity chosen unless you made it a standard
 naming system, but then that creates extra requirements for the server, and
 then how do you know whether to GET from the platform-specific directory or
 the platform-independent/cross-platform directory, etc. And what about
 having a windows package and a combination OSX/linux package? It's workable,
 but I'm not sure it's worth it in light of the first two options.

I don't understand what you mean. Why would you have one package for Windows and one for osx/linux? Or have I misunderstood something.
 Although, on second thought, if a good system for the third option could be
 worked out, then maybe that would be best. (But maybe just not for the
 initial release?)

An other issue with one package for all platforms is when building the package. The developer first needs to build the package on one platform, then move the package to all other platforms and rebuild the package. The tool needs to open the package and add files for the other platforms.
 - Is it really necessary to have separate "build_dependencies" and
 "runtime_dependencies"? And why have both "runtime_dependencies" and
 "orbs"
 instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. Therefore it would be unnecessary to do a permanent installation on those dependencies. The user could get an option to either permanently installed these dependencies or to temporarily install them.

So you mean a statically-linked dependency then? "Runtime dependency" makes me think it's needed at runtime and thus *would* need to be installed. Actually, what you're describing sounds just like a build dependency to me.

Statically-linked dependency is basically the same as build dependency. If you have a look at the DVM project page you can see the list of build dependencies. The only thing you need to actually run the application is zlib (on Poisx), because it's dynamically linked, aka a runtime dependency.
 Is there really all that much benefit to doing a temporary install of things
 temporarally needed anyway? And is it that common of a need? I'm wondering
 if it's worth the extra complexity.

The package would need access to the dependencies when its build. How should it else be done?
 The other way around would be possible as well. A package can depend on a
 dynamically linked library and use it only through function pointers. Then
 the package would only have a runtime dependency on the library, i.e. it
 wouldn't be needed when building.

Hmm I think I still don't understand the whole runtime dep vs build dep matter.

If you use, on Posix, dlopen and friends to open a dynamic library you don't need to link to it when building. You call the functions via function pointers, that's how Derelict works, if you have used it.
 That would be up to them. The idea is just that the package metadata is
 extendable for whatever uses may arise.

Ok, that won't be any problem including.
 - What's the point of the fields "files", "libraries" and "executables"?
 Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. "libraries" and "executables" are all libraries and executables that should be installed (regardless if they are pre-compiled or built during installation).

So it's purely for creating a package?

Yes, the "files" field is just when creating the package.
 If not, then I'm afraid I still don't understand.

 If so, then I still don't see the use of "libraries" and "executables". Also
 I think they should all be optional (if they aren't already) because all
 that info is already going to exist in the buildscript, and so a lot of
 people (like me) are going to want to generate the package via their build
 system rather than duplicating all that work and information just so that
 the package manager can do it "automatically".

Both the "libraries" and "executables" fields are optional. "summary" and "version" are the only required fields (so far), as you can see on the wiki. As I said before, "libraries" and "executables" are what actually should be installed. This is what I'm thinking and is planning for Orbit and my build tool Dake. Dake will be able to generate an orbspec file, because, as you said, the build tool will have all the necessary information. Then you don't have to duplicate any information. But, I'm not forcing anyone to use my build tool if they want to use Orbit, therefore the fields "libraries" and "executables" are available.
 Other things to think about are: How does the user actually specify the
 login info? Wouldn't want them in the orbfile/orbspec/metadata/etc.

No, absolutely not. Passing user name and password using HTTP basic authentication on the command line?
 Ouch. I have to say, I don't like that at all. The way I see it, this is one
 of the two primary responsibilities of a package manager for a language (the
 other being automatic dependency handling). Without it, we're not much ahead
 of where we are right now.

If you have suggestions I'm listening.
 That doesn't work: If orb is managing the packages, then how is
 dake/drake/etc supposed to know what path to include for -I? Orb knows where
 it's sticking packages. The build systems don't know that.

The build tool invokes Orbit and asks about the include path.
 This also reminds me of another issue: For D libraries, the
 orbfile/orbspec/metadata/whatever needs to give the relative path for the
 base include directory.

Something like that. Or if given full path the tool will assume the current directory is the base directory and remove that from the path. For example: $ cd ~/projects/d/foo Contains: main.d bar/foobar.d # foo.orbspec files << ["~/projects/d/foo/main.d", "~/projects/d/foo/bar/foobar.d"] $ orb build foo Since I run the "orb build" command in the "~/projects/d/foo" directory the tool will just remove that from the paths when including the files in the package.
 I think what we need is to have DVM handle dmd.conf/sc.ini separately from
 the compiler version (which kind of goes along with DVM Issue #9). Orb and
 DVM need to work together to manage dmd.conf/sc.ini. It's complicated, but I
 think it's necessary.

I was hoping I didn't need to drag in DVM in this. And I don't won't to force anyone using DVM if they want to use Orbit.
 This makes me think of another, too: How does the buildscript know what
 version of a lib orb has chosen? Since orb packages (rightfully) are allowed
 to specify a range of versions instead of just one specific version, I think
 that will be needed.

It doesn't need to know what exact version is chosen, it just needs to know the include paths and libraries to link with. The rest is up to Orbit to handle and decide. The build tool just links with what it received from Orbit.
 I think that's very, very bad. I should be able to compile something with a
 different version of DMD without reinstalling everything. "dvm use xxx"
 should *just work*, just as it already does now.

The whole point is to be able to have different packages installed with different compilers. There could be a command for moving over packages from one DMD installation to another. Say I'm using DMD 2.053, then installing package "foo". Then I'm installing DMD 2.054 and switching to it. Say also that package "foo" doesn't work with DMD 2.054, what happens when you switch to 2.054?
 Orb manages different versions of arbitrary packages. DVM manages different
 versions of one specific package. If they're so different despite all that,
 then either Orbit isn't generalized enough, or DVM is too specialized.

DVM is very specialized, yes. The installation and switching of D compiler, using DVM, is very specialized. DMD requires wrappers, manipulating the PATH or registry. None of the Orbit packages require this. Orbit would require loads of special cases to have DVM built-in. On your description DVM and Orbit sound very similar but how they actually work when installing and switch compilers are very different. You should know this, you ported DVM to Windows. The current compiler decides what packages are available/installed, that wouldn't work if the compiler package would just be a regular orb package. What would be possible it to create a tool that is a front end for DVM and Orbit. I'm currently building Orbit as a library with a thin wrapper that is the tool the user will use. I'm planning to rewrite DVM in the same way. This also allows to create GUIs and integrate the tools into IDEs. -- /Jacob Carlborg
Jul 18 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-18 18:03, Johannes Pfau wrote:
 It would be possible to install libraries into the dmd default search
 path. Right now, this would be /usr/include/d/dmd and library files
 in /usr/lib on posix, but any path can be used as long as it's included
 in dmd.conf. However, this means that only one version of a library can
 be installed system-wide, so it's not optimal.

 It should also be noted, that linking against a specific library
 version can only work well with static libraries. With static libraries
 you can give linker a specific path to the library at compile time
 (So you can have multiple versions in different directories). Using
 different directories with dynamic libraries requires setting
 LD_LIBRARY_PATH before executing a program, so this won't work. We'll
 have to use the library versioning mechanism that's used by C
 libraries (e.g for linux:
 http://www.ibm.com/developerworks/linux/library/l-shlibs/index.html ).
 I don't know if windows even supports library versioning, but as
 windows programs usually don't install dlls globally that's less of a
 problem.

You can give a specific path to the compiler with dynamic libraries as well. Link with libfoo, version 3.4.0: dmd -L-L/path/to/libfoo-3.4.0 -L-lfoo Or have I missed something? -- /Jacob Carlborg
Jul 18 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-19 10:16, Johannes Pfau wrote:
 I'd have to test that, but I doubt it will work. This will help to find
 the library at compile time, but not at runtime. The runtime linker
 will only search in directories listed in /etc/ld.so.conf.d or in the
 LD_LIBRARY_PATH variable. Each .so library has a 'soname' embedded. If
 you link like in your example command the resulting binary only
 contains the sonames of the libraries it needs, not the full path. At
 runtime, the linker then reads that soname and searches in its cache for
 a library with the same soname. It might be possible to make this
 soname mechanism use absolute paths or subdirectories, but this seems
 like a hack. Sonames are usually just "libfoo.so.3" where 3 is a ABI
 revision. I think we'll eventually have to install shared libraries
 exactly the way C does it, i.e all in /usr/lib and using the soname
 versioning. But we can think about that when dmd finally supports
 shared libraries on linux, it's not important right now.

Oh, at runtime, didn't think of that :). The above command is only for compile time. What about the linker flag "-rpath"? That seems it could be used. Linux is not the only OS, it's easy to add support for dynamic libraries on Mac OS X. All code is already in druntime, it just needs to be enabled. Tango already supports this. -- /Jacob Carlborg
Jul 19 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

Won't the same problem occur if rpath isn't used? With LD_LIBRARY_PATH for example.
 I'd prefer installing shared libraries system wide though. The
 soname/version approach is not that bad. Your proposed package
 versioning scheme could even be mapped 1:1 to the soname versions. Or
 we could use libtools versioning scheme, which is similar, ('major' and
 'minor' are one field, 'build' stays the same, and an additional 'age'
 field is added)
 http://sourceware.org/autobook/autobook/autobook_91.html

I don't want to install the libraries system wide. Again your assuming Linux only. It has to work on all supported platforms. At least: Linux, Mac OS X and Windows.
 Having read more about it, i think I have to correct my previous
 statement: It is possible to link to specific versions with the soname
 approach. It's maybe a little more limited (You can't say: "I want to
 use libfoo.so.1.2.0", You can only say: "I want to use libfoo 1.x.x",
 and the linker could end up using 1.1.0, 1.2.0 ...) but it seems this
 should be good enough.

No, I want to be able to use an exact version. -- /Jacob Carlborg
Jul 19 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-19 13:43, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

Won't the same problem occur if rpath isn't used? With LD_LIBRARY_PATH for example.

I'm not sure about that, but using LD_LIBRARY_PATH is even more discouraged (because it propagates to child processes)

Ok. What is used to tell the system where to find a library? Only the default paths as /usr/lib ?
 I'm not sure if we can have one perfect approach for all operating
 systems. But we could just use the rpath way and see how it works out.

Yeah. We'll see how it turns out.
 I agree that's a nice-to-have feature, but is it ever really necessary?
 This mechanism always picks up the newest ABI compatible library
 version. Why would you want to specify one version explicitly if the
 newer version can be used as a 1:1 replacement?

I don't know if it's really necessary but that's how I've been thinking that I want it to behave. -- /Jacob Carlborg
Jul 19 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-22 12:38, Johannes Pfau wrote:
 The rpath page says this:
 1. the RPATH binary header (set at build-time) of the library causing
     the lookup (if any)

 2. the RPATH binary header (set at build-time) of
     the executable

 3. the LD_LIBRARY_PATH environment variable (set at
     run-time)

 4. the RUNPATH binary header (set at build-time) of the
     executable

 5. /etc/ld.so.cache (generated from /etc/ld.so.conf
     and /etc/ld.so.conf.d)

 6. base library directories (/lib and /usr/lib)

 But that depends on the C library / linker implementation.

Since RPATH and LD_LIBRARY_PATH is discouraged only 4, 5 and 6 are left? -- /Jacob Carlborg
Jul 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-07-22 14:53, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-22 12:38, Johannes Pfau wrote:
 The rpath page says this:
 1. the RPATH binary header (set at build-time) of the library causing
      the lookup (if any)

 2. the RPATH binary header (set at build-time) of
      the executable

 3. the LD_LIBRARY_PATH environment variable (set at
      run-time)

 4. the RUNPATH binary header (set at build-time) of the
      executable

 5. /etc/ld.so.cache (generated from /etc/ld.so.conf
      and /etc/ld.so.conf.d)

 6. base library directories (/lib and /usr/lib)

 But that depends on the C library / linker implementation.

Since RPATH and LD_LIBRARY_PATH is discouraged only 4, 5 and 6 are left?

As far as I know, yes. However, note that 5 and 6 are 'global' options, they affect all executables and (dynamic) libraries.

So far I've understood that :) -- /Jacob Carlborg
Jul 22 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

The problem mentioned on that page is: A problem arises when binary A defines a NEEDED dependency on libraries B.so.1 and C.so.2, while library B.so.1 depends on library C.so.1 How is this handled when rpath isn't used and in general? -- /Jacob Carlborg
Jul 19 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-07-22 12:38, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

The problem mentioned on that page is: A problem arises when binary A defines a NEEDED dependency on libraries B.so.1 and C.so.2, while library B.so.1 depends on library C.so.1 How is this handled when rpath isn't used and in general?

To be honest, I don't know. I'm not even sure if I understand the issue with rpath at all, but I thought I'd better mention it.

I can see that there could be a problem but I don't see how this can be a problem only for rpath. Seems to me that it would be a problem regardless which paths are used. So what happens if you link two libraries which are the same library but of different versions? Conflicting symbols? -- /Jacob Carlborg
Jul 22 2011
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:j01dlr$nm$1 digitalmars.com...
 On 2011-07-18 00:26, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 But why should the name of the project and the name of the tool be
 different? It would be less confusing to have a packager manager named 
 Orb
 that gets invoked via "orb". (Or a package manager named Orbit that gets
 invoked via "orbit".

I don't know. It's also easy to separate the library code in one package named "orbit" and the application code in on package named "orb".

What about "orb"/"orblib", or "orbit"/"liborbit", etc? Be easier to tell which is which that way anyway.
 Also, I don't see why the name of "orbspec" needs to be prefixed with the
 package's name. Just like an orbfile, or makefile, or rakefile, etc., you
 already know what package it's for from what directory/package it's 
 inside
 of.

The package needs a name. One option is to specify that in the orbspec file. Another option is to specify it in the name of the orbspec file. I just thought that it was convenient.

Ok, I see. That makes sense.
 Is it a good idea to take the name directory where the orbspec is located?

No, I don't think it is. When orb downloads/installs a package it should definitely put it in a directory named the same as the package. But I often have reason to projects in a directory with a (slightly) different name from the project, so I don't think they should be expected to be the same names a packages the user is developing. So I'm fine with the orbspec filename being prepended with the package name...Only one minor potential issue I see: What happens if a single project has both "foo.orbspec" and "bar.orbspec"?
 Note that many of my choices are based on RubyGems.

I guess I should look at that :P
 That would make sense. But, it could always be forcefully done by doing 
 it
 manually without going through orb. Don't know if that's worth worrying
 about though.

Do you mean if you have direct access to the server where the repository is located?

Right.
 I don't think that's something worth worrying about.

Fair enough. I guess just as long as it's understood that that shouldn't be done because if it is done, other users won't get the new version "a.b.c" unless they uninstall and reinstall it.
 I think it should be up to the project developer to choose how many parts 
 if
 appropriate for their project. And I don't see any problems with doing 
 that.

Ok. But then I think we most allow: "> 0.3.4+".

Sorry, can you be more specific on what you mean by that?
 I see three options:

 * One package for all platforms
 * Include the platform in the package name and in the orbspec
 * Have a sub path (on the server) for every platform, i.e.:

 dorbit.org/orbs/linux/dwt-1.3.2.orb.zip

I think the first option makes the most sense, and the second option sounds like it would be a sensible workaround when someone really wants things separate.

There's always this issue with downloading files you don't need.

True, so I guess there should be a way to do platform-specific packages.
 I'm not sure I like the third option because that wouldn't work for
 repositories that aren't explicity chosen unless you made it a standard
 naming system, but then that creates extra requirements for the server, 
 and
 then how do you know whether to GET from the platform-specific directory 
 or
 the platform-independent/cross-platform directory, etc. And what about
 having a windows package and a combination OSX/linux package? It's 
 workable,
 but I'm not sure it's worth it in light of the first two options.

I don't understand what you mean.

Suppose there's a (possibly default) repository: http://www.super-d-repos.com/joesmith/ And the user runs: $ orb install foo 1.2.3 What url does orb try to retreive?: http://www.super-d-repos.com/joesmith/foo-1.2.3.zip http://www.super-d-repos.com/joesmith/linux/foo-1.2.3.zip http://www.super-d-repos.com/joesmith/multiplatform/foo-1.2.3.zip Etc... We'd need to have some well-designed standard for how that works. I'm not sure we should require the repo actually be specified as: http://www.super-d-repos.com/joesmith/linux/ Because that would be platform-dependent and require extra platform-handling code for the orbspec author. And it would be different for different repos. Come to think of it, the same issue applies to making the platform part of the package name. So maybe a good path-based system would be better. This makes me think of another question: Once different comression formats are allowed, if someone uses orb to download/install a package, how does orb know whether to grab a .zip, or a .bz2, or a .7z, etc?
 Why would you have one package for Windows and one for osx/linux? Or have 
 I misunderstood something.

If, for example, the author wants the windows package to be separate because it (hypothetically) has a lot of differences from posix, but the different posix versions are very similar so may as well be in the same package.
 Although, on second thought, if a good system for the third option could 
 be
 worked out, then maybe that would be best. (But maybe just not for the
 initial release?)

An other issue with one package for all platforms is when building the package. The developer first needs to build the package on one platform, then move the package to all other platforms and rebuild the package. The tool needs to open the package and add files for the other platforms.

Good point. Although, of course, that only applies to binary packages. So yea, it seems that both multi-platform and platform-specific packages should be supported. And there needs to be some system for dealing with that.
 So it's purely for creating a package?

Yes, the "files" field is just when creating the package.

Ok.
 As I said before, "libraries" and "executables" are what actually should 
 be installed.

This is another thing I'm unclear on, the nature of "installing". It almost sounds like you're saying that all the files in the package other than "libraries" and "executables" are just not used for anything and just thrown away. I doubt that's it though. Is this how you're envisioning it?: 1. User says "orb install foo". 2. Orb downloads and extracts the foo package to a temp directory. 3. Orb invokes the build process to build foo (BTW, how is that "how to build" commandline string specified in the orbspec? The closest thing I see is the "build" field, but it looks like that's just the name of the tool used. An actual command line string is going to be needed.) 4. Orb copies the files listed in "libraries" and "executables" from the temp dir to their permanent location. 5. Temp dir is cleared. If so, how do you account for non-compiled source-based libraries? Include the *.d files in "libraries"?
 This is what I'm thinking and is planning for Orbit and my build tool 
 Dake. Dake will be able to generate an orbspec file, because, as you said, 
 the build tool will have all the necessary information. Then you don't 
 have to duplicate any information.

Hmm, I guess "files" isn't so bad.
 Ouch. I have to say, I don't like that at all. The way I see it, this is 
 one
 of the two primary responsibilities of a package manager for a language 
 (the
 other being automatic dependency handling). Without it, we're not much 
 ahead
 of where we are right now.

If you have suggestions I'm listening.

Lemme think about it a bit and get back to you.
 That doesn't work: If orb is managing the packages, then how is
 dake/drake/etc supposed to know what path to include for -I? Orb knows 
 where
 it's sticking packages. The build systems don't know that.

The build tool invokes Orbit and asks about the include path.

I think it'd be nicer if invoking a process wasn't needed for that. (Although it could just be done in the "configure" step, but it'd be nicer if orb could just handle it without the buildscript needing to worry about it.)
 This also reminds me of another issue: For D libraries, the
 orbfile/orbspec/metadata/whatever needs to give the relative path for the
 base include directory.

Something like that. Or if given full path the tool will assume the current directory is the base directory and remove that from the path. For example: $ cd ~/projects/d/foo Contains: main.d bar/foobar.d # foo.orbspec files << ["~/projects/d/foo/main.d", "~/projects/d/foo/bar/foobar.d"] $ orb build foo Since I run the "orb build" command in the "~/projects/d/foo" directory the tool will just remove that from the paths when including the files in the package.

What about projects that don't use "{project_root}" as the include directory? My projects, for instance, typically use "{project_root}/src".
 I think that's very, very bad. I should be able to compile something with 
 a
 different version of DMD without reinstalling everything. "dvm use xxx"
 should *just work*, just as it already does now.

The whole point is to be able to have different packages installed with different compilers. There could be a command for moving over packages from one DMD installation to another.

Seems unnecessary.
 Say I'm using DMD 2.053, then installing package "foo". Then I'm 
 installing DMD 2.054 and switching to it. Say also that package "foo" 
 doesn't work with DMD 2.054, what happens when you switch to 2.054?

The same thing as if you manually did: $ dvm use 2.054 $ rdmd -I~/proj/foo-1.2.3/includes myApp.d It just won't work.
 Orb manages different versions of arbitrary packages. DVM manages 
 different
 versions of one specific package. If they're so different despite all 
 that,
 then either Orbit isn't generalized enough, or DVM is too specialized.

DVM is very specialized, yes. The installation and switching of D compiler, using DVM, is very specialized. DMD requires wrappers, manipulating the PATH or registry. None of the Orbit packages require this.

What about "orb use" on binary packages?
 Orbit would require loads of special cases to have DVM built-in. On your 
 description DVM and Orbit sound very similar but how they actually work 
 when installing and switch compilers are very different. You should know 
 this, you ported DVM to Windows.

What I'm saying is that maybe they shouldn't be so different.
 The current compiler decides what packages are available/installed, that 
 wouldn't work if the compiler package would just be a regular orb package.

I don't believe that the current compiler *should* decide what packages are available/installed. Maybe there could be some optional magic to warn when using a compiler/package combination that isn't known to work. But aside from that, I really think these should be orthogonal.
Jul 18 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-07-19 03:58, Nick Sabalausky wrote:
 What about "orb"/"orblib", or "orbit"/"liborbit", etc? Be easier to tell
 which is which that way anyway.

Yeah, that would be one option. You can also see like this: "Hey, I have a compiler, it's named 'GNU C Compiler', it's invoked with the 'gcc' command". What's the confusion with that?
 Ok, I see. That makes sense.

Ok, good.
 Is it a good idea to take the name directory where the orbspec is located?

No, I don't think it is. When orb downloads/installs a package it should definitely put it in a directory named the same as the package. But I often have reason to projects in a directory with a (slightly) different name from the project, so I don't think they should be expected to be the same names a packages the user is developing. So I'm fine with the orbspec filename being prepended with the package name...Only one minor potential issue I see: What happens if a single project has both "foo.orbspec" and "bar.orbspec"?

When building the package? You just run "orb build foo" or "or build bar". Were you hoping for only "orb build"? In that case I guess it would be possible if there's only one .orbspec in the current directory. That know if that's confusing.
 Sorry, can you be more specific on what you mean by that?

The syntax you suggested. Instead of having "~> 0.3.4" then you could have "~> 0.3.4+" or "~> 0.3+.4+".
 True, so I guess there should be a way to do platform-specific packages.

Don't know how much of an issue this is in reality. Although there could be fairly large libraries and applications. I know DWT is quite large.
 Suppose there's a (possibly default) repository:

 http://www.super-d-repos.com/joesmith/

 And the user runs:

 $ orb install foo 1.2.3

 What url does orb try to retreive?:

 http://www.super-d-repos.com/joesmith/foo-1.2.3.zip
 http://www.super-d-repos.com/joesmith/linux/foo-1.2.3.zip
 http://www.super-d-repos.com/joesmith/multiplatform/foo-1.2.3.zip

I guess that just needs to be standardized.
 Etc...

 We'd need to have some well-designed standard for how that works.

 I'm not sure we should require the repo actually be specified as:

 http://www.super-d-repos.com/joesmith/linux/

 Because that would be platform-dependent and require extra platform-handling
 code for the orbspec author. And it would be different for different repos.
 Come to think of it, the same issue applies to making the platform part of
 the package name. So maybe a good path-based system would be better.

How would it require extra platform-handling for the orbspec author? All repositories behave in the same way, if not, it's not an orb repository.
 This makes me think of another question: Once different comression formats
 are allowed, if someone uses orb to download/install a package, how does orb
 know whether to grab a .zip, or a .bz2, or a .7z, etc?

I guees it's same as with the platform. Just standardize on a specific order. i.e. 7z, then bz2 and last zip.
 If, for example, the author wants the windows package to be separate because
 it (hypothetically) has a lot of differences from posix, but the different
 posix versions are very similar so may as well be in the same package.

That is for a source package?
 Good point. Although, of course, that only applies to binary packages. So
 yea, it seems that both multi-platform and platform-specific packages should
 be supported. And there needs to be some system for dealing with that.

There is way more to think about than one actually want to recognize :)
 This is another thing I'm unclear on, the nature of "installing". It almost
 sounds like you're saying that all the files in the package other than
 "libraries" and "executables" are just not used for anything and just thrown
 away. I doubt that's it though.

If you but "main.d" and "main.exe" in the "files" field. And you have "main.exe" in the "executables" field it, will only install "main.exe". There is no point in installing "main.d" as well, you actually don't need to put in the package in the first place. But there could be an option to install the source code as well, if present.
 Is this how you're envisioning it?:

 1. User says "orb install foo".

 2. Orb downloads and extracts the foo package to a temp directory.

 3. Orb invokes the build process to build foo (BTW, how is that "how to
 build" commandline string  specified in the orbspec? The closest thing I see
 is the "build" field, but it looks like that's just the name of the tool
 used. An actual command line string is going to be needed.)

 4. Orb copies the files listed in "libraries" and "executables" from the
 temp dir to their permanent location.

 5. Temp dir is cleared.

Something like that.
 If so, how do you account for non-compiled source-based libraries? Include
 the *.d files in "libraries"?

That's a good question. Maybe the "type" field needs to include "source".
 Hmm, I guess "files" isn't so bad.

Oh, and Dake does this with the help of Orbit. It's acutally Orbit that generates the orbspec but Dake passes all information about the orbspec to Orbit.
 I think it'd be nicer if invoking a process wasn't needed for that.
 (Although it could just be done in the "configure" step, but it'd be nicer
 if orb could just handle it without the buildscript needing to worry about
 it.)

It doesn't technically needs to create a process and invoke the orb command. Dake can just like with liborbit and take the information it needs. Every tool should be built as a library and a thin wrapper on top of that for the executable. Makes everything easier :).
 What about projects that don't use "{project_root}" as the include
 directory? My projects, for instance, typically use "{project_root}/src".

Hmm, that's a good question. Wonder how other tools handle this.
 The same thing as if you manually did:

 $ dvm use 2.054
 $ rdmd -I~/proj/foo-1.2.3/includes myApp.d

 It just won't work.

Once you have installed the packages for the new compiler you can use dvm as before.
 What about "orb use" on binary packages?

I don't follow.
 What I'm saying is that maybe they shouldn't be so different.

As far as I know dvm HAS to work as it does. And I'm certainly don't arbitrary packages manipulating the PATH, registry and the shell start up file.
 I don't believe that the current compiler *should* decide what packages are
 available/installed. Maybe there could be some optional magic to warn when
 using a compiler/package combination that isn't known to work. But aside
 from that, I really think these should be orthogonal.

Ok, one option would be to have all packages installed globally in DVM, i.e. available for all compilers. DVM could let you create orb sets and switch between them. Then you could have one orb set for each compiler if you wanted. Example: $ dvm use 2.053 $ orb list Installed packages: foo bar $ dvm install 2.054 $ dvm use 2.054 $ orb list Installed packages: foo bar $ dvm orbset create test $ dvm orbset use test $ orb list No installed packages Now, when installing a new package it will only be installed to the "test" orb set. How those this sound? -- /Jacob Carlborg
Jul 19 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-07-19 03:58, Nick Sabalausky wrote:
 3. Orb invokes the build process to build foo (BTW, how is that "how to
 build" commandline string  specified in the orbspec? The closest thing I see
 is the "build" field, but it looks like that's just the name of the tool
 used. An actual command line string is going to be needed.)

Yes, it's the name of the build tool to invoke. By default it has support for a couple of build tools and knows how to invoke them. Example: To invoke Make it would just be "make". To invoke DSSS it would be "dsss build". In addition to that the "build" field accepts a variable argument list with arguments to the build tool. Example: build :make, "-f foo.mak" Would invoke "make -f foo.mak". One of the known build tools is "shell" which basically lets you run arbitrary commands. Example: build :shell, "./build_my_app.sh" build :shell, "rdmd --build-only main.d".
 What about projects that don't use "{project_root}" as the include
 directory? My projects, for instance, typically use "{project_root}/src".

Actually I don't think this will be a problem. The working directory when installing a package will look the same as the project directory where you created the package. Example: 1. You have all source files in "{project_root}/src". When you build the package, from "{project_root}", it will create a "src" directory in the package and all you source files will be in this directory. 2. When the package is to be installed the package manager extracts the package and you will have the "src" folder in the working directory/a temp directory. 3. The package manager will execute the build command from the working directory. So the build system you have chosen just needs to find the "src" folder relative to the working directory. Lets take Make as an example, because it's a tool that exists and we know how it works. Either you have the makefile in the working directory (the original place was the project root). Or it will be in the "src" directory. This is how the orbspec will look like when the makefile is in the root folder: build :make Or when the makefile is in the "src" folder: build :make, "-f src/makefile" -- /Jacob Carlborg
Jul 19 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-14 11:07, Johannes Pfau wrote:
 jdrewsen wrote:
 Den 13-07-2011 21:19, Jacob Carlborg skrev:
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language
 for all the files mentioned in the link above.

 The current status is that building packages and installing them
 works, but quite limited. No dependency tracking or central
 repository so far.

 Please comment and suggest.

Nice work! Orb - tool section: Describe what the "use" command does. I guess it simply adds a required orb to the Orbfile in the current directory? Orbfile section: The "orb" command that accepts git/hg/svn repositories should also allow for a tag/commit parameter I think. The "orb" commands second parameter could also be a list of serveral repositories to try in order for fallback. I guess a user configuration file in ~/.orb could contain "source" commands as well. Orb package section: I think the versioning scheme should be set in stone actually. Most other packaging systems does that. It makes your life much easier. Central repository section: Please let us settle for one format for the metadata.xxx file. My vote is for json or yaml. XML is too verbose for my taste. I also think that it should be compressed e.g. metadata.json.bzip since it will quickly grow quite large and the packaging system has to be fast. Maybe add the build revision on /orb/<package>-<version>_<build> since it is quite common reupload the same package version with at simple build fix. Additionally the architecture should be added to the name: /orb/<package>-<version>_<build>-<arch> Now it is just like how debian files look like :) Maybe put the file in an arch subdir /orb/<arch>/<package>-<version>_<build> I'm really in favor of doing this in D instead of ruby though. /Jonas

And in Package SubTypes: No documentation subtype for now?

I though they were quite self explanatory. But I can add documentation for them if necessary.

Sorry, my question wasn't clear: I meant where will 'api documentation' for libraries go? In a special 'documentation' subtype or in the library packages?
 Do Library and Dynamic library always contain the .d/.di headers?

Library (which is static library) needs to contain headers yes. Dynamic library will probably contain headers as well.
 In orbspec:
 The callbacks are run when the package is being build, correct? (The
 install callback could also be called when the package is being
 installed, but I think we don't need this functionality)

I see now that I have to give this some thought. The "build" callback could be called both when all files are built into the package and when the package later is built to be installed.
 So those are the hooks to use any custom build system?

Yes and no. There is also the "build" field which allow you yo call a custom shell script or custom build tool. Something like this: build :shell, "my_custom_build_script.sh"
 Are Check Platform and Set Platform really needed in the orbspec? I
 think this can be left to drake or other build systems, can't we keep
 orbspecs platform agnostic?

Absolutely, the orbspecs are as platform agnostic as possible. But I see no harm in having the possibility to check the platform. I'm pretty sure at least someone will find a need for it.
 I'd make build_dependencies and dependencies a required field.
 (Although empty is ok. Can we distinguish between not set and empty?)

I see no point in setting the build_dependencies and dependencies fields if the package doesn't have any dependencies. nil/null would be not set and an empty array (in this case) would be empty.

Well, having to set it explicitly makes it less likely to forget those fields, but that's the only reason. -- Johannes Pfau
Jul 15 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-15 09:14, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ivkrdj$ci4$1 digitalmars.com...
 I've written a more formal specification of my ideas for a package
 manager for D.

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 Note that I am exploring the possibility of using D as the language
 for all the files mentioned in the link above.

 The current status is that building packages and installing them
 works, but quite limited. No dependency tracking or central
 repository so far.

 Please comment and suggest.

Good start :) Here are my random thoughts on it (sorry if you've already answered some of them, I haven't read the rest of this thread yet): - I find the "orb" vs "orbit" distinction a little confusing and unnecessary. Why not just call it all "orb"? (Or call everything "orbit"?)

This is how I'm thinking: Orbit is the name of the package manager, "orb" is what you type on the command line when interacting with the package manager. A package is also called an "orb".
 - I think the files "orbfile", "{name}.orbspec" and "metadata" (the
 one inside the packages) should all just be one file. Frankly, the
 existence of all three of them confuses the hell out of me.

Ok, this is how it works: "orbfile" is a completely optional file listing all packages a project depends on, it shouldn't contain anything else. It allows you to run "orb install" in the project directory to install all needed packages. Just because a project uses packages doesn't mean it self needs to be a package. The orbspec file requires more than just listing dependencies. I guess the tool could look for an .orbspec file and install all its dependencies. But what happends if it finds several .orbspec files in a directory. This is just like the Gemfile for those how have used bundler: http://gembundler.com/ "orbspec" is a specification of how a package looks like and what it contains. It's intended for creating a package out of a project. "metadata" is basically the orbspec copied into the archive. I was first thinking about "compiling" down the Ruby code into YAML or JSON but for now the Ruby code is included in the archive.
 - Are the subtypes a "pick one" or "pick all that are included"
 deal? I think that latter would make more sense.

I'm thinking this can be inferred from other fields like "executables" and "libraries". A package could contain several types.
 - Instead of "~>  0.3.4", what about ">  0.3+.4+"? Or ""~>
 0.3+.4+"? Or something vaguely like that. That would be more
 flexible.

So you mean I can have a version like this: "> 0.3.4+" meaning any version from "0.3.4" to "0.3.9"? It might be a good idea.
 - What happens if someone tries to upload a newer version with the
 same old version number? Or if they forcefully do it?

I haven't thought about that. It probably shouldn't be possible.
 - Are version numbers allowed to have more or less than three parts?
 I think they should. Do version comparisons still work on version
 numbers with an arbitrary number of parts? Again, I think they
 should.

I was hoping to only have version numbers with three parts. If fewer parts are used it would probably easiest to infer a 0 for the missing parts, i.e. "1" == "1.0.0". Is there a need for more parts than three? The whole idea of having three version parts is to be able to use "~> 0.3.4". But if "> 0.3.4+" would be allowed then arbitrary number of parts could be allowed.
 - Is version "2.10" after "2.1" or are they the same? What about
 "2.01" vs "2.1"? I would vote for "2.10>  2.1 == 2.01", because I
 see version "parts" as distinct numbers separated by a period,
 rather than fractional numbers.

I haven't thought about that. I see version parts as distinct numbers as well.
 - It should allow boolean operators and parens for the version
 selections. For instance: "(>= 2.1&&  <= 2.6&&  != 2.4) ||>=
 3.4" (Ie, "Any version from 2.1 through 2.6, but 2.4 has critical
 bugs, and 3.4+ contains a 2.x compatibility layer.")

Hehe. Now this is getting quite complicate, but it would be nice to have yes. Not something I will aim for in the first release.
 - There should be a "list" command and a "list {package name}"
 command to see what's installed. Maybe even "list {package name}
 {version expression}"? And maybe something too see what's available
 but not yet installed? They should all list the versions in
 guaranteed-sorted order so you can see which are the newest and
 oldest installed versions by looking at the first and last.

Absolutely, I've completely forgot about the "list" command.
 - At some point, 7z should be supported (and tarballs, of course).

If someone is willing to create D module of create bindings for any available libraries. Any libraries the forces a specific license is out of the question.

The lzma SDK is public domain (lzma is the compression algorithm used in 7z), xzutils has a library called liblzma which is based on the lzma sdk, easier to use and also public domain. Tar should be easy enough to implement in d. Seems like there are more or less recent D bindings for liblzma: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=117344 But someone still has to combine those parts together. LZMA bindings would also fit nicely into phobos, as we already have zlib in there.
 - How should platforms be handled WRT packages? Ie, Do all platforms
 need to be in the same orb package? Do they all need to be in
 separate packages? Either way? If they're not all required to be in
 the same package, how does orb find the package that had the right
 platform?

I good questions. I haven't given binary packages that much thought. I was first going for a source only package manager that requires all packages to be built before installed. I see three options: * One package for all platforms * Include the platform in the package name and in the orbspec * Have a sub path (on the server) for every platform, i.e.: dorbit.org/orbs/linux/dwt-1.3.2.orb.zip
 - Is it really necessary to have separate "build_dependencies" and
 "runtime_dependencies"? And why have both "runtime_dependencies" and
 "orbs" instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. Therefore it would be unnecessary to do a permanent installation on those dependencies. The user could get an option to either permanently installed these dependencies or to temporarily install them. The other way around would be possible as well. A package can depend on a dynamically linked library and use it only through function pointers. Then the package would only have a runtime dependency on the library, i.e. it wouldn't be needed when building.
 - Would it be a good idea to have and additional field "extra" for
 non-standard expandability? So people could add extra fields they
 felt would be useful, like "extra.foo" and "extra.bar", etc. And
 popular ones could eventually be formally added to the specification
 as just simply "foo" and "bar".

I guess there could be an "extra" field accepting a hash. How would the field be used, by other tools?
 - What's the point of the fields "files", "libraries" and
 "executables"? Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. "libraries" and "executables" are all libraries and executables that should be installed (regardless if they are pre-compiled or built during installation).
 - This supports having multiple versions of the same package
 installed at the same time, right? If not it should.

Yes, that's the whole point of having version, as I see it.
 - I see there's an "upgrade" callback, but I didn't see an "upgrade"
 command. Is upgrading in or out? I think that there should be an
 "upgrade" command that upgrades the installed versions of packages
 as far as it can *without* breaking any other installed packages
 that depend on it. Ex, if Foo requires Bar v2.6 or earlier, and
 SuperFoo requires Bar v2.7 or earlier, and Bar v2.3 is installed,
 but the latest Bar is v2.9, then "upgrade bar" would upgrade Bar
 v2.3 to v2.6 and display a message that says "Bar upgraded from 2.3
 to 2.6, but the newest is 2.9, run "orb install Bar" to install the
 newest Bar, too." (Or maybe it should install both 2.6 and 2.7? Or
 one/both of those and 2.9?) For upgrading, we should also think
 about how to do upgrades without clobbering any of it's settings.

I guess I didn't think this through. I think it will require some thought.
 - For POSTing a package to a repository, how does authentication
 work? All repos don't have to provide unrestricted upload access do
 they?

I haven't thought about this more than there will be some kind of authentication. Probably HTTP basic authentication.
 - I'm not sure I understand how the "source" command works. Can it be
 provided more than once? And then it just picks the first one that
 actually has the package?

The "source" command specifies a path to a repository where to fetch packages from. I haven't thought about if the it can be provided more than once. It might be a good idea.
 - The "central repositories" don't necessarily sound all that
 central, so they probably should just be called "repositories".

Ok.
 - What about default repositories? It should support that. (Kinda
 makes sense, otherwise how would "orb install xxx" know where to
 look?) And there should be simple commands to add/update/remove/list
 (and reorder?) the default repositories. If a package A specifies a
 dependency B and a repository for that dependency B, then which one
 has priority for downloading B: The default repositories or the
 repository specified by package A?

Yes there will be a default repository. As someone else suggested there could be an "orbfile" in the users home directory that can contain default settings like for "source". Or a more general config file for orbit.
 - Here's a problem with using an actual programming language for the
 orbfile/name.orbspec/metadata file: Suppose Orb version X uses Ruby
 version Y. Then, Orb X+1 comes out and which has Ruby upgraded to
 Y+1. Now, someone creates PackageA with an
 orbfile/name.orbspec/whatever that relies on Ruby Y+1. Someone else
 still has Orb version X and tries to get PackageA. Kaboom!

Yeah, that is a problem. But I wonder how much this will be a problem in practice. I don't think this will so big problem in practice using Ruby as the language. On the other hand, using D, will be a big problem. D breaks something in every release.
 Therefore, the orbfile/name.orbspec/whatever needs to specify which
 version of Orb (or Ruby) it requires. But now we have a
 chicken-and-the-egg problem: How can Orb X figure out that PackageA
 requires Orb X+1 if Orb X can't properly read PackageA's
 orbfile/name.orbspec/whatever?

That is a problem. If the metadata is "compiled" YAML/JSON then we can get around this.
 - If I install D library "libfoo", then I should be able write
 myapp.d with "include foo.blah;" and then do "dmd myapp.d" *without*
 manually specifying -Ipath_to_libfoo. It should just work. How will
 Orb handle that?

It can't. The solution to this is a build tool, as I see it. The build tool knows about the package manager and let you specify dependencies on package. Think about Drake, it could look like this: target("myapp.d", { orb("libfoo"); // automatically links with "libfoo" and includes its header path. }); -Ipath_to_libfoo needs somehow be passed to the compiler, and linking with the library as well. Maybe it would be possible to manipulate the dmd.conf/sc.ini but this seems very complicated.
 And how will that interact with DVM? Ie, if I do "dvm use 2.051",
 then "orb install libfoo", then "dvm use 2.054", then I should still
 have access to libfoo without needing to specify -Ipath_to_libfoo.

If you do "dvm use 2.051", then "orb install libfoo" then "libfoo" will only be installed for dmd 2.051, that's the whole point. You would have to run "orb install libfoo" again after switching compiler. Orbit could of course share the same package if possible.
 - Where does everything get installed?

For now, on Posix, in /usr/local/orbit. If used through DVM it will be installed in somewhere in ~/.dvm.
 - In many ways this sounds a lot like a generalized DVM. Maybe Orb
 should eventually take over DVM's duties by making a DMD orb package.

No, I don't think so. DVM is quite specialized in what it does. Manipulating the PATH variable (or the registy on Windows) to be able to do what it does. I don't what to mix DVM and Orbit.

-- Johannes Pfau
Jul 15 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-18 00:26, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 But why should the name of the project and the name of the tool be
 different? It would be less confusing to have a packager manager
 named Orb that gets invoked via "orb". (Or a package manager named
 Orbit that gets invoked via "orbit".

I don't know. It's also easy to separate the library code in one package named "orbit" and the application code in on package named "orb".
 I think the "metadata" would make sence if, and *only* if, it were
 "compiled" down to a pure-data format.

Ok, fair enough. That was the initial idea, to have the "metadata" file in a pure-data format.
 As for orbfile vs orbspec: I still think these could and should be
 merged. I don't need a separate buildscript for different actions.
 *Everything* the buildsystem does is configured in one centralized
 script (and whatever other files it may or may not choose to
 delegate out to). I see no reason why the same can't or shouldn't be
 done with package management.

I guess I can do that.
 If your "orbblah" file doesn't contain the parts needed for orb to
 auto-create a package, then you just can't have orb auto-create your
 package. Simple.

Ok.
 Also, I don't see why the name of "orbspec" needs to be prefixed
 with the package's name. Just like an orbfile, or makefile, or
 rakefile, etc., you already know what package it's for from what
 directory/package it's inside of.

The package needs a name. One option is to specify that in the orbspec file. Another option is to specify it in the name of the orbspec file. I just thought that it was convenient. Is it a good idea to take the name directory where the orbspec is located? Note that many of my choices are based on RubyGems.
 Well, any version that's>= 0.3.4 and<  0.4.0

Ok.
 That would make sense. But, it could always be forcefully done by
 doing it manually without going through orb. Don't know if that's
 worth worrying about though.

Do you mean if you have direct access to the server where the repository is located? I don't think that's something worth worrying about. Isn't that the same with all servers you don't own yourself? If you have a website on a hosted server they can, if they want to, just remove it.
 I agree with inferring 0 for any parts not provided.

Good.
 I think it should be up to the project developer to choose how many
 parts if appropriate for their project. And I don't see any problems
 with doing that.

Ok. But then I think we most allow: "> 0.3.4+". Or else you cannot know if a given version will break the API.
 I think that might be a bit limiting. And keep in mind, just because
 something is source-only doesn't mean it'll actually work on more
 than one platform.

As you can read in the specification I do intend to have binary packages. But as I said, that was not my original thought and that's why I haven't given it much thought. Of course it won't work on all platform just because it's source only but if there are source only packages then the problem below doesn't exist.
 I see three options:

 * One package for all platforms
 * Include the platform in the package name and in the orbspec
 * Have a sub path (on the server) for every platform, i.e.:

 dorbit.org/orbs/linux/dwt-1.3.2.orb.zip

I think the first option makes the most sense, and the second option sounds like it would be a sensible workaround when someone really wants things separate.

There's always this issue with downloading files you don't need.
 I'm not sure I like the third option because that wouldn't work for
 repositories that aren't explicity chosen unless you made it a
 standard naming system, but then that creates extra requirements for
 the server, and then how do you know whether to GET from the
 platform-specific directory or the
 platform-independent/cross-platform directory, etc. And what about
 having a windows package and a combination OSX/linux package? It's
 workable, but I'm not sure it's worth it in light of the first two
 options.

I don't understand what you mean. Why would you have one package for Windows and one for osx/linux? Or have I misunderstood something.
 Although, on second thought, if a good system for the third option
 could be worked out, then maybe that would be best. (But maybe just
 not for the initial release?)

An other issue with one package for all platforms is when building the package. The developer first needs to build the package on one platform, then move the package to all other platforms and rebuild the package. The tool needs to open the package and add files for the other platforms.
 - Is it really necessary to have separate "build_dependencies" and
 "runtime_dependencies"? And why have both "runtime_dependencies"
 and "orbs"
 instead of just picking one name and sticking with it?

There is no runtime dependency on something that is statically linked. Therefore it would be unnecessary to do a permanent installation on those dependencies. The user could get an option to either permanently installed these dependencies or to temporarily install them.

So you mean a statically-linked dependency then? "Runtime dependency" makes me think it's needed at runtime and thus *would* need to be installed. Actually, what you're describing sounds just like a build dependency to me.

Statically-linked dependency is basically the same as build dependency. If you have a look at the DVM project page you can see the list of build dependencies. The only thing you need to actually run the application is zlib (on Poisx), because it's dynamically linked, aka a runtime dependency.
 Is there really all that much benefit to doing a temporary install
 of things temporarally needed anyway? And is it that common of a
 need? I'm wondering if it's worth the extra complexity.

The package would need access to the dependencies when its build. How should it else be done?
 The other way around would be possible as well. A package can
 depend on a dynamically linked library and use it only through
 function pointers. Then the package would only have a runtime
 dependency on the library, i.e. it wouldn't be needed when building.

Hmm I think I still don't understand the whole runtime dep vs build dep matter.

If you use, on Posix, dlopen and friends to open a dynamic library you don't need to link to it when building. You call the functions via function pointers, that's how Derelict works, if you have used it.
 That would be up to them. The idea is just that the package metadata
 is extendable for whatever uses may arise.

Ok, that won't be any problem including.
 - What's the point of the fields "files", "libraries" and
 "executables"? Seems like extra work for no real benefit.

"files" is basically all files that should be put in the package. "libraries" and "executables" are all libraries and executables that should be installed (regardless if they are pre-compiled or built during installation).

So it's purely for creating a package?

Yes, the "files" field is just when creating the package.
 If not, then I'm afraid I still don't understand.

 If so, then I still don't see the use of "libraries" and
 "executables". Also I think they should all be optional (if they
 aren't already) because all that info is already going to exist in
 the buildscript, and so a lot of people (like me) are going to want
 to generate the package via their build system rather than
 duplicating all that work and information just so that the package
 manager can do it "automatically".

Both the "libraries" and "executables" fields are optional. "summary" and "version" are the only required fields (so far), as you can see on the wiki. As I said before, "libraries" and "executables" are what actually should be installed. This is what I'm thinking and is planning for Orbit and my build tool Dake. Dake will be able to generate an orbspec file, because, as you said, the build tool will have all the necessary information. Then you don't have to duplicate any information. But, I'm not forcing anyone to use my build tool if they want to use Orbit, therefore the fields "libraries" and "executables" are available.
 Other things to think about are: How does the user actually specify
 the login info? Wouldn't want them in the
 orbfile/orbspec/metadata/etc.

No, absolutely not. Passing user name and password using HTTP basic authentication on the command line?
 Ouch. I have to say, I don't like that at all. The way I see it,
 this is one of the two primary responsibilities of a package manager
 for a language (the other being automatic dependency handling).
 Without it, we're not much ahead of where we are right now.

If you have suggestions I'm listening.

It would be possible to install libraries into the dmd default search path. Right now, this would be /usr/include/d/dmd and library files in /usr/lib on posix, but any path can be used as long as it's included in dmd.conf. However, this means that only one version of a library can be installed system-wide, so it's not optimal. It should also be noted, that linking against a specific library version can only work well with static libraries. With static libraries you can give linker a specific path to the library at compile time (So you can have multiple versions in different directories). Using different directories with dynamic libraries requires setting LD_LIBRARY_PATH before executing a program, so this won't work. We'll have to use the library versioning mechanism that's used by C libraries (e.g for linux: http://www.ibm.com/developerworks/linux/library/l-shlibs/index.html ). I don't know if windows even supports library versioning, but as windows programs usually don't install dlls globally that's less of a problem.
 That doesn't work: If orb is managing the packages, then how is
 dake/drake/etc supposed to know what path to include for -I? Orb
 knows where it's sticking packages. The build systems don't know
 that.

The build tool invokes Orbit and asks about the include path.
 This also reminds me of another issue: For D libraries, the
 orbfile/orbspec/metadata/whatever needs to give the relative path
 for the base include directory.

Something like that. Or if given full path the tool will assume the current directory is the base directory and remove that from the path. For example: $ cd ~/projects/d/foo Contains: main.d bar/foobar.d # foo.orbspec files << ["~/projects/d/foo/main.d", "~/projects/d/foo/bar/foobar.d"] $ orb build foo Since I run the "orb build" command in the "~/projects/d/foo" directory the tool will just remove that from the paths when including the files in the package.
 I think what we need is to have DVM handle dmd.conf/sc.ini
 separately from the compiler version (which kind of goes along with
 DVM Issue #9). Orb and DVM need to work together to manage
 dmd.conf/sc.ini. It's complicated, but I think it's necessary.

I was hoping I didn't need to drag in DVM in this. And I don't won't to force anyone using DVM if they want to use Orbit.
 This makes me think of another, too: How does the buildscript know
 what version of a lib orb has chosen? Since orb packages
 (rightfully) are allowed to specify a range of versions instead of
 just one specific version, I think that will be needed.

It doesn't need to know what exact version is chosen, it just needs to know the include paths and libraries to link with. The rest is up to Orbit to handle and decide. The build tool just links with what it received from Orbit.
 I think that's very, very bad. I should be able to compile something
 with a different version of DMD without reinstalling everything.
 "dvm use xxx" should *just work*, just as it already does now.

The whole point is to be able to have different packages installed with different compilers. There could be a command for moving over packages from one DMD installation to another. Say I'm using DMD 2.053, then installing package "foo". Then I'm installing DMD 2.054 and switching to it. Say also that package "foo" doesn't work with DMD 2.054, what happens when you switch to 2.054?
 Orb manages different versions of arbitrary packages. DVM manages
 different versions of one specific package. If they're so different
 despite all that, then either Orbit isn't generalized enough, or DVM
 is too specialized.

DVM is very specialized, yes. The installation and switching of D compiler, using DVM, is very specialized. DMD requires wrappers, manipulating the PATH or registry. None of the Orbit packages require this. Orbit would require loads of special cases to have DVM built-in. On your description DVM and Orbit sound very similar but how they actually work when installing and switch compilers are very different. You should know this, you ported DVM to Windows. The current compiler decides what packages are available/installed, that wouldn't work if the compiler package would just be a regular orb package. What would be possible it to create a tool that is a front end for DVM and Orbit. I'm currently building Orbit as a library with a thin wrapper that is the tool the user will use. I'm planning to rewrite DVM in the same way. This also allows to create GUIs and integrate the tools into IDEs.

-- Johannes Pfau
Jul 18 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-18 18:03, Johannes Pfau wrote:
 It would be possible to install libraries into the dmd default search
 path. Right now, this would be /usr/include/d/dmd and library files
 in /usr/lib on posix, but any path can be used as long as it's
 included in dmd.conf. However, this means that only one version of a
 library can be installed system-wide, so it's not optimal.

 It should also be noted, that linking against a specific library
 version can only work well with static libraries. With static
 libraries you can give linker a specific path to the library at
 compile time (So you can have multiple versions in different
 directories). Using different directories with dynamic libraries
 requires setting LD_LIBRARY_PATH before executing a program, so this
 won't work. We'll have to use the library versioning mechanism
 that's used by C libraries (e.g for linux:
 http://www.ibm.com/developerworks/linux/library/l-shlibs/index.html
 ). I don't know if windows even supports library versioning, but as
 windows programs usually don't install dlls globally that's less of a
 problem.

You can give a specific path to the compiler with dynamic libraries as well. Link with libfoo, version 3.4.0: dmd -L-L/path/to/libfoo-3.4.0 -L-lfoo Or have I missed something?

I'd have to test that, but I doubt it will work. This will help to find the library at compile time, but not at runtime. The runtime linker will only search in directories listed in /etc/ld.so.conf.d or in the LD_LIBRARY_PATH variable. Each .so library has a 'soname' embedded. If you link like in your example command the resulting binary only contains the sonames of the libraries it needs, not the full path. At runtime, the linker then reads that soname and searches in its cache for a library with the same soname. It might be possible to make this soname mechanism use absolute paths or subdirectories, but this seems like a hack. Sonames are usually just "libfoo.so.3" where 3 is a ABI revision. I think we'll eventually have to install shared libraries exactly the way C does it, i.e all in /usr/lib and using the soname versioning. But we can think about that when dmd finally supports shared libraries on linux, it's not important right now. -- Johannes Pfau
Jul 19 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-19 10:16, Johannes Pfau wrote:
 I'd have to test that, but I doubt it will work. This will help to
 find the library at compile time, but not at runtime. The runtime
 linker will only search in directories listed in /etc/ld.so.conf.d
 or in the LD_LIBRARY_PATH variable. Each .so library has a 'soname'
 embedded. If you link like in your example command the resulting
 binary only contains the sonames of the libraries it needs, not the
 full path. At runtime, the linker then reads that soname and
 searches in its cache for a library with the same soname. It might
 be possible to make this soname mechanism use absolute paths or
 subdirectories, but this seems like a hack. Sonames are usually just
 "libfoo.so.3" where 3 is a ABI revision. I think we'll eventually
 have to install shared libraries exactly the way C does it, i.e all
 in /usr/lib and using the soname versioning. But we can think about
 that when dmd finally supports shared libraries on linux, it's not
 important right now.

Oh, at runtime, didn't think of that :). The above command is only for compile time. What about the linker flag "-rpath"? That seems it could be used. Linux is not the only OS, it's easy to add support for dynamic libraries on Mac OS X. All code is already in druntime, it just needs to be enabled. Tango already supports this.

Seems like rpath could indeed work in this case. I can't find much documentation about it though. Debian recommends not to use it: http://wiki.debian.org/RpathIssue but I'm not sure if this problem applies to orbit. I'd prefer installing shared libraries system wide though. The soname/version approach is not that bad. Your proposed package versioning scheme could even be mapped 1:1 to the soname versions. Or we could use libtools versioning scheme, which is similar, ('major' and 'minor' are one field, 'build' stays the same, and an additional 'age' field is added) http://sourceware.org/autobook/autobook/autobook_91.html Having read more about it, i think I have to correct my previous statement: It is possible to link to specific versions with the soname approach. It's maybe a little more limited (You can't say: "I want to use libfoo.so.1.2.0", You can only say: "I want to use libfoo 1.x.x", and the linker could end up using 1.1.0, 1.2.0 ...) but it seems this should be good enough. -- Johannes Pfau
Jul 19 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

Won't the same problem occur if rpath isn't used? With LD_LIBRARY_PATH for example.

I'm not sure about that, but using LD_LIBRARY_PATH is even more discouraged (because it propagates to child processes)
 I'd prefer installing shared libraries system wide though. The
 soname/version approach is not that bad. Your proposed package
 versioning scheme could even be mapped 1:1 to the soname versions. Or
 we could use libtools versioning scheme, which is similar, ('major'
 and 'minor' are one field, 'build' stays the same, and an additional
 'age' field is added)
 http://sourceware.org/autobook/autobook/autobook_91.html

I don't want to install the libraries system wide. Again your assuming Linux only. It has to work on all supported platforms. At least: Linux, Mac OS X and Windows.

I'm not sure if we can have one perfect approach for all operating systems. But we could just use the rpath way and see how it works out.
 Having read more about it, i think I have to correct my previous
 statement: It is possible to link to specific versions with the
 soname approach. It's maybe a little more limited (You can't say: "I
 want to use libfoo.so.1.2.0", You can only say: "I want to use
 libfoo 1.x.x", and the linker could end up using 1.1.0, 1.2.0 ...)
 but it seems this should be good enough.

No, I want to be able to use an exact version.

This mechanism always picks up the newest ABI compatible library version. Why would you want to specify one version explicitly if the newer version can be used as a 1:1 replacement? -- Johannes Pfau
Jul 19 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-19 13:43, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

Won't the same problem occur if rpath isn't used? With LD_LIBRARY_PATH for example.

I'm not sure about that, but using LD_LIBRARY_PATH is even more discouraged (because it propagates to child processes)

Ok. What is used to tell the system where to find a library? Only the default paths as /usr/lib ?

The rpath page says this: 1. the RPATH binary header (set at build-time) of the library causing the lookup (if any) 2. the RPATH binary header (set at build-time) of the executable 3. the LD_LIBRARY_PATH environment variable (set at run-time) 4. the RUNPATH binary header (set at build-time) of the executable 5. /etc/ld.so.cache (generated from /etc/ld.so.conf and /etc/ld.so.conf.d) 6. base library directories (/lib and /usr/lib) But that depends on the C library / linker implementation. -- Johannes Pfau
Jul 22 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

The problem mentioned on that page is: A problem arises when binary A defines a NEEDED dependency on libraries B.so.1 and C.so.2, while library B.so.1 depends on library C.so.1 How is this handled when rpath isn't used and in general?

To be honest, I don't know. I'm not even sure if I understand the issue with rpath at all, but I thought I'd better mention it. -- Johannes Pfau
Jul 22 2011
prev sibling next sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-22 12:38, Johannes Pfau wrote:
 Jacob Carlborg wrote:
 On 2011-07-19 12:33, Johannes Pfau wrote:
 Seems like rpath could indeed work in this case. I can't find much
 documentation about it though. Debian recommends not to use it:
 http://wiki.debian.org/RpathIssue but I'm not sure if this problem
 applies to orbit.

The problem mentioned on that page is: A problem arises when binary A defines a NEEDED dependency on libraries B.so.1 and C.so.2, while library B.so.1 depends on library C.so.1 How is this handled when rpath isn't used and in general?

To be honest, I don't know. I'm not even sure if I understand the issue with rpath at all, but I thought I'd better mention it.

I can see that there could be a problem but I don't see how this can be a problem only for rpath. Seems to me that it would be a problem regardless which paths are used. So what happens if you link two libraries which are the same library but of different versions? Conflicting symbols?

Yes I think you'll get conflicting symbols. It is possible to add version information to a symbol, but I don't know how often this feature is used. (Or how it works exactly) Example: objdump -T /lib/i386-linux-gnu/libc.so.6 00000043 GLIBC_2.0 pthread_attr_getinheritsched 0000002d GLIBC_2.5 __readlinkat_chk 00000076 GLIBC_2.1 key_decryptsession 00000097 GLIBC_PRIVATE __nss_hosts_lookup2 -- Johannes Pfau
Jul 22 2011
prev sibling parent Johannes Pfau <spam example.com> writes:
Jacob Carlborg wrote:
On 2011-07-22 12:38, Johannes Pfau wrote:
 The rpath page says this:
 1. the RPATH binary header (set at build-time) of the library causing
     the lookup (if any)

 2. the RPATH binary header (set at build-time) of
     the executable

 3. the LD_LIBRARY_PATH environment variable (set at
     run-time)

 4. the RUNPATH binary header (set at build-time) of the
     executable

 5. /etc/ld.so.cache (generated from /etc/ld.so.conf
     and /etc/ld.so.conf.d)

 6. base library directories (/lib and /usr/lib)

 But that depends on the C library / linker implementation.

Since RPATH and LD_LIBRARY_PATH is discouraged only 4, 5 and 6 are left?

As far as I know, yes. However, note that 5 and 6 are 'global' options, they affect all executables and (dynamic) libraries. -- Johannes Pfau
Jul 22 2011