www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP11

reply jdrewsen <jdrewsen nospam.com> writes:
What is the status of DIP11

http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

Has anyone started implementing it? Has it been rejected?

/Jonas
Aug 10 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D -- /Jacob Carlborg
Aug 11 2011
next sibling parent reply Jonas Drewsen <jdrewsen nospam.com> writes:
On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works. /Jonas
Aug 11 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 09:41, Jonas Drewsen wrote:
 On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works. /Jonas

I think that DIP11 is too limited, for example, it doesn't deal with versions. Orbit combined with a build tool will be seamless as well. RDMD is a great tool but as soon as you need to add compiler flags or compile a library you need either some kind of script or a build tool. And in that case you can just go with the built tool and have it work on all platforms. -- /Jacob Carlborg
Aug 11 2011
next sibling parent Jonas Drewsen <jdrewsen nospam.com> writes:
On 11/08/11 09.49, Jacob Carlborg wrote:
 On 2011-08-11 09:41, Jonas Drewsen wrote:
 On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works. /Jonas

I think that DIP11 is too limited, for example, it doesn't deal with versions. Orbit combined with a build tool will be seamless as well. RDMD is a great tool but as soon as you need to add compiler flags or compile a library you need either some kind of script or a build tool. And in that case you can just go with the built tool and have it work on all platforms.

Some refinements needs to be done to the DIP yes. One of them is version handling IMHO. /Jonas
Aug 11 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 14:52, Steven Schveighoffer wrote:
 Given that the implementation would be a compiler-used tool, and the
 tool can implement any protocol it wants, I think it has very few
 limitations. I envision the tool being able to handle any network
 protocol or packaging system we want it to.

That might be the case. Since it's arbitrary URLs that represents D modules and packages, to me it seems that there needs to be a lot of conventions: * Where to put the packages * How to name them * How to indicate a specific version and so on.
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc. To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

 -Steve

Until the compiler can automatically compile dependencies we need build tools. What about linking with pre-compiled libraries, how would that work? Currently the linking paths needs to be known before the compiler is invoked. You would first need to compile without linking and then link, or something like that. Assuming the compiler isn't changed to be able to receive linker options form the external tool. Note that I'm basing all this on what's written in the DIP (and what you've said), as far as I know that's the current suggestion. But the DIP can of course be enhanced and updated. -- /Jacob Carlborg
Aug 11 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 17:45, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 10:47:36 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 14:52, Steven Schveighoffer wrote:
 Given that the implementation would be a compiler-used tool, and the
 tool can implement any protocol it wants, I think it has very few
 limitations. I envision the tool being able to handle any network
 protocol or packaging system we want it to.

That might be the case. Since it's arbitrary URLs that represents D modules and packages, to me it seems that there needs to be a lot of conventions: * Where to put the packages * How to name them * How to indicate a specific version

These are implementation details. The compiler just knows "hm.. there's some external file, I don't know how to get it, tool, can you help me out?" The tool could potentially download the files and build them, or download a pre-compiled package. Or it could add files to the compiler's todo list somehow (having recursive compiler invocations might be bad...)

In that case that needs to be said in the DIP.
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc. To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

 -Steve

Until the compiler can automatically compile dependencies we need build tools.

I agree it would be advantageous to have the build tool add files to the compiler's todo list. As of now, just downloading source for import does not help because you still need to compile the file, not just import it.

Yes, exactly.
 But you have to admit, just having the source file or include path
 include parts of the internet would be pretty cool. I really like that
 approach. Its sort of like how dsss used to work, except dsss contained
 a huge portion of the compiler in it. At least with DIP11, the compiler
 is the driver, no need to maintain a separate "compiler". It could be
 that DIP11 needs more work to get a technically useful and implementable
 feature.

It sounds pretty cool but the question is how well it would work and if the right thing to do.
 What about linking with pre-compiled libraries, how would that work?
 Currently the linking paths needs to be known before the compiler is
 invoked. You would first need to compile without linking and then
 link, or something like that. Assuming the compiler isn't changed to
 be able to receive linker options form the external tool.

I'm assuming the linker path in dmd.conf would include some global or user-specific cache directory where pre-compiled libraries are downloaded. Then the download tool puts the libraries in the right spot, so the path does not need adjusting.

Assuming that, you would still need to link with the libraries. I don't know if pragma(lib, ""); could work but I don't like that pragma in general. It's platform dependent and I'm not sure if it works for dynamic libraries. I don't think GDC implements it.
 Note that I'm basing all this on what's written in the DIP (and what
 you've said), as far as I know that's the current suggestion. But the
 DIP can of course be enhanced and updated.

Yes, and much better than a DIP is a working solution, so it might very well be that Orbit wins :) I certainly don't have the time or expertise to implement it... -Steve

I guess we just have to see what happens. I will at lease continue working on Orbit. -- /Jacob Carlborg
Aug 11 2011
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:j21190$ls8$1 digitalmars.com...
 Assuming that, you would still need to link with the libraries. I don't 
 know if pragma(lib, ""); could work but I don't like that pragma in 
 general. It's platform dependent

This works cross-platform: pragma(lib, "LibNameWithoutExt"); And if you do need platform-specific, there's always version().
 and I'm not sure if it works for dynamic libraries. I don't think GDC 
 implements it.

A major downside of GDC, IMO.
Aug 11 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-11 23:30, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:j21190$ls8$1 digitalmars.com...
 Assuming that, you would still need to link with the libraries. I don't
 know if pragma(lib, ""); could work but I don't like that pragma in
 general. It's platform dependent

This works cross-platform: pragma(lib, "LibNameWithoutExt");

Will that work with all available library types (static, dynamic) on all platforms?
 And if you do need platform-specific, there's always version().

You do, since on Posix most library names are prefixed with "lib". DSSS/Rebuild did this very well with the pragma "link". It prefixed the library names depending on the platform.
 and I'm not sure if it works for dynamic libraries. I don't think GDC
 implements it.

A major downside of GDC, IMO.

Yes. -- /Jacob Carlborg
Aug 12 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 19:07, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley
 <wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc. To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

So how would that be different if the compiler drives everything? Say you begin with a few local files. The compiler then scans through them looking for URL imports. Then asks a tool to download the dependencies it found and starts all over again. This is how my package manager will work. You have a local file containing all the direct dependencies needed to build your project. When invoked, the package manager tool fetches a file containing all packages and all their dependencies, from the repository. It then figures out all dependencies, both direct and indirect. Then it downloads all dependencies. It does all this before the compiler is even invoked once. Then, preferably, but optionally, it hands over to a build tool that builds everything. The build tool would need to invoke the compiler twice, first to get all the dependencies of all the local files in the project that is being built. Then it finally runs the compiler to build everything. Well, actually if you're using a build tool it would drive everything. You have the package dependencies in the build script file. The build tool starts by invoking the package manager (see above), then it can query the package manager for include and library paths and libraries to link with. As the final step it invokes the compiler to build everything (see above).
 And there is no parsing of the output data, the problem boils down to a
 simple get tool. Running a simple get tool over and over doesn't consume
 as much time/resources as running the compiler over and over.

 There are still problems with the DIP -- there is no way yet to say "oh
 yeah, compiler, you have to build this file that I downloaded too". But
 if nothing else, I like the approach of having the compiler drive
 everything. It reduces the problem space to a smaller more focused task
 -- get a file based on a url. We also already have many tools in
 existence that can parse a url and download a file/package.

 -Steve

The best would be if the compiler could be a library. Then the build tool could drive everything and ask other tools, like the a package manager and compiler about information it needs to build everything. -- /Jacob Carlborg
Aug 11 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 20:31, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 14:19:35 -0400, Jacob Carlborg <doob me.com> wrote:
 So how would that be different if the compiler drives everything? Say
 you begin with a few local files. The compiler then scans through them
 looking for URL imports. Then asks a tool to download the dependencies
 it found and starts all over again.

Forgive my compiler ignorance (not a compiler writer), but why does the compiler have to start over? It's no different than importing a file, is it?

Probably it's no different. Well what's different is it will first parse a couple of files, then downloads a few files, then parse the downloaded files and the fetch some more files and so on. To me this seems inefficient but since it's not implemented I don't know. It feels more efficient if it could download all needed files in one step. And then compile all files in the next step. I don't know what's possible with this DIP but to me it seems that the current suggestion will download individual files. This also seems inefficient, my solution deals with packages, i.e. zip files.
 This is how my package manager will work. You have a local file
 containing all the direct dependencies needed to build your project.
 When invoked, the package manager tool fetches a file containing all
 packages and all their dependencies, from the repository. It then
 figures out all dependencies, both direct and indirect. Then it
 downloads all dependencies. It does all this before the compiler is
 even invoked once.

 Then, preferably, but optionally, it hands over to a build tool that
 builds everything. The build tool would need to invoke the compiler
 twice, first to get all the dependencies of all the local files in the
 project that is being built. Then it finally runs the compiler to
 build everything.

The benefit of using source is the source code is already written with an import statement, there is no need to write an external build file (all you need is command line that configures the compiler).

I don't see the big difference. I think the most project (not the smallest ones) will end up with a special file anyway, containing the pragmas declaring these imports. In addition to that as soon as you need to pass flags to the compiler you will most likely to put that in a file of some kind. In that case you can just as easily put them in a build script and use a build tool.
 Essentially, the import statements become your "build file". I think
 dsss worked like this, but I don't remember completely.

Yes, this is similar how DSSS worked. The difference is that you didn't need a pragma to link a package to an URL you just wrote the import declarations as you do now. One problem I think DSSS has, is, as far as I know, it can't handle top level packages with same name. Or at least not in any good way. If you go with the Java package naming scheme and name your top level package after your domain, example: module com.foo.bar; module com.foo.foobar; And another project does the same: module com.abc.defg; Then these two projects will both end up in the "com" folder. Not very good in my opinion. In my solution every package has a name independent of the packages it contains an all packages are placed in a folder named after the package, including the version number.
 My ideal solution, no matter how it's implemented is, I get a file
 blah.d, and I do:

 xyz blah.d

 and xyz handles all the dirty work of figuring out what to build along
 with blah.d as well as where to get those resources. Whether xyz == dmd,
 I don't know. It sure sounds like it could be...


 -Steve

Yeah, I would like that too. But as I said above, as soon as you need compiler flags you need an additional file. With a built tool it can then be just: "$ build" -- /Jacob Carlborg
Aug 12 2011
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
I did write build2.d, which tries to simulate dip11 outside
the compiler.

http://arsdnet.net/dcode/build2.d

Since it's not actually in the compiler, it can't be perfect
but it sorta tries.
Aug 12 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-12 15:49, Steven Schveighoffer wrote:
 Note that the pragmas are specific to that file only. So you don't have
 an import file which defines pragmas. This is to prevent conflicts
 between two files that declare the same package override.

Now I'm not quite sure I understand. Are you saying that every file needs to have these pragma imports ?
 In addition to that as soon as you need to pass flags to the compiler
 you will most likely to put that in a file of some kind. In that case
 you can just as easily put them in a build script and use a build tool.

a batch files/shell script should suffice, no need for a "special" tool.

Yeah, but you would need to duplicate the shell script, one for Posix and one for Windows. As far as I know there is no scripting like file that both Windows and Posix can read out of the box.
 Essentially, the import statements become your "build file". I think
 dsss worked like this, but I don't remember completely.

Yes, this is similar how DSSS worked. The difference is that you didn't need a pragma to link a package to an URL you just wrote the import declarations as you do now.

IIRC, dsss still had a global config file that defined where to import things from. The DIP defines that -I switches can also define internet resources along with pragmas, so sticking those in the dmd.conf would probably be the equivalent.

Ok, I see.
 One problem I think DSSS has, is, as far as I know, it can't handle
 top level packages with same name. Or at least not in any good way. If
 you go with the Java package naming scheme and name your top level
 package after your domain, example:

 module com.foo.bar;
 module com.foo.foobar;

 And another project does the same:

 module com.abc.defg;

 Then these two projects will both end up in the "com" folder. Not very
 good in my opinion. In my solution every package has a name
 independent of the packages it contains an all packages are placed in
 a folder named after the package, including the version number.

These all seem like implementation details. I don't care how the tool caches the files.

Well yes, but that is how DSSS works and that's what I'm explaining.
 Or instructions on the web site "use 'dmd -O -inline -release
 -version=SpecificVersion project.d' to compile"

 Or build.sh (build.bat) Note that dcollections has no makefile,
 everything is built from shell scripts. I almost never have to edit the
 build file, because the line's like:

You would need to shell script files, one for Windows and one for Posix, see above.
 dmd -lib -O -release -inline dcollections/*.d dcollections/model/*.d

 Any new files get included automatically. And it takes a second to
 build, so who cares if you rebuild every file every time?

 Interestingly, libraries would still need to specify all the files since
 they may not import eachother :) I don't know if there's a "good"
 solution that isn't too coarse for that.

That's what a build tool can handle. I think you should read this: http://dsource.org/projects/dsss/wiki/DSSSByExample and hopefully you'll understand why a built tool is a good thing.
 All of this discussion is good to determine the viability, and clarify
 some misinterpretations of DIP11, but I think unless someone steps up
 and tries to implement it, it's a moot conversation. I certainly don't
 have the time or knowledge to implement it. So is there anyone who is
 interested, or has tried (to re-ask the original question)?

 -Steve

I guess that's right. -- /Jacob Carlborg
Aug 12 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-12 20:36, Steven Schveighoffer wrote:
 On Fri, 12 Aug 2011 14:24:46 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:49, Steven Schveighoffer wrote:
 Note that the pragmas are specific to that file only. So you don't have
 an import file which defines pragmas. This is to prevent conflicts
 between two files that declare the same package override.

Now I'm not quite sure I understand. Are you saying that every file needs to have these pragma imports ?

Let's say file a.d pragmas that module foo means http://foo.com/projectx, and module b.d from another project pragmas that module foo means http://bar.com/projecty. If I import both a and b, what happens? It only makes sense for a pragma to affect the current file. This is similar to how version=x statements only affect the current file.

Again, will that mean you have to specify a pragma for each file? Just for the record, you cannot always solve all dependencies, it can happen that two packages conflict with each other. -- /Jacob Carlborg
Aug 12 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-12 21:22, Steven Schveighoffer wrote:

 Yes, or specify it on the command line/config file. I think the risk of
 inadvertently importing incorrect files is too great.

 Looking back at it, however, we probably would need some mechanism for
 files in the same package to inherit the source location.

 For example, if you pragma that a module foo = http://foo.com/projectx,
 and import foo.xyz, and foo.xyz imports foo.abc, we don't want foo.xyz
 to have to pragma the same url just to include another file in its own
 package.

 Clearly we need some more thought around this.

Yeah, that's what I was afraid of.
 Just for the record, you cannot always solve all dependencies, it can
 happen that two packages conflict with each other.

As long as the dependencies are contained, there should be no conflict. If I can compile project x and y separately, and both have a conflicting dependency, then I should still be able to compile a project that depends on both x and y, as long as they don't import eachother. -Steve

If x and y depends on two different versions of z, how would that be solved. As far as I know you cannot link the same library of two different version twice, you will get conflicting symbols. -- /Jacob Carlborg
Aug 13 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-15 15:11, Steven Schveighoffer wrote:
 What could happen though, is this situation:

 a.d pragma's foo.bar as being http://somedreposiotry.com/foo/barv1.0
 b.d includes the repository path http://somedrepository.com, whose
 default foo.bar goes to foo/barv2.0, but b does not depend on foo.bar

That would be a problem.
 I don't think a.d's pragma should include b's pragma, or vice versa.
 Especially if it is some sort of weird import order precedent.

 It's just much simpler to say "b's pragma only affects b, and a's pragma
 only affects a." However, I think we need to add that any files imported
 via a pragma implicitly include that path. I should probably add that to
 the DIP.

I guess so.
 Obviously if a.d depends on foo.bar, and b.d depends on foo.bar, but the
 locations are different, it should be a compiler error.

 -Steve

Ok. My original concern was to have to add pragmas to every D file. That is not a solution that will last long. But if you can add a compiler flag instead I guess that's ok. -- /Jacob Carlborg
Aug 15 2011
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
news:op.vz166yaieav7ka localhost.localdomain...
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley 
 <wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc.  To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

That's *only* true if you go along with DIP11's misguided file-oriented approach. With a real package manager, none of that is needed. Your app just says "I need packages X, Y and Z." And X, Y and Z do the same for their requirements. This is all trivial metadata. Emphasis on *trivial*. So, before DMD is ever invoked at all, before one line of the source is ever even read, the package manager has make sure that *everything* is *already* right there. No need to go off on some goofy half-cocked "compile/download/complile/download" dance. So DMD *never* needs to be invoked more than twice. Once to get the deps, once to compile. Better yet, if DMD gets the switch --compile-everything-dammit-not-just-the-explicit-files-fr m-the-command-line, then DMD never needs to be invoked more than *once*: Once to figure out the deps *while* being intelligent enough to actually compile all of them.
 And there is no parsing of the output data,

Parsing the .deps file is extremely simple. RDMD does it with one regex. Personally, I think even that is overkill. Better yet, with a switch to make DMD incorporate RDMD's --build-only functionality, there is *still* no parsing of output data. So all in all, there is *nothing* that DIP11 does that can't be done *better* by other (more typical) means.
Aug 11 2011
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:j219ck$3i0$1 digitalmars.com...
 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message 
 news:op.vz166yaieav7ka localhost.localdomain...
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley 
 <wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc.  To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

That's *only* true if you go along with DIP11's misguided file-oriented approach. With a real package manager, none of that is needed. Your app just says "I need packages X, Y and Z." And X, Y and Z do the same for their requirements. This is all trivial metadata. Emphasis on *trivial*. So, before DMD is ever invoked at all, before one line of the source is ever even read, the package manager has make sure that *everything* is *already* right there. No need to go off on some goofy half-cocked "compile/download/complile/download" dance. So DMD *never* needs to be invoked more than twice. Once to get the deps, once to compile. Better yet, if DMD gets the switch --compile-everything-dammit-not-just-the-explicit-files-fr m-the-command-line, then DMD never needs to be invoked more than *once*: Once to figure out the deps *while* being intelligent enough to actually compile all of them.
 And there is no parsing of the output data,

Parsing the .deps file is extremely simple. RDMD does it with one regex. Personally, I think even that is overkill. Better yet, with a switch to make DMD incorporate RDMD's --build-only functionality, there is *still* no parsing of output data. So all in all, there is *nothing* that DIP11 does that can't be done *better* by other (more typical) means.

In other words, DIP11 just reinvents the wheel, poorly.
Aug 11 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/11/11 1:09 PM, Nick Sabalausky wrote:
 "Steven Schveighoffer"<schveiguy yahoo.com>  wrote in message
 news:op.vz166yaieav7ka localhost.localdomain...
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley
 <wiley.andrew.j gmail.com>  wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc.  To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

That's *only* true if you go along with DIP11's misguided file-oriented approach. With a real package manager, none of that is needed. Your app just says "I need packages X, Y and Z." And X, Y and Z do the same for their requirements. This is all trivial metadata. Emphasis on *trivial*. So, before DMD is ever invoked at all, before one line of the source is ever even read, the package manager has make sure that *everything* is *already* right there. No need to go off on some goofy half-cocked "compile/download/complile/download" dance. So DMD *never* needs to be invoked more than twice. Once to get the deps, once to compile. Better yet, if DMD gets the switch --compile-everything-dammit-not-just-the-explicit-files-from-the-command-line, then DMD never needs to be invoked more than *once*: Once to figure out the deps *while* being intelligent enough to actually compile all of them.

It's difficult to get all dependencies when not all sources have been yet downloaded. Andrei
Aug 11 2011
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have been yet 
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.
Aug 11 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-12 00:02, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 17:20:04 -0400, Nick Sabalausky <a a.a> wrote:

 "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message
 news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have been
 yet
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.

With either style, you need to download a package in order to determine if you need to download other packages (package a may depend on package b even though your project does not depend on package b). The DIP11 version does this JIT, whereas your version does it before compilation. It's not really any different. -Steve

No, not if you have a single meta file with all the dependencies. -- /Jacob Carlborg
Aug 12 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 23:02, Andrei Alexandrescu wrote:
 It's difficult to get all dependencies when not all sources have been
 yet downloaded.

 Andrei

No, not when you have a single meta file containing all the dependencies of all packages. -- /Jacob Carlborg
Aug 12 2011
next sibling parent David Nadlinger <see klickverbot.at> writes:
On 8/12/11 11:05 AM, Jacob Carlborg wrote:
 On 2011-08-11 23:02, Andrei Alexandrescu wrote:
 It's difficult to get all dependencies when not all sources have been
 yet downloaded.

 Andrei

No, not when you have a single meta file containing all the dependencies of all packages.

Or a »spec« file along with each package containing the metadata, e.g. name, description, requirements, conflicts, … David
Aug 12 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/12/11 3:05 AM, Jacob Carlborg wrote:
 On 2011-08-11 23:02, Andrei Alexandrescu wrote:
 It's difficult to get all dependencies when not all sources have been
 yet downloaded.

 Andrei

No, not when you have a single meta file containing all the dependencies of all packages.

I understand. I believe there is value in avoiding meta files. Andrei
Aug 12 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-12 16:11, Andrei Alexandrescu wrote:
 On 8/12/11 3:05 AM, Jacob Carlborg wrote:
 On 2011-08-11 23:02, Andrei Alexandrescu wrote:
 It's difficult to get all dependencies when not all sources have been
 yet downloaded.

 Andrei

No, not when you have a single meta file containing all the dependencies of all packages.

I understand. I believe there is value in avoiding meta files. Andrei

This meta file lives on in the repository and is an implementation detail that the user never needs to know about. If this meta file can be created without meta files for individual packages I don't know. -- /Jacob Carlborg
Aug 12 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-11 21:36, Steven Schveighoffer wrote:
be one-file based, you just need to have a way to map
 physical packages to module packages.

 The DIP doesn't explain all this except for the sections titled
 "Packaging" and "protocols"

The DIP needs to explain this, is that the whole point? -- /Jacob Carlborg
Aug 12 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-12 15:56, Steven Schveighoffer wrote:
 On Fri, 12 Aug 2011 05:04:13 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 21:36, Steven Schveighoffer wrote:
 be one-file based, you just need to have a way to map
 physical packages to module packages.

 The DIP doesn't explain all this except for the sections titled
 "Packaging" and "protocols"

The DIP needs to explain this, is that the whole point?

The DIP focuses on the compiler changes needed + gives rudimentary functionality based on a simple protocol (http). I think the scope of work for creating the tool which does full-fledged packaging is way too much for the DIP, and probably would be counter productive to create a spec for it right now. The point of those two sections is to remind you not to think in terms of "this only downloads individual files via http", but to think about the future functinality that such a tool could provide once the compiler has been hooked. The details of the tool are purposely left open for design. -Steve

Ok, I see. -- /Jacob Carlborg
Aug 12 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-12 00:42, Martin Nowak wrote:
 It really looks sound to me.

 ---
 module myfile;
 pragma(imppath, "dep=www.dpan.org/dep");
 import dep.a;
 ---

 remote file
 ---
 module dep.a;

 // link directives
 pragma(libpath, "dep=www.dpan.org/dep");
 pragma(lib, "dep");

 // or alternatively some new pragma like this to cause linking the
 imported package
 pragma(build, "dep")

 // additional dependency
 pragma(imppatch, "dep2=www.dpan.org/dep2");
 ---

 Versioning can be easily resolved by using urls like
 www.dpan.org/dep?version=0.45 or www.dpan.org/dep-greaterthan-0.45.

 Pro:
 - scales from local dependencies to VPN sharing to package websites to
 even dynamically generated source code
 - simple implementation compilerwise

We will be very dependent on Walter or anyone else that knows the compiler, which, as far as I know, are quite few. I'm not sure if anything _is_ simple in the compiler. -- /Jacob Carlborg
Aug 12 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-12 15:53, Steven Schveighoffer wrote:
 This is not true. The compiler implements *hooks* for a download tool.
 The download tool will be a separate process which turns urls (generated
 by the compiler) into source files. Once the hooks are implemented, the
 tool is independent, and we would be idiotic not to implement it in D.

 I think you may not have read the DIP fully, or it is not clear enough.

 -Steve

I've read the whole DIP and I know there is an external tool that downloads the files. I also know that DMD doesn't have these hooks, I rest my case. -- /Jacob Carlborg
Aug 12 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-12 19:08, Steven Schveighoffer wrote:
 On Fri, 12 Aug 2011 12:30:42 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:53, Steven Schveighoffer wrote:
 This is not true. The compiler implements *hooks* for a download tool.
 The download tool will be a separate process which turns urls (generated
 by the compiler) into source files. Once the hooks are implemented, the
 tool is independent, and we would be idiotic not to implement it in D.

 I think you may not have read the DIP fully, or it is not clear enough.

 -Steve

I've read the whole DIP and I know there is an external tool that downloads the files. I also know that DMD doesn't have these hooks, I rest my case.

I thought you meant we would be dependent on Walter to write the *download part* of the DIP, not the compiler hooks. The hooks should be pretty simple I would think. But in any case, you should have a look at github and all the new people who are working on pull requests for the compiler. The group of dmd code contributors has significantly grown. -Steve

Yeah, that's true and it's a good thing. I can tell you this, I've looked at the DMD source code and tried to do modifications but I couldn't understand much of the code at all and failed with the most simple things. -- /Jacob Carlborg
Aug 12 2011
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jonas Drewsen" <jdrewsen nospam.com> wrote in message 
news:j20139$2sev$1 digitalmars.com...
 On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works.

I really see it as solving the wrong problem the wrong way. DIP11 tries to solve two main things: 1. Automatic downloading of external dependencies. 2. Eliminate the need for a separate invokation of DMD to find all needed D files (ie, efficiency). I strongly believe that #1 is best handled at the "package/library" level (and by that I'm referring to the "package manager" concept of "packages", not D's module system). DIP11 handles this at the individual source-file level, which I believe is wrong and causes plenty of problems. Plus, DIP11 is very, very limited compared to a tradtional-style package manager (and pretty much has to be since it works at the individual source-file level) and thus encourages very bad things like hardcoding exactly one possible location from which to retrieve a given file. Regarding #2, if that is deemed a real problem, then that can be solved by incorporating RDMD's "--build-only" functionality into the compiler (as an optional flag). This has two benefits over DIP11: A, It just works, nobody ever has to deal with DIP11's weird "DMD wants X to be retrieved", umm, "callback" mechanism. B, It's not tied to DIP11's broken package management design. Not only that, but DIP11, strangely enough, *fails* to solve issue #2 at all: It may download the needed D files, but it won't compile them. A separate tool *still* has to pass all the D files to DMD. And to do that it needs to know what D files are needed. And to do that, it needs to either ask DMD or be told by DMD, and *then* send all the D files to DMD as a separate invokation. The motivation for DIP11 was clearly to get a D package-retreiving system up and running with minimal effort. And that would be great if it were a good proposal. But DIP11 just smacks of corner-cutting.
Aug 11 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-11 17:41, Nick Sabalausky wrote:
 "Jonas Drewsen"<jdrewsen nospam.com>  wrote in message
 news:j20139$2sev$1 digitalmars.com...
 On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works.

I really see it as solving the wrong problem the wrong way. DIP11 tries to solve two main things: 1. Automatic downloading of external dependencies. 2. Eliminate the need for a separate invokation of DMD to find all needed D files (ie, efficiency). I strongly believe that #1 is best handled at the "package/library" level (and by that I'm referring to the "package manager" concept of "packages", not D's module system). DIP11 handles this at the individual source-file level, which I believe is wrong and causes plenty of problems. Plus, DIP11 is very, very limited compared to a tradtional-style package manager (and pretty much has to be since it works at the individual source-file level) and thus encourages very bad things like hardcoding exactly one possible location from which to retrieve a given file. Regarding #2, if that is deemed a real problem, then that can be solved by incorporating RDMD's "--build-only" functionality into the compiler (as an optional flag). This has two benefits over DIP11: A, It just works, nobody ever has to deal with DIP11's weird "DMD wants X to be retrieved", umm, "callback" mechanism. B, It's not tied to DIP11's broken package management design. Not only that, but DIP11, strangely enough, *fails* to solve issue #2 at all: It may download the needed D files, but it won't compile them. A separate tool *still* has to pass all the D files to DMD. And to do that it needs to know what D files are needed. And to do that, it needs to either ask DMD or be told by DMD, and *then* send all the D files to DMD as a separate invokation. The motivation for DIP11 was clearly to get a D package-retreiving system up and running with minimal effort. And that would be great if it were a good proposal. But DIP11 just smacks of corner-cutting.

I completely agree with this. -- /Jacob Carlborg
Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 03:49:56 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 09:41, Jonas Drewsen wrote:
 On 11/08/11 09.07, Jacob Carlborg wrote:
 On 2011-08-10 21:55, jdrewsen wrote:
 What is the status of DIP11

 http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works. /Jonas

I think that DIP11 is too limited, for example, it doesn't deal with versions. Orbit combined with a build tool will be seamless as well. RDMD is a great tool but as soon as you need to add compiler flags or compile a library you need either some kind of script or a build tool. And in that case you can just go with the built tool and have it work on all platforms.

Given that the implementation would be a compiler-used tool, and the tool can implement any protocol it wants, I think it has very few limitations. I envision the tool being able to handle any network protocol or packaging system we want it to. I think the benefit of this approach over a build tool which wraps the compiler is, the compiler already has the information needed for dependencies, etc. To a certain extent, the wrapping build tool has to re-implement some of the compiler pieces. -Steve
Aug 11 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I don't trust DIP11 to produce anything stable. We already have a
half-broken compiler, and now we're going to have a half-broken build
system to go along with it? No way.

I'd rather it be an external tool (built with D so we can actually
hack on it), that can be *downloaded from anywhere*, without having to
rely on dmars.com (which is painfully slow), and being encumbered by
licensing issues (you can't distribute DMD).
Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 10:47:36 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 14:52, Steven Schveighoffer wrote:
 Given that the implementation would be a compiler-used tool, and the
 tool can implement any protocol it wants, I think it has very few
 limitations. I envision the tool being able to handle any network
 protocol or packaging system we want it to.

That might be the case. Since it's arbitrary URLs that represents D modules and packages, to me it seems that there needs to be a lot of conventions: * Where to put the packages * How to name them * How to indicate a specific version

These are implementation details. The compiler just knows "hm.. there's some external file, I don't know how to get it, tool, can you help me out?" The tool could potentially download the files and build them, or download a pre-compiled package. Or it could add files to the compiler's todo list somehow (having recursive compiler invocations might be bad...)
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc. To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

 -Steve

Until the compiler can automatically compile dependencies we need build tools.

I agree it would be advantageous to have the build tool add files to the compiler's todo list. As of now, just downloading source for import does not help because you still need to compile the file, not just import it. But you have to admit, just having the source file or include path include parts of the internet would be pretty cool. I really like that approach. Its sort of like how dsss used to work, except dsss contained a huge portion of the compiler in it. At least with DIP11, the compiler is the driver, no need to maintain a separate "compiler". It could be that DIP11 needs more work to get a technically useful and implementable feature.
 What about linking with pre-compiled libraries, how would that work?  
 Currently the linking paths needs to be known before the compiler is  
 invoked. You would first need to compile without linking and then link,  
 or something like that. Assuming the compiler isn't changed to be able  
 to receive linker options form the external tool.

I'm assuming the linker path in dmd.conf would include some global or user-specific cache directory where pre-compiled libraries are downloaded. Then the download tool puts the libraries in the right spot, so the path does not need adjusting.
 Note that I'm basing all this on what's written in the DIP (and what  
 you've said), as far as I know that's the current suggestion. But the  
 DIP can of course be enhanced and updated.

Yes, and much better than a DIP is a working solution, so it might very well be that Orbit wins :) I certainly don't have the time or expertise to implement it... -Steve
Aug 11 2011
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--001636ed6ca87c175604aa3d3d21
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
<schveiguy yahoo.com>wrote:

 On Thu, 11 Aug 2011 03:49:56 -0400, Jacob Carlborg <doob me.com> wrote:

  On 2011-08-11 09:41, Jonas Drewsen wrote:
 On 11/08/11 09.07, Jacob Carlborg wrote:

 On 2011-08-10 21:55, jdrewsen wrote:

 What is the status of DIP11

 http://www.wikiservice.at/d/**wiki.cgi?LanguageDevel/DIPs/**DIP11<http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11>

 Has anyone started implementing it? Has it been rejected?

 /Jonas

Not sure, personally I don't like it. Instead I'm working on a more traditional package manager called Orbit: https://github.com/jacob-**carlborg/orbit/wiki/Orbit-** Package-Manager-for-D<https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D>

Yes I've noticed that. Seems very promising. What I do like about DIP11 is how seamless it would work. You just have to compile and stuff works. /Jonas

I think that DIP11 is too limited, for example, it doesn't deal with versions. Orbit combined with a build tool will be seamless as well. RDMD is a great tool but as soon as you need to add compiler flags or compile a library you need either some kind of script or a build tool. And in that case you can just go with the built tool and have it work on all platforms.

Given that the implementation would be a compiler-used tool, and the tool can implement any protocol it wants, I think it has very few limitations. I envision the tool being able to handle any network protocol or packaging system we want it to. I think the benefit of this approach over a build tool which wraps the compiler is, the compiler already has the information needed for dependencies, etc. To a certain extent, the wrapping build tool has to re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done. --001636ed6ca87c175604aa3d3d21 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveigh= offer <span dir=3D"ltr">&lt;<a href=3D"mailto:schveiguy yahoo.com">schveigu= y yahoo.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" styl= e=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <div><div></div><div class=3D"h5">On Thu, 11 Aug 2011 03:49:56 -0400, Jacob= Carlborg &lt;<a href=3D"mailto:doob me.com" target=3D"_blank">doob me.com<= /a>&gt; wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On 2011-08-11 09:41, Jonas Drewsen wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On 11/08/11 09.07, Jacob Carlborg wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On 2011-08-10 21:55, jdrewsen wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> What is the status of DIP11<br> <br> <a href=3D"http://www.wikiservice.at/d/wiki.cgi?LanguageDevel/DIPs/DIP11" t= arget=3D"_blank">http://www.wikiservice.at/d/<u></u>wiki.cgi?LanguageDevel/= DIPs/<u></u>DIP11</a><br> <br> Has anyone started implementing it? Has it been rejected?<br> <br> /Jonas<br> </blockquote> <br> Not sure, personally I don&#39;t like it. Instead I&#39;m working on a more= <br> traditional package manager called Orbit:<br> <a href=3D"https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manag= er-for-D" target=3D"_blank">https://github.com/jacob-<u></u>carlborg/orbit/= wiki/Orbit-<u></u>Package-Manager-for-D</a><br> </blockquote> <br> Yes I&#39;ve noticed that. Seems very promising.<br> <br> What I do like about DIP11 is how seamless it would work. You just have<br> to compile and stuff works.<br> <br> /Jonas<br> </blockquote> <br> I think that DIP11 is too limited, for example, it doesn&#39;t deal with ve= rsions. Orbit combined with a build tool will be seamless as well. RDMD is = a great tool but as soon as you need to add compiler flags or compile a lib= rary you need either some kind of script or a build tool. And in that case = you can just go with the built tool and have it work on all platforms.<br> </blockquote> <br></div></div> Given that the implementation would be a compiler-used tool, and the tool c= an implement any protocol it wants, I think it has very few limitations. = =A0I envision the tool being able to handle any network protocol or packagi= ng system we want it to.<br> <br> I think the benefit of this approach over a build tool which wraps the comp= iler is, the compiler already has the information needed for dependencies, = etc. =A0To a certain extent, the wrapping build tool has to re-implement so= me of the compiler pieces.<br> </blockquote><div><br></div><div>This last bit doesn&#39;t really come into= play here because you can already ask the compiler to output all that info= rmation. and easily use it in a separate program. That much is already done= .=A0</div> </div><br> --001636ed6ca87c175604aa3d3d21--
Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley  
<wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc.  To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile. And there is no parsing of the output data, the problem boils down to a simple get tool. Running a simple get tool over and over doesn't consume as much time/resources as running the compiler over and over. There are still problems with the DIP -- there is no way yet to say "oh yeah, compiler, you have to build this file that I downloaded too". But if nothing else, I like the approach of having the compiler drive everything. It reduces the problem space to a smaller more focused task -- get a file based on a url. We also already have many tools in existence that can parse a url and download a file/package. -Steve
Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 14:19:35 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 19:07, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley
 <wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc. To a certain extent, the wrapping build tool has to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

So how would that be different if the compiler drives everything? Say you begin with a few local files. The compiler then scans through them looking for URL imports. Then asks a tool to download the dependencies it found and starts all over again.

Forgive my compiler ignorance (not a compiler writer), but why does the compiler have to start over? It's no different than importing a file, is it?
 This is how my package manager will work. You have a local file  
 containing all the direct dependencies needed to build your project.  
 When invoked, the package manager tool fetches a file containing all  
 packages and all their dependencies, from the repository. It then  
 figures out all dependencies, both direct and indirect. Then it  
 downloads all dependencies. It does all this before the compiler is even  
 invoked once.

 Then, preferably, but optionally, it hands over to a build tool that  
 builds everything. The build tool would need to invoke the compiler  
 twice, first to get all the dependencies of all the local files in the  
 project that is being built. Then it finally runs the compiler to build  
 everything.

The benefit of using source is the source code is already written with an import statement, there is no need to write an external build file (all you need is command line that configures the compiler). Essentially, the import statements become your "build file". I think dsss worked like this, but I don't remember completely. My ideal solution, no matter how it's implemented is, I get a file blah.d, and I do: xyz blah.d and xyz handles all the dirty work of figuring out what to build along with blah.d as well as where to get those resources. Whether xyz == dmd, I don't know. It sure sounds like it could be... -Steve
Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 15:09:15 -0400, Nick Sabalausky <a a.a> wrote:

 "Steven Schveighoffer" <schveiguy yahoo.com> wrote in message
 news:op.vz166yaieav7ka localhost.localdomain...
 On Thu, 11 Aug 2011 12:24:48 -0400, Andrew Wiley
 <wiley.andrew.j gmail.com> wrote:

 On Thu, Aug 11, 2011 at 5:52 AM, Steven Schveighoffer
 <schveiguy yahoo.com>wrote:
 I think the benefit of this approach over a build tool which wraps the
 compiler is, the compiler already has the information needed for
 dependencies, etc.  To a certain extent, the wrapping build tool has  
 to
 re-implement some of the compiler pieces.

This last bit doesn't really come into play here because you can already ask the compiler to output all that information. and easily use it in a separate program. That much is already done.

Yes, but then you have to restart the compiler to figure out what's next. Let's say a source file needs a.d, and a.d needs b.d, and both a.d and b.d are on the network. You potentially need to run the compiler 3 times just to make sure you have all the files, then run it a fourth time to compile.

That's *only* true if you go along with DIP11's misguided file-oriented approach. With a real package manager, none of that is needed. Your app just says "I need packages X, Y and Z."

It already does this: import std.stdio; says "I need the package std" The purpose of the DIP is to try and reuse this information without having to have an extraneous file that says "I depend on package std."
 And X, Y and Z do the same for their requirements. This is all trivial
 metadata. Emphasis on *trivial*. So, before DMD is ever invoked at all,
 before one line of the source is ever even read, the package manager has
 make sure that *everything* is *already* right there. No need to go off  
 on
 some goofy half-cocked "compile/download/complile/download" dance.

With the DIP, I envision encoding the package in the URL. For example, you do: -Ispiffy=dpm://spiffy.com/latest.dpm And then blah.d imports spiffy.neat. The download tool is given the url dpm://spiffy.com/latest.dpm/neat.d and understanding the d package module (dpm) protocol, downloads the package that contains neat.d, (along with compiled lib) and then pipes the contents of neat.d to the compiler, which treats it just like a file it just read from the filesystem. Then any other file needed from that package is simply piped from the already-downloaded package. It doesn't have to be one-file based, you just need to have a way to map physical packages to module packages. The DIP doesn't explain all this except for the sections titled "Packaging" and "protocols"
 So DMD *never* needs to be invoked more than twice. Once to get the deps,
 once to compile. Better yet, if DMD gets the
 switch  
 --compile-everything-dammit-not-just-the-explicit-files-from-the-command-line,
 then DMD never needs to be invoked more than *once*: Once to figure out  
 the
 deps *while* being intelligent enough to actually compile all of them.

This would be beneficial to DIP11 as well, since downloading and importing the file is only half the battle. -Steve
Aug 11 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 11 Aug 2011 17:20:04 -0400, Nick Sabalausky <a a.a> wrote:

 "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message
 news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have been  
 yet
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.

With either style, you need to download a package in order to determine if you need to download other packages (package a may depend on package b even though your project does not depend on package b). The DIP11 version does this JIT, whereas your version does it before compilation. It's not really any different. -Steve
Aug 11 2011
next sibling parent reply kennytm <kennytm gmail.com> writes:
"Steven Schveighoffer" <schveiguy yahoo.com> wrote:
 On Thu, 11 Aug 2011 17:20:04 -0400, Nick Sabalausky <a a.a> wrote:
 
 "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in
 message
 news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have
 been  >> yet
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.

With either style, you need to download a package in order to determine if you need to download other packages (package a may depend on package b even though your project does not depend on package b). The DIP11 version does this JIT, whereas your version does it before compilation. It's not really any different. -Steve

In Debian's apt, there will be a central index that records all dependencies for packages in the repository. So the client only needs to synchronize that index file regularly. The system will know package A depends on B which depends on C and D and download all 4 packages. That said, since you need to download the pakcgaes anyway, having a central index doesn't reduce the bytes you need to transfer and parse if DIP11 doesn't support updating.
Aug 11 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-12 08:17, kennytm wrote:
 "Steven Schveighoffer"<schveiguy yahoo.com>  wrote:
 On Thu, 11 Aug 2011 17:20:04 -0400, Nick Sabalausky<a a.a>  wrote:

 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in
 message
 news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have
 been>>  yet
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.

With either style, you need to download a package in order to determine if you need to download other packages (package a may depend on package b even though your project does not depend on package b). The DIP11 version does this JIT, whereas your version does it before compilation. It's not really any different. -Steve

In Debian's apt, there will be a central index that records all dependencies for packages in the repository. So the client only needs to synchronize that index file regularly. The system will know package A depends on B which depends on C and D and download all 4 packages.

Exactly
 That said, since you need to download the pakcgaes anyway, having a
 central index doesn't reduce the bytes you need to transfer and parse if
 DIP11 doesn't support updating.

No, but I'm guessing it's more efficient to download zip files instead of individual D files. -- /Jacob Carlborg
Aug 12 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 02:17:36 -0400, kennytm <kennytm gmail.com> wrote:

 "Steven Schveighoffer" <schveiguy yahoo.com> wrote:
 On Thu, 11 Aug 2011 17:20:04 -0400, Nick Sabalausky <a a.a> wrote:

 "Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in
 message
 news:j21g1a$ea4$1 digitalmars.com...
 It's difficult to get all dependencies when not all sources have
 been  >> yet
 downloaded.

With DIP11, yes. With a traditional-style package manager, no.

With either style, you need to download a package in order to determine if you need to download other packages (package a may depend on package b even though your project does not depend on package b). The DIP11 version does this JIT, whereas your version does it before compilation. It's not really any different. -Steve

In Debian's apt, there will be a central index that records all dependencies for packages in the repository. So the client only needs to synchronize that index file regularly. The system will know package A depends on B which depends on C and D and download all 4 packages.

This could be done (in fact, the tool could have a url system that uses apt). But it's beyond the scope of the DIP, which is to define a way to hook the compiler for a downloading tool when a file is on the Internet.
 That said, since you need to download the pakcgaes anyway, having a
 central index doesn't reduce the bytes you need to transfer and parse if
 DIP11 doesn't support updating.

All the DIP does is provide a hook so the compiler can ask an external tool to download files/packages/whatever. It leaves open the protocol used by the tool, except that it should be in url form. The tool can implement the download any way it wants (individual files, packages, full-blown metadata-based centralized-indexed kitchen-sinked github or apt or whatever). -Steve
Aug 12 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
It really looks sound to me.

---
module myfile;
pragma(imppath, "dep=www.dpan.org/dep");
import dep.a;
---

remote file
---
module dep.a;

// link directives
pragma(libpath, "dep=www.dpan.org/dep");
pragma(lib, "dep");

// or alternatively some new pragma like this to cause linking the  
imported package
pragma(build, "dep")

// additional dependency
pragma(imppatch, "dep2=www.dpan.org/dep2");
---

Versioning can be easily resolved by using urls like  
www.dpan.org/dep?version=0.45 or www.dpan.org/dep-greaterthan-0.45.

Pro:
  - scales from local dependencies to VPN sharing to package websites to  
even dynamically generated source code
  - simple implementation compilerwise
Con:
  - doesn't help with mixed source builds except for prebuild libraries


Some thoughts:
  - allowing packages/libs to be packed in zip files would be nice as it's  
getting closer to single file packages
  - remote locations would need an accompanying hash file per source or a  
checksum index to allow local caching
  - sorting out some security issues would be nice but I've never seen  
anything in other package managers

On Thu, 11 Aug 2011 09:49:56 +0200, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 09:41, Jonas Drewsen wrote:
 On 11/08/11 09.07, Jacob Carlborg wrote:


 Yes I've noticed that. Seems very promising.

 What I do like about DIP11 is how seamless it would work. You just have
 to compile and stuff works.

 /Jonas

I think that DIP11 is too limited, for example, it doesn't deal with versions. Orbit combined with a build tool will be seamless as well. RDMD is a great tool but as soon as you need to add compiler flags or compile a library you need either some kind of script or a build tool. And in that case you can just go with the built tool and have it work on all platforms.

Aug 11 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 03:23:30 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 20:31, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 14:19:35 -0400, Jacob Carlborg <doob me.com> wrote:
 So how would that be different if the compiler drives everything? Say
 you begin with a few local files. The compiler then scans through them
 looking for URL imports. Then asks a tool to download the dependencies
 it found and starts all over again.

Forgive my compiler ignorance (not a compiler writer), but why does the compiler have to start over? It's no different than importing a file, is it?

Probably it's no different. Well what's different is it will first parse a couple of files, then downloads a few files, then parse the downloaded files and the fetch some more files and so on. To me this seems inefficient but since it's not implemented I don't know. It feels more efficient if it could download all needed files in one step. And then compile all files in the next step. I don't know what's possible with this DIP but to me it seems that the current suggestion will download individual files. This also seems inefficient, my solution deals with packages, i.e. zip files.

The extendability is in the url. For example, yes, http://server/file.d downloads a single file, and would be slow if every import needed to download an individual file. But some other protocol, or some other cue to the download tool (like if the download has a path component that ends in .tgz or something), then you download all the files at once, and on subsequent imports, the cached file is used (no download necessary). I think there's even a suggestion in there for passing directives back to the compiler like "file already downloaded, open it here..." I think unless a single file *is* the package, it's going to be foolish to download individual files. I also think a protocol which defines a central repository would be beneficial. So you only need one -I parameter to include a whole community of D code (like dsource).
 This is how my package manager will work. You have a local file
 containing all the direct dependencies needed to build your project.
 When invoked, the package manager tool fetches a file containing all
 packages and all their dependencies, from the repository. It then
 figures out all dependencies, both direct and indirect. Then it
 downloads all dependencies. It does all this before the compiler is
 even invoked once.

 Then, preferably, but optionally, it hands over to a build tool that
 builds everything. The build tool would need to invoke the compiler
 twice, first to get all the dependencies of all the local files in the
 project that is being built. Then it finally runs the compiler to
 build everything.

The benefit of using source is the source code is already written with an import statement, there is no need to write an external build file (all you need is command line that configures the compiler).

I don't see the big difference. I think the most project (not the smallest ones) will end up with a special file anyway, containing the pragmas declaring these imports.

Note that the pragmas are specific to that file only. So you don't have an import file which defines pragmas. This is to prevent conflicts between two files that declare the same package override.
 In addition to that as soon as you need to pass flags to the compiler  
 you will most likely to put that in a file of some kind. In that case  
 you can just as easily put them in a build script and use a build tool.

a batch files/shell script should suffice, no need for a "special" tool.
 Essentially, the import statements become your "build file". I think
 dsss worked like this, but I don't remember completely.

Yes, this is similar how DSSS worked. The difference is that you didn't need a pragma to link a package to an URL you just wrote the import declarations as you do now.

IIRC, dsss still had a global config file that defined where to import things from. The DIP defines that -I switches can also define internet resources along with pragmas, so sticking those in the dmd.conf would probably be the equivalent.
 One problem I think DSSS has, is, as far as I know, it can't handle top  
 level packages with same name. Or at least not in any good way. If you  
 go with the Java package naming scheme and name your top level package  
 after your domain, example:

 module com.foo.bar;
 module com.foo.foobar;

 And another project does the same:

 module com.abc.defg;

 Then these two projects will both end up in the "com" folder. Not very  
 good in my opinion. In my solution every package has a name independent  
 of the packages it contains an all packages are placed in a folder named  
 after the package, including the version number.

These all seem like implementation details. I don't care how the tool caches the files.
 My ideal solution, no matter how it's implemented is, I get a file
 blah.d, and I do:

 xyz blah.d

 and xyz handles all the dirty work of figuring out what to build along
 with blah.d as well as where to get those resources. Whether xyz == dmd,
 I don't know. It sure sounds like it could be...


 -Steve

Yeah, I would like that too. But as I said above, as soon as you need compiler flags you need an additional file. With a built tool it can then be just: "$ build"

Or instructions on the web site "use 'dmd -O -inline -release -version=SpecificVersion project.d' to compile" Or build.sh (build.bat) Note that dcollections has no makefile, everything is built from shell scripts. I almost never have to edit the build file, because the line's like: dmd -lib -O -release -inline dcollections/*.d dcollections/model/*.d Any new files get included automatically. And it takes a second to build, so who cares if you rebuild every file every time? Interestingly, libraries would still need to specify all the files since they may not import eachother :) I don't know if there's a "good" solution that isn't too coarse for that. All of this discussion is good to determine the viability, and clarify some misinterpretations of DIP11, but I think unless someone steps up and tries to implement it, it's a moot conversation. I certainly don't have the time or knowledge to implement it. So is there anyone who is interested, or has tried (to re-ask the original question)? -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 05:12:47 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 00:42, Martin Nowak wrote:
 It really looks sound to me.

 ---
 module myfile;
 pragma(imppath, "dep=www.dpan.org/dep");
 import dep.a;
 ---

 remote file
 ---
 module dep.a;

 // link directives
 pragma(libpath, "dep=www.dpan.org/dep");
 pragma(lib, "dep");

 // or alternatively some new pragma like this to cause linking the
 imported package
 pragma(build, "dep")

 // additional dependency
 pragma(imppatch, "dep2=www.dpan.org/dep2");
 ---

 Versioning can be easily resolved by using urls like
 www.dpan.org/dep?version=0.45 or www.dpan.org/dep-greaterthan-0.45.

 Pro:
 - scales from local dependencies to VPN sharing to package websites to
 even dynamically generated source code
 - simple implementation compilerwise

We will be very dependent on Walter or anyone else that knows the compiler, which, as far as I know, are quite few. I'm not sure if anything _is_ simple in the compiler.

This is not true. The compiler implements *hooks* for a download tool. The download tool will be a separate process which turns urls (generated by the compiler) into source files. Once the hooks are implemented, the tool is independent, and we would be idiotic not to implement it in D. I think you may not have read the DIP fully, or it is not clear enough. -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 05:04:13 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 21:36, Steven Schveighoffer wrote:
 be one-file based, you just need to have a way to map
 physical packages to module packages.

 The DIP doesn't explain all this except for the sections titled
 "Packaging" and "protocols"

The DIP needs to explain this, is that the whole point?

The DIP focuses on the compiler changes needed + gives rudimentary functionality based on a simple protocol (http). I think the scope of work for creating the tool which does full-fledged packaging is way too much for the DIP, and probably would be counter productive to create a spec for it right now. The point of those two sections is to remind you not to think in terms of "this only downloads individual files via http", but to think about the future functinality that such a tool could provide once the compiler has been hooked. The details of the tool are purposely left open for design. -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 12:30:42 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:53, Steven Schveighoffer wrote:
 This is not true. The compiler implements *hooks* for a download tool.
 The download tool will be a separate process which turns urls (generated
 by the compiler) into source files. Once the hooks are implemented, the
 tool is independent, and we would be idiotic not to implement it in D.

 I think you may not have read the DIP fully, or it is not clear enough.

 -Steve

I've read the whole DIP and I know there is an external tool that downloads the files. I also know that DMD doesn't have these hooks, I rest my case.

I thought you meant we would be dependent on Walter to write the *download part* of the DIP, not the compiler hooks. The hooks should be pretty simple I would think. But in any case, you should have a look at github and all the new people who are working on pull requests for the compiler. The group of dmd code contributors has significantly grown. -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 14:24:46 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:49, Steven Schveighoffer wrote:
 Note that the pragmas are specific to that file only. So you don't have
 an import file which defines pragmas. This is to prevent conflicts
 between two files that declare the same package override.

Now I'm not quite sure I understand. Are you saying that every file needs to have these pragma imports ?

Let's say file a.d pragmas that module foo means http://foo.com/projectx, and module b.d from another project pragmas that module foo means http://bar.com/projecty. If I import both a and b, what happens? It only makes sense for a pragma to affect the current file. This is similar to how version=x statements only affect the current file.
 In addition to that as soon as you need to pass flags to the compiler
 you will most likely to put that in a file of some kind. In that case
 you can just as easily put them in a build script and use a build tool.

a batch files/shell script should suffice, no need for a "special" tool.

Yeah, but you would need to duplicate the shell script, one for Posix and one for Windows. As far as I know there is no scripting like file that both Windows and Posix can read out of the box.

But I don't have to have the user install yet other build tool. There already are script interpreters on both windows and posix systems. Especially for one-liners.
 One problem I think DSSS has, is, as far as I know, it can't handle
 top level packages with same name. Or at least not in any good way. If
 you go with the Java package naming scheme and name your top level
 package after your domain, example:

 module com.foo.bar;
 module com.foo.foobar;

 And another project does the same:

 module com.abc.defg;

 Then these two projects will both end up in the "com" folder. Not very
 good in my opinion. In my solution every package has a name
 independent of the packages it contains an all packages are placed in
 a folder named after the package, including the version number.

These all seem like implementation details. I don't care how the tool caches the files.

Well yes, but that is how DSSS works and that's what I'm explaining.

OK.
 Or instructions on the web site "use 'dmd -O -inline -release
 -version=SpecificVersion project.d' to compile"

 Or build.sh (build.bat) Note that dcollections has no makefile,
 everything is built from shell scripts. I almost never have to edit the
 build file, because the line's like:

You would need to shell script files, one for Windows and one for Posix, see above.

Yes.
 That's what a build tool can handle. I think you should read this:  
 http://dsource.org/projects/dsss/wiki/DSSSByExample and hopefully you'll  
 understand why a built tool is a good thing.

I'll take a look when I get a moment. -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 14:31:28 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 19:08, Steven Schveighoffer wrote:
 On Fri, 12 Aug 2011 12:30:42 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:53, Steven Schveighoffer wrote:
 This is not true. The compiler implements *hooks* for a download tool.
 The download tool will be a separate process which turns urls  
 (generated
 by the compiler) into source files. Once the hooks are implemented,  
 the
 tool is independent, and we would be idiotic not to implement it in D.

 I think you may not have read the DIP fully, or it is not clear  
 enough.

 -Steve

I've read the whole DIP and I know there is an external tool that downloads the files. I also know that DMD doesn't have these hooks, I rest my case.

I thought you meant we would be dependent on Walter to write the *download part* of the DIP, not the compiler hooks. The hooks should be pretty simple I would think. But in any case, you should have a look at github and all the new people who are working on pull requests for the compiler. The group of dmd code contributors has significantly grown. -Steve

Yeah, that's true and it's a good thing. I can tell you this, I've looked at the DMD source code and tried to do modifications but I couldn't understand much of the code at all and failed with the most simple things.

/me in same boat as you -Steve
Aug 12 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Aug 2011 15:12:07 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 20:36, Steven Schveighoffer wrote:
 On Fri, 12 Aug 2011 14:24:46 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 15:49, Steven Schveighoffer wrote:
 Note that the pragmas are specific to that file only. So you don't  
 have
 an import file which defines pragmas. This is to prevent conflicts
 between two files that declare the same package override.

Now I'm not quite sure I understand. Are you saying that every file needs to have these pragma imports ?

Let's say file a.d pragmas that module foo means http://foo.com/projectx, and module b.d from another project pragmas that module foo means http://bar.com/projecty. If I import both a and b, what happens? It only makes sense for a pragma to affect the current file. This is similar to how version=x statements only affect the current file.

Again, will that mean you have to specify a pragma for each file?

Yes, or specify it on the command line/config file. I think the risk of inadvertently importing incorrect files is too great. Looking back at it, however, we probably would need some mechanism for files in the same package to inherit the source location. For example, if you pragma that a module foo = http://foo.com/projectx, and import foo.xyz, and foo.xyz imports foo.abc, we don't want foo.xyz to have to pragma the same url just to include another file in its own package. Clearly we need some more thought around this.
 Just for the record, you cannot always solve all dependencies, it can  
 happen that two packages conflict with each other.

As long as the dependencies are contained, there should be no conflict. If I can compile project x and y separately, and both have a conflicting dependency, then I should still be able to compile a project that depends on both x and y, as long as they don't import eachother. -Steve
Aug 12 2011
prev sibling next sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Fri, 12 Aug 2011 15:49:30 +0200, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Fri, 12 Aug 2011 03:23:30 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-11 20:31, Steven Schveighoffer wrote:
 On Thu, 11 Aug 2011 14:19:35 -0400, Jacob Carlborg <doob me.com> wrote:
 So how would that be different if the compiler drives everything? Say
 you begin with a few local files. The compiler then scans through them
 looking for URL imports. Then asks a tool to download the dependencies
 it found and starts all over again.

Forgive my compiler ignorance (not a compiler writer), but why does the compiler have to start over? It's no different than importing a file, is it?

Probably it's no different. Well what's different is it will first parse a couple of files, then downloads a few files, then parse the downloaded files and the fetch some more files and so on. To me this seems inefficient but since it's not implemented I don't know. It feels more efficient if it could download all needed files in one step. And then compile all files in the next step. I don't know what's possible with this DIP but to me it seems that the current suggestion will download individual files. This also seems inefficient, my solution deals with packages, i.e. zip files.

The extendability is in the url. For example, yes, http://server/file.d downloads a single file, and would be slow if every import needed to download an individual file. But some other protocol, or some other cue to the download tool (like if the download has a path component that ends in .tgz or something), then you download all the files at once, and on subsequent imports, the cached file is used (no download necessary). I think there's even a suggestion in there for passing directives back to the compiler like "file already downloaded, open it here..." I think unless a single file *is* the package, it's going to be foolish to download individual files. I also think a protocol which defines a central repository would be beneficial. So you only need one -I parameter to include a whole community of D code (like dsource).
 This is how my package manager will work. You have a local file
 containing all the direct dependencies needed to build your project.
 When invoked, the package manager tool fetches a file containing all
 packages and all their dependencies, from the repository. It then
 figures out all dependencies, both direct and indirect. Then it
 downloads all dependencies. It does all this before the compiler is
 even invoked once.

 Then, preferably, but optionally, it hands over to a build tool that
 builds everything. The build tool would need to invoke the compiler
 twice, first to get all the dependencies of all the local files in the
 project that is being built. Then it finally runs the compiler to
 build everything.

The benefit of using source is the source code is already written with an import statement, there is no need to write an external build file (all you need is command line that configures the compiler).

I don't see the big difference. I think the most project (not the smallest ones) will end up with a special file anyway, containing the pragmas declaring these imports.

Note that the pragmas are specific to that file only. So you don't have an import file which defines pragmas. This is to prevent conflicts between two files that declare the same package override.
 In addition to that as soon as you need to pass flags to the compiler  
 you will most likely to put that in a file of some kind. In that case  
 you can just as easily put them in a build script and use a build tool.

a batch files/shell script should suffice, no need for a "special" tool.
 Essentially, the import statements become your "build file". I think
 dsss worked like this, but I don't remember completely.

Yes, this is similar how DSSS worked. The difference is that you didn't need a pragma to link a package to an URL you just wrote the import declarations as you do now.

IIRC, dsss still had a global config file that defined where to import things from. The DIP defines that -I switches can also define internet resources along with pragmas, so sticking those in the dmd.conf would probably be the equivalent.
 One problem I think DSSS has, is, as far as I know, it can't handle top  
 level packages with same name. Or at least not in any good way. If you  
 go with the Java package naming scheme and name your top level package  
 after your domain, example:

 module com.foo.bar;
 module com.foo.foobar;

 And another project does the same:

 module com.abc.defg;

 Then these two projects will both end up in the "com" folder. Not very  
 good in my opinion. In my solution every package has a name independent  
 of the packages it contains an all packages are placed in a folder  
 named after the package, including the version number.

These all seem like implementation details. I don't care how the tool caches the files.
 My ideal solution, no matter how it's implemented is, I get a file
 blah.d, and I do:

 xyz blah.d

 and xyz handles all the dirty work of figuring out what to build along
 with blah.d as well as where to get those resources. Whether xyz ==  
 dmd,
 I don't know. It sure sounds like it could be...


 -Steve

Yeah, I would like that too. But as I said above, as soon as you need compiler flags you need an additional file. With a built tool it can then be just: "$ build"

Or instructions on the web site "use 'dmd -O -inline -release -version=SpecificVersion project.d' to compile" Or build.sh (build.bat) Note that dcollections has no makefile, everything is built from shell scripts. I almost never have to edit the build file, because the line's like: dmd -lib -O -release -inline dcollections/*.d dcollections/model/*.d Any new files get included automatically. And it takes a second to build, so who cares if you rebuild every file every time? Interestingly, libraries would still need to specify all the files since they may not import eachother :) I don't know if there's a "good" solution that isn't too coarse for that. All of this discussion is good to determine the viability, and clarify some misinterpretations of DIP11, but I think unless someone steps up and tries to implement it, it's a moot conversation. I certainly don't have the time or knowledge to implement it. So is there anyone who is interested, or has tried (to re-ask the original question)? -Steve

Well, I would give it a try to implement a prototype. Given that we can sort out how to handle the actual building of remote sources. As long we as are only talking about imports this remains not so useful. Some approaches I could think of: I. make 'dmd mysource.d acme.a=http://acme.org/a -Iacme.b=http://acme.org/b' mean build every source that you download from acme.a II. for every import that is actually a source file (*.d) let the compiler decide if linking will be needed, if so build an object for that module III. specify a good naming scheme for distributing binary libraries Actually I only like the second solution. So instead of 'dmd a.d b.d' one could would write 'dmd -build-deps a.d' where a importing b. martin
Aug 12 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 13 Aug 2011 08:24:53 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-12 21:22, Steven Schveighoffer wrote:

 Yes, or specify it on the command line/config file. I think the risk of
 inadvertently importing incorrect files is too great.

 Looking back at it, however, we probably would need some mechanism for
 files in the same package to inherit the source location.

 For example, if you pragma that a module foo = http://foo.com/projectx,
 and import foo.xyz, and foo.xyz imports foo.abc, we don't want foo.xyz
 to have to pragma the same url just to include another file in its own
 package.

 Clearly we need some more thought around this.

Yeah, that's what I was afraid of.
 Just for the record, you cannot always solve all dependencies, it can
 happen that two packages conflict with each other.

As long as the dependencies are contained, there should be no conflict. If I can compile project x and y separately, and both have a conflicting dependency, then I should still be able to compile a project that depends on both x and y, as long as they don't import eachother. -Steve

If x and y depends on two different versions of z, how would that be solved. As far as I know you cannot link the same library of two different version twice, you will get conflicting symbols.

It wouldn't be an actual conflict, it would be a naming issue. In other words, I'm not talking about two different versions of the same library. That has to be a compiler error. What could happen though, is this situation: a.d pragma's foo.bar as being http://somedreposiotry.com/foo/barv1.0 b.d includes the repository path http://somedrepository.com, whose default foo.bar goes to foo/barv2.0, but b does not depend on foo.bar I don't think a.d's pragma should include b's pragma, or vice versa. Especially if it is some sort of weird import order precedent. It's just much simpler to say "b's pragma only affects b, and a's pragma only affects a." However, I think we need to add that any files imported via a pragma implicitly include that path. I should probably add that to the DIP. Obviously if a.d depends on foo.bar, and b.d depends on foo.bar, but the locations are different, it should be a compiler error. -Steve
Aug 15 2011