www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Dub integrated into the compiler?

reply "Tofu Ninja" <emmons0 purdue.edu> writes:
I don't think this is a new idea but it would be pretty awesome.

So the idea is that the compiler could check dub for libraries 
that it can't find and automatically download and integrate them. 
It would essentially make all of dub seem like it was a part of 
the standard lib and make integrating dub into projects seamless.

Something like "import derelict.opengl3.gl3;" could automatically 
trigger dub if the import was not found on any local path.

It also might need some special syntax for version specifications.

It would require some work on both sides. Obviously the compiler 
would need to change but dub would have to as well, I don't think 
currently there is any way to look up a dub package by an import 
statement alone.



Pros:
Greatly increase ease of dub use.
Greatly increase libraries that "seem" to be a part of the std 
lib.
Would solve the large vs small stdlib dilemma.

Cons:
Could degrade perceived quality of stdlib if bad dub packages got 
in.
Would mean some things could not be compiled without an internet 
connection.

My answers to those cons would be

1)
It might require some policing to make sure quality stays high.
Maybe even with votes on what dub packages could automatically be 
picked up by this similar to how packages get into phobos but 
with a much lower bar.

Also maybe a different import syntax for these imports to clearly 
differentiate them so that we can draw a line between the 
standard lib and dub. (personally I don't like that idea, I like 
the idea of seamless integration)

2)
Normal code would still compile like normal, this would only open 
up the option of automatic library download for basically the 99% 
of people who always have an internet connection.




So yeah...
Thoughts?
Oct 26 2014
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 27/10/2014 3:33 p.m., Tofu Ninja wrote:
 I don't think this is a new idea but it would be pretty awesome.

 So the idea is that the compiler could check dub for libraries that it
 can't find and automatically download and integrate them. It would
 essentially make all of dub seem like it was a part of the standard lib
 and make integrating dub into projects seamless.

 Something like "import derelict.opengl3.gl3;" could automatically
 trigger dub if the import was not found on any local path.

 It also might need some special syntax for version specifications.

 It would require some work on both sides. Obviously the compiler would
 need to change but dub would have to as well, I don't think currently
 there is any way to look up a dub package by an import statement alone.



 Pros:
 Greatly increase ease of dub use.
 Greatly increase libraries that "seem" to be a part of the std lib.
 Would solve the large vs small stdlib dilemma.

 Cons:
 Could degrade perceived quality of stdlib if bad dub packages got in.
 Would mean some things could not be compiled without an internet
 connection.

 My answers to those cons would be

 1)
 It might require some policing to make sure quality stays high.
 Maybe even with votes on what dub packages could automatically be picked
 up by this similar to how packages get into phobos but with a much lower
 bar.

 Also maybe a different import syntax for these imports to clearly
 differentiate them so that we can draw a line between the standard lib
 and dub. (personally I don't like that idea, I like the idea of seamless
 integration)

 2)
 Normal code would still compile like normal, this would only open up the
 option of automatic library download for basically the 99% of people who
 always have an internet connection.




 So yeah...
 Thoughts?
Just an idea, but we could take a leaf out of lua's book in this case. Lua has the require statement. Which includes external scripts and is highly configurable in lua code. With skeleton[0] I have my repository/file downloaded hooked into require. It uses a very specific format. E.g. skeleton rikkimax means a repository for username rikkimax, but it can go one step further skeleton rikkimax/file.d for a singular file. There's more to it then just that but small recap. For singular files import could be used quite well for this format. import repo user/foo.bar; For that module the file loader handler within dmd could be switched out to that repository. But that's not really dub support. So next step: import repo user/p1/foo.bar; Repository for user, in directory p1 with foo.bar module. Location of dub library is in p1. That's a little better. But it doesn't explain how to get support between dmd and dub. First the reading of files (either passed in via cli or via import's) would have to be abstracted out in dmd. Basically who handles the import statement. Next dub would need a new compiler support which would act as a callback instead of directly running dmd. Except when compiling dependencies. It would also need to use the existing settings of the host dmd call (to emulate as if dub was doing dependency compilation). Okay now this is getting a little scary for feature set of dmd not to mention dmd requiring D program gah yeah no. So, shared library plugins for dmd to enable this as an optional feature? Sounds like a good idea. Of course we can go one step further and add in support for e.g. gists, pastebin, dpaste and of course plain text files stored on a web server. Lastly, versions. version(`>=0.2.1`) import repo user; [0] https://github.com/rikkimax/skeleton
Oct 26 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
i think it would be better to develop something like "universal
interface" for this. so we can configure compiler to exec external
program which does all the things. this way it wouldn't be tied to dub.

by the way, i believe that something like this can be done with
external wrapper like rdmd. dubdmd, for example, which analyses imports
the same way rdmd does and invokes dub for all missing libraries, and
then invokes dmd with necessary options added.
Oct 26 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 27 October 2014 at 02:58:15 UTC, ketmar via 
Digitalmars-d wrote:
 i think it would be better to develop something like "universal
 interface" for this. so we can configure compiler to exec 
 external
 program which does all the things. this way it wouldn't be tied 
 to dub.
The universal interface to such a things could simply be an exe that gets called when an import is not found asking for the path of the import. The exe can do all sorts of things(like downloading from dub) before it finally returns the path to the import. It would need someway to pass extra information to it though(like version specifiers) because I think it wouldn't work if it was embedded in the import statement it self(i mean as part of the import path which would not match with the actual module name).
 by the way, i believe that something like this can be done with
 external wrapper like rdmd. dubdmd, for example, which analyses 
 imports
 the same way rdmd does and invokes dub for all missing 
 libraries, and
 then invokes dmd with necessary options added.
It probably could, I didn't even think about that. But I think you're "universal" interface idea is a bit better and it seems like something that could very easily be added to rdmd or dmd itself in a jiffy.
Oct 26 2014
next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 27 Oct 2014 03:34:34 +0000
Tofu Ninja via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 It probably could, I didn't even think about that. But I think=20
 you're "universal" interface idea is a bit better and it seems=20
 like something that could very easily be added to rdmd or dmd=20
 itself in a jiffy.
i'm dreaming about having this as CTFE interface. one, for example, could write special module like 'core.import_resolver', DMD will load and parse it when it encounters missing import, and then evaluate some of it's well-defined API funcions that can prepare command line for the necessary tool or even execute that tool and parse it's output. sure, this is not safe and not secure, so such such things as "allow import resolver to execute external programs" must be explicitly enabled with some CLI switch.
Oct 26 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 27 October 2014 at 03:42:08 UTC, ketmar via 
Digitalmars-d wrote:
 i'm dreaming about having this as CTFE interface. one, for 
 example,
 could write special module like 'core.import_resolver', DMD 
 will load
 and parse it when it encounters missing import, and then 
 evaluate some
 of it's well-defined API funcions that can prepare command line 
 for the
 necessary tool or even execute that tool and parse it's output.

 sure, this is not safe and not secure, so such such things as 
 "allow
 import resolver to execute external programs" must be explicitly
 enabled with some CLI switch.
I have brought up the idea of executing external exe's at compile time and capturing the output a few times before. Specifically for the use you just mentioned. But every time I suggest it, it gets shot down with concerns over security.
Oct 26 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 27 Oct 2014 03:55:50 +0000
Tofu Ninja via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 I have brought up the idea of executing external exe's at compile=20
 time and capturing the output a few times before. Specifically=20
 for the use you just mentioned. But every time I suggest it, it=20
 gets shot down with concerns over security.
at least i can still summon OOM-killer with CTFE code and hope that it will kill something completely unrelated to the compiler. still fun! ;-)
Oct 26 2014
prev sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 27 Oct 2014 03:34:34 +0000
Tofu Ninja via Digitalmars-d <digitalmars-d puremagic.com> wrote:

p.s. no, really, we have a VERY powerful scripting language built into
DMD itself! why we should resort to plugin libraries or so?

i know that it's a mad idea. well, that's 'cause i'm a madman! ;-)
Oct 26 2014
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
 Cons:
 Could degrade perceived quality of stdlib if bad dub packages 
 got in.
Bad for security.
Oct 26 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 27 Oct 2014 03:00:49 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
 Cons:
 Could degrade perceived quality of stdlib if bad dub packages=20
 got in.
=20 Bad for security.
this feature can be made explicit opt-in with proper message from the compiler.
Oct 26 2014
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 27 October 2014 at 03:00:50 UTC, Ola Fosheim Grøstad 
wrote:
 On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
 Cons:
 Could degrade perceived quality of stdlib if bad dub packages 
 got in.
Bad for security.
My response to that is that any library you ever download is bad for security (including dmd and phobos). We need to draw the line somewhere for things we trust and things we don't trust, personally I draw the line where it best suits me to get things done. If giving up some small about of security allowed me to automatically integrate dub packages into my projects, I would happily give it up. :) Also it is why I suggested that it could be policed.
Oct 26 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 27 October 2014 at 03:15:45 UTC, Tofu Ninja wrote:
 On Monday, 27 October 2014 at 03:00:50 UTC, Ola Fosheim Grøstad
 Bad for security.
My response to that is that any library you ever download is bad for security (including dmd and phobos).
I currently run dmd on a separate user account…
 We need to draw the line somewhere for things we trust and 
 things we don't trust, personally I draw the line where it best 
 suits me to get things done. If giving up some small about of 
 security allowed me to automatically integrate dub packages 
 into my projects, I would happily give it up. :)
That's ok for a personal user account, but not for a work account IMO. Then again, I prefer to fetch directly from repos manually and only use dub-like features for languages that run in a VM. Another point is that if you make fetching libraries too easy it means bloat starts creeping in. OK for a scripting language, but for a system level language…? You risk ending up with Tango-bloat, where you cannot include anything without pulling inn all kinds of dependencies.
 Also it is why I suggested that it could be policed.
But the D community is too small for that atm.
Oct 26 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 27 Oct 2014 03:24:46 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Another point is that if you make fetching libraries too easy it=20
 means bloat starts creeping in. OK for a scripting language, but=20
 for a system level language=E2=80=A6?
D is not only system-level language. D's metaprogramming features makes it very high-level too. writing CLI scripts with D and rdmd is a joy. and i never was a big fan of writing CLI scripts in C, despite the TCC's ability to to this.
Oct 26 2014
prev sibling next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 27 October 2014 at 03:24:47 UTC, Ola Fosheim Grøstad 
wrote:
 That's ok for a personal user account, but not for a work 
 account IMO.

 Then again, I prefer to fetch directly from repos manually and 
 only use dub-like features for languages that run in a VM.
What I am saying is that unless you read every line of all the libraries that you want to use and de-compile every precompiled library and read the asm, you are taking a risk, any of that code could do malicious things. You draw a line.
 Another point is that if you make fetching libraries too easy 
 it means bloat starts creeping in. OK for a scripting language, 
 but for a system level language…? You risk ending up with 
 Tango-bloat, where you cannot include anything without pulling 
 inn all kinds of dependencies.
The whole point of the thing is to get the benefits of a large library without having a bloated standard lib. Obviously none of phobos would depend on anything in dub. But users need things out side of phobos, why would we want to make getting that harder?
Oct 26 2014
prev sibling parent "Laeeth Isharc" <laeethnospam spammenot_laeeth.com> writes:
 Also it is why I suggested that it could be policed.
But the D community is too small for that atm.
which means that it is easy to have a concept of relatively trusted vs unknown contributors. of course if someone trusted gets hacked or socially engineered then that is a risk, but on the other hand that is true of many other library repositories and there are probably juicier targets.
Nov 07 2014
prev sibling next sibling parent reply "Kyoji Klyden" <trampzy yahoo.com> writes:
On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
 I don't think this is a new idea but it would be pretty awesome.

 So the idea is that the compiler could check dub for libraries 
 that it can't find and automatically download and integrate 
 them. It would essentially make all of dub seem like it was a 
 part of the standard lib and make integrating dub into projects 
 seamless.

 Something like "import derelict.opengl3.gl3;" could 
 automatically trigger dub if the import was not found on any 
 local path.

 It also might need some special syntax for version 
 specifications.

 It would require some work on both sides. Obviously the 
 compiler would need to change but dub would have to as well, I 
 don't think currently there is any way to look up a dub package 
 by an import statement alone.



 Pros:
 Greatly increase ease of dub use.
 Greatly increase libraries that "seem" to be a part of the std 
 lib.
 Would solve the large vs small stdlib dilemma.

 Cons:
 Could degrade perceived quality of stdlib if bad dub packages 
 got in.
 Would mean some things could not be compiled without an 
 internet connection.

 My answers to those cons would be

 1)
 It might require some policing to make sure quality stays high.
 Maybe even with votes on what dub packages could automatically 
 be picked up by this similar to how packages get into phobos 
 but with a much lower bar.

 Also maybe a different import syntax for these imports to 
 clearly differentiate them so that we can draw a line between 
 the standard lib and dub. (personally I don't like that idea, I 
 like the idea of seamless integration)

 2)
 Normal code would still compile like normal, this would only 
 open up the option of automatic library download for basically 
 the 99% of people who always have an internet connection.




 So yeah...
 Thoughts?
I'm against it. dub already tracks down libraries for you if need them, and dub already calls dmd for you. So all this seems like to me is calling dmd on the command line would just essentially be calling dub. I much rather have a clean distinction between the toolkit and the compiler. Anyways, automating things just for the sake of automating them isn't a very good idea. I'm mainly offset by the idea that my compiler is chatting away with some server in hopes of finding a missing library or file. Downloading whatever I need and properly calling it is cleaner, safer, and more organized. The idea sounds like in the end it would primarily just help hobbyist or someone looking to just quickly prototype something. So instead of fancying up/complicating dmd a better solution would be to improve dub in someway, make a new tool, or if we really have to do something to dmd then the solution that would make the most people the happiest would be to allow plugins to dmd or something, so if people want something like this then they can add it and then grumpy people like me won't be bothered by it ;P
Oct 26 2014
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 27 October 2014 at 04:34:41 UTC, Kyoji Klyden wrote:
 I'm against it. dub already tracks down libraries for you if 
 need
 them, and dub already calls dmd for you. So all this seems like
 to me is calling dmd on the command line would just essentially
 be calling dub.
 I much rather have a clean distinction between the toolkit and
 the compiler.
It isn't a big leap from what we have now, but personally I think it would help greatly with generating a larger user base by just making things easier.
 Anyways, automating things just for the sake of
 automating them isn't a very good idea.
We are programmers, isn't that what we do? :P
 I'm mainly offset by the
 idea that my compiler is chatting away with some server in hopes
 of finding a missing library or file. Downloading whatever I 
 need
 and properly calling it is cleaner, safer, and more organized.
I can definitely understand how some people don't like the idea of a compiler that connects to the internet but I don't feel that at all. I have had internet all my life so I am very used to all of my things being connected to the internet 100% of the time. Also I think it could definitely be argued that an automated(well more automated) solution would be cleaner and more organized.
 The idea sounds like in the end it would primarily just help
 hobbyist or someone looking to just quickly prototype something.
That is true. It would get people to play with D more, which is a good thing. Bring more people to D.
 So instead of fancying up/complicating dmd a better solution
 would be to improve dub in someway, make a new tool, or if we
 really have to do something to dmd then the solution that would
 make the most people the happiest would be to allow plugins to
 dmd or something, so if people want something like this then 
 they
 can add it and then grumpy people like me won't be bothered by 
 it
 ;P
I am not sure how dub it self could be changed to do what I proposed, but a new tool like what ketmar suggested or a plugin system could definitely be an option.
Oct 26 2014
next sibling parent Mike Parker <aldacron gmail.com> writes:
On 10/27/2014 3:40 PM, Tofu Ninja wrote:

 It isn't a big leap from what we have now, but personally I think it
 would help greatly with generating a larger user base by just making
 things easier.
I can't count the number of times over the years I have heard this as a reason to add feature X to software Y. In reality, it rarely works out that way. For my two cents, I don't see any benefit to integrating dub with dmd when invoking dub does what you want already. dub will eventually ship with dmd releases, so at that point it will be even easier to get going with than it is now without tying it directly to the compiler.
Oct 27 2014
prev sibling parent "Kyoji Klyden" <trampzy yahoo.com> writes:
On Monday, 27 October 2014 at 06:40:52 UTC, Tofu Ninja wrote:
...
I definitely get your perspective on this whole thing and can see the appeal to it all, but I feel like it really is just complicating things in ways it doesn't need to be complicated. It definitely would make things easier for new users, but for long-term/hardcore D user, it would just be a complete liability. Especially large teams or companies, wouldn't gain anything from this. So I guess what I'm trying to get to is this feature would benefit lone, new programmers/users, especially as a draw in, but nobody else has much at all to gain. I dont intend this to sound harsh, but if a new programmer can't handle a good old compiler, then they'd probably be better off using an interpreted language, such as python. (At least until they're ready for compilers). Also as Mike Parker just mentioned, dub will be shipping with dmd in the future, so if you really need to, just use dub in newbie tutorials or something like that while people get the hang of D. The only other way I could see this happening aside from a plugin system, and this suggestion is probably better than my last, is do a custom dmd build with whatever features you want in it. I'm happy that your proposed the idea and I love hearing propositions like this, but I'm gonna pass on this one.
Oct 27 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-10-27 03:33, Tofu Ninja wrote:
 I don't think this is a new idea but it would be pretty awesome.

 So the idea is that the compiler could check dub for libraries that it
 can't find and automatically download and integrate them. It would
 essentially make all of dub seem like it was a part of the standard lib
 and make integrating dub into projects seamless.
I don't really like the idea. I don't see the point of it, for a couple of reasons: * There are still other parts of the configuration that needs to live somewhere. Which compiler switches should be used and so on * I don't know exactly how Dub works but in theory it could ask the registry of the complete set of dependencies for a particular build and start to download all of them. With your suggestion the compiler might need to ask the registry many times to get all dependencies. First it sees "import foo" which it can't resolve and asks the registry about that. Then it needs to compile "foo" to find out if it has any dependencies as well. This applies recursively to all imports -- /Jacob Carlborg
Oct 27 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2014-10-27 at 02:33 +0000, Tofu Ninja via Digitalmars-d wrote:
[=E2=80=A6]
 Something like "import derelict.opengl3.gl3;" could automatically=20
 trigger dub if the import was not found on any local path.
[=E2=80=A6] Groovy has "grapes" for automated download of dependencies. Grab("org.spockframework:spock-core:0.7-groovy2.0") import spock.core.Specification The artefact is cached locally after first download, but you have to remember to run the script while connected to the Internet. Not that useful for projects where the Gradle script would manage the dependencies, but great for scripts. Go also allows for import of packages from elsewhere, and I rather like the way that is done, but I think that whole Go mechanism has been slagged off on this list a number of times already. Pity. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Oct 27 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
I has been proposed some time ago already and I am still strictly 
against it. It greatly reduces predictability of compiler 
invocations which causes problems for larger projects while 
slightly improving experience of scripters and hobbyists. Not a 
good trade-off.

It also can't work that way without imposing new restrictions on 
dub registry as currently it is legal to have same package names 
in different packages (which is a nice feature to keep).

At the same time convenience gain over simply using dub instead 
of dmd is almost neglectible.
Oct 27 2014
prev sibling parent "Jason den Dulk" <public2 jasondendulk.com> writes:
On Monday, 27 October 2014 at 02:33:18 UTC, Tofu Ninja wrote:
 I don't think this is a new idea but it would be pretty awesome.

 So the idea is that the compiler could check dub for libraries 
 that it can't find and automatically download and integrate 
 them. It would essentially make all of dub seem like it was a 
 part of the standard lib and make integrating dub into projects 
 seamless.
[snip]
 Cons:
 Could degrade perceived quality of stdlib if bad dub packages 
 got in.
 Would mean some things could not be compiled without an 
 internet connection.
Cons: - Breaks modularity (each thing should do one thing well). - Puts pressure on people to use Dub who would prefer not to. - Adds complexity to an already highly complex system. - Inhibits the development of a "better-than-Dub" alternative. - Inhibits the development of alternative compilers. Regards Jason
Nov 06 2014