www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [upforgrabs] tools/update

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We need a script to update all of dmd's paraphernalia. By this I'm 
putting the task up for grabs and am describing it.

The script would require git to be installed, and would orchestrate git 
commands to freshen the installation. Either a tag can be given, or the 
script would automatically detect the latest "semistable" release, or 
even the script can be told to get the latest and greatest. The exact 
tagging convention is to be defined later.

The trick with such tools is they must "just work". That means wherever 
and however you installed dmd, the script should be able to detect 
everything it needs for a successful update. This means: dmd and rdmd 
binary locations, druntime and phobos library locations, and druntime 
and phobos import locations.

The way I'm thinking this all can be done is by running dmd -v against a 
small do-nothing program that imports one std module, for example 
std.stdio. The run will print (among other things):

binary    dmd
version   v2.054
config    /Users/john/dmd.conf
parse     test
importall test
import    object	(/Users/john/code/dmd/druntime/import/object.di)
import    std.stdio	(/Users/john/code/dmd/phobos/std/stdio.d)
...

Simple parsing reveals the location of the configuration file and the 
locations of druntime and std imports. Further simple parsing of the 
conf file reveals the location of libraries.

With this information in hand, the script goes and fetches the 
appropriate code in the appropriate places.

Also, the tool needs to do all of its work transactionally, i.e. first 
download everything next to the target with an added suffix (e.g. 
".tmp"). Only after a full successful download, the tool would rename 
everything .tmp to its final name (an operation that can't fail under 
most circumstances).

If anyone is willing to take up working on this, it would be great. D 
itself is a great candidate as the language to write the tool in, and 
has the advantage of being more portable than shell scripts.

We'll add the tool to the tools/ repository and offer it with the 
standard distribution.


Thanks,

Andrei
Jun 10 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I thought this is what DVM already does? Or are you talking about
fetching unstable branches and compiling/installing them?
Jun 10 2011
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 10/06/2011 15:14, Andrei Alexandrescu wrote:
 We need a script to update all of dmd's paraphernalia. By this I'm
 putting the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or
 even the script can be told to get the latest and greatest. The exact
 tagging convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever
 and however you installed dmd, the script should be able to detect
 everything it needs for a successful update. This means: dmd and rdmd
 binary locations, druntime and phobos library locations, and druntime
 and phobos import locations.

 The way I'm thinking this all can be done is by running dmd -v against a
 small do-nothing program that imports one std module, for example
 std.stdio. The run will print (among other things):

 binary dmd
 version v2.054
 config /Users/john/dmd.conf
 parse test
 importall test
 import object (/Users/john/code/dmd/druntime/import/object.di)
 import std.stdio (/Users/john/code/dmd/phobos/std/stdio.d)
 ...

 Simple parsing reveals the location of the configuration file and the
 locations of druntime and std imports. Further simple parsing of the
 conf file reveals the location of libraries.

 With this information in hand, the script goes and fetches the
 appropriate code in the appropriate places.

 Also, the tool needs to do all of its work transactionally, i.e. first
 download everything next to the target with an added suffix (e.g.
 ".tmp"). Only after a full successful download, the tool would rename
 everything .tmp to its final name (an operation that can't fail under
 most circumstances).

 If anyone is willing to take up working on this, it would be great. D
 itself is a great candidate as the language to write the tool in, and
 has the advantage of being more portable than shell scripts.

 We'll add the tool to the tools/ repository and offer it with the
 standard distribution.


 Thanks,

 Andrei

This sounds like a rather limited tool, why limit it to dmd? The tool you describe is just a watered down version of a full package manager. Also, as already mentioned, dvm does this. -- Robert http://octarineparrot.com/
Jun 10 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/11 12:34 PM, Robert Clipsham wrote:
 This sounds like a rather limited tool, why limit it to dmd? The tool
 you describe is just a watered down version of a full package manager.

Then define a full package manager and have the tool use it.
 Also, as already mentioned, dvm does this.

From what I saw it doesn't do all that. Andrei
Jun 10 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-06-10 20:09, Andrei Alexandrescu wrote:
 On 6/10/11 12:34 PM, Robert Clipsham wrote:
 This sounds like a rather limited tool, why limit it to dmd? The tool
 you describe is just a watered down version of a full package manager.

Then define a full package manager and have the tool use it.
 Also, as already mentioned, dvm does this.

From what I saw it doesn't do all that. Andrei

What is it that you need it to do? -- /Jacob Carlborg
Jun 17 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Plus thanks to Nick dvm now works on Windoze.
Jun 10 2011
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Andrei Alexandrescu" <SeeWebsiteForEmail erdani.org> wrote in message 
news:ist957$19nj$1 digitalmars.com...
 We need a script to update all of dmd's paraphernalia. By this I'm putting 
 the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git 
 commands to freshen the installation. Either a tag can be given, or the 
 script would automatically detect the latest "semistable" release, or even 
 the script can be told to get the latest and greatest. The exact tagging 
 convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever 
 and however you installed dmd, the script should be able to detect 
 everything it needs for a successful update. This means: dmd and rdmd 
 binary locations, druntime and phobos library locations, and druntime and 
 phobos import locations.

I've already been planning to add the features of "recompile DMD" and "check out DMD from git and compile it" to DVM (unless Jacob plans to beat me to it). You do have good suggestions on some of the implementation details. If there's still another feature in your proposal that I've overlooked here, let me know. I was going to wait to do this until Jacob merged in my Windows pull request (he said he'd get to it end of next week, IIRC), because I want to base it off a DVM that already has Windows support, but maybe I shouldn't wait and should just fork my "windows fork" and get to it? (I'm still fairly new to DVCS and "proper DVCS procedures".) DVM is written in D1/Tango, though. Would that be a problem? If so, and if Jacob doesn't have any objections to it being ported to D2/Phobos, then I'd volunteer to do the port. I already have experience porting a bunch of my own code from D1/Tango to D2/Phobos. But I don't want to be hasty about porting it: I assume Jacob had a reason for using D1/Tango. I don't know how important the reason was.
Jun 10 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-10 22:09, Nick Sabalausky wrote:
 "Andrei Alexandrescu"<SeeWebsiteForEmail erdani.org>  wrote in message
 news:ist957$19nj$1 digitalmars.com...
 We need a script to update all of dmd's paraphernalia. By this I'm putting
 the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or even
 the script can be told to get the latest and greatest. The exact tagging
 convention is to be defined later.

 The trick with such tools is they must "just work". That means wherever
 and however you installed dmd, the script should be able to detect
 everything it needs for a successful update. This means: dmd and rdmd
 binary locations, druntime and phobos library locations, and druntime and
 phobos import locations.

I've already been planning to add the features of "recompile DMD" and "check out DMD from git and compile it" to DVM (unless Jacob plans to beat me to it). You do have good suggestions on some of the implementation details. If there's still another feature in your proposal that I've overlooked here, let me know.

This has been on my long-term-todo list since I started the project. Although I have no idea if this is possible, because, as far as I know, there is no the "connection" between the DMD, druntime and Phobos repositories. How do I know if a particular revision/hash from one repository will work with one from the other repositories?
 I was going to wait to do this until Jacob merged in my Windows pull request
 (he said he'd get to it end of next week, IIRC), because I want to base it
 off a DVM that already has Windows support, but maybe I shouldn't wait and
 should just fork my "windows fork" and get to it? (I'm still fairly new to
 DVCS and "proper DVCS procedures".)

 DVM is written in D1/Tango, though. Would that be a problem? If so, and if
 Jacob doesn't have any objections to it being ported to D2/Phobos, then I'd
 volunteer to do the port. I already have experience porting a bunch of my
 own code from D1/Tango to D2/Phobos. But I don't want to be hasty about
 porting it: I assume Jacob had a reason for using D1/Tango. I don't know how
 important the reason was.

I would prefer to use Tango. The reasons I chose Tango were/are: * I'm used to D1/Tango * Every time I try to use D2 something isn't working * I still think Tango is more superior than Phobos, although Phobos is doing a great job of filling in the feature holes * A particular feature and the most important that I needed was to easy download files via HTTP, Tango makes this very easy. Now Phobos is getting a curl wrapper which should make it as easy, don't know the status of that though. I don't want to and don't know how to use a low level socket to do this. -- /Jacob Carlborg
Jun 17 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-17 18:15, Jose Armando Garcia wrote:
 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com>  wrote:
 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.

You can always exec a program. You can just call the wget or curl or even scp!

That just creates additional dependencies and these tools are not available on Windows. Creating a new process also takes additional resources, time, memory and so on. -- /Jacob Carlborg
Jun 17 2011
parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:itgadv$2ggr$3 digitalmars.com...
 On 2011-06-17 18:15, Jose Armando Garcia wrote:
 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com>  wrote:
 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is 
 getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to 
 do
 this.

You can always exec a program. You can just call the wget or curl or even scp!

That just creates additional dependencies and these tools are not available on Windows. Creating a new process also takes additional resources, time, memory and so on.

wget is available in Windows (I have it installed), but finding/downloading/installing any of the GNU command line tools on Windows is a royal pain. And even if it weren't, I agree, it's better to just not have that dependency anyway.
Jun 18 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-06-17 22:12, Jimmy Cao wrote:
 On Fri, Jun 17, 2011 at 11:07 AM, Jacob Carlborg <doob me.com
 <mailto:doob me.com>> wrote:


     * A particular feature and the most important that I needed was to
     easy download files via HTTP, Tango makes this very easy. Now Phobos
     is getting a curl wrapper which should make it as easy, don't know
     the status of that though. I don't want to and don't know how to use
     a low level socket to do this.


 etc.c.curl seems to be working great on Windows.  It does create a
 dependency on libcurl.dll, libeay32.dll, libidn-11.dll, and libssl32.dll
 though.

Yet another dependency to worry about. This is not as easy to distribute as zlib due to the license. -- /Jacob Carlborg
Jun 17 2011
prev sibling next sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg <doob me.com> wrote:
 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.

You can always exec a program. You can just call the wget or curl or even scp!
Jun 17 2011
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
--002354186e648345bc04a5ebb679
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Jun 17, 2011 at 10:15 AM, Jose Armando Garcia <jsancio gmail.com>wrote:

 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg <doob me.com> wrote:
 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is

 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to

 this.

You can always exec a program. You can just call the wget or curl or even scp!

Windows complicates this as neither are available by typically. --002354186e648345bc04a5ebb679 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Jun 17, 2011 at 10:15 AM, Jose Armando Garcia <span dir=3D"ltr">&lt= ;<a href=3D"mailto:jsancio gmail.com">jsancio gmail.com</a>&gt;</span> wrot= e:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D= "margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <div class=3D"im">On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg &lt;<a hr= ef=3D"mailto:doob me.com">doob me.com</a>&gt; wrote:<br> &gt; On 2011-06-10 22:09, Nick Sabalausky wrote:<br> </div><div class=3D"im">&gt; * A particular feature and the most important = that I needed was to easy<br> &gt; download files via HTTP, Tango makes this very easy. Now Phobos is get= ting a<br> &gt; curl wrapper which should make it as easy, don&#39;t know the status o= f that<br> &gt; though. I don&#39;t want to and don&#39;t know how to use a low level = socket to do<br> &gt; this.<br> &gt;<br> <br> </div>You can always exec a program. You can just call the wget or curl or = even scp!<br> </blockquote></div><br><div>Windows complicates this as neither are availab= le by typically.</div> --002354186e648345bc04a5ebb679--
Jun 17 2011
prev sibling next sibling parent Jimmy Cao <jcao219 gmail.com> writes:
--0016363b7fecd07d3304a5ee01d7
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Jun 17, 2011 at 11:07 AM, Jacob Carlborg <doob me.com> wrote:

 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is getting a
 curl wrapper which should make it as easy, don't know the status of that
 though. I don't want to and don't know how to use a low level socket to do
 this.

etc.c.curl seems to be working great on Windows. It does create a dependency on libcurl.dll, libeay32.dll, libidn-11.dll, and libssl32.dll though. --0016363b7fecd07d3304a5ee01d7 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Jun 17, 2011 at 11:07 AM, Jacob Carlborg <span dir=3D"ltr">&lt;<a h= ref=3D"mailto:doob me.com">doob me.com</a>&gt;</span> wrote:<div class=3D"g= mail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex;"> <div class=3D"im"><br></div> * A particular feature and the most important that I needed was to easy dow= nload files via HTTP, Tango makes this very easy. Now Phobos is getting a c= url wrapper which should make it as easy, don&#39;t know the status of that= though. I don&#39;t want to and don&#39;t know how to use a low level sock= et to do this.<br> </blockquote><div><br></div><div>etc.c.curl seems to be working great on Wi= ndows. =A0It does create a dependency on=A0libcurl.dll,=A0libeay32.dll,=A0l= ibidn-11.dll, and=A0libssl32.dll though.=A0</div></div> --0016363b7fecd07d3304a5ee01d7--
Jun 17 2011
prev sibling next sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
On Fri, Jun 17, 2011 at 4:32 PM, Jacob Carlborg <doob me.com> wrote:
 On 2011-06-17 18:15, Jose Armando Garcia wrote:
 On Fri, Jun 17, 2011 at 1:07 PM, Jacob Carlborg<doob me.com> =A0wrote:
 On 2011-06-10 22:09, Nick Sabalausky wrote:
 * A particular feature and the most important that I needed was to easy
 download files via HTTP, Tango makes this very easy. Now Phobos is
 getting a
 curl wrapper which should make it as easy, don't know the status of tha=



 though. I don't want to and don't know how to use a low level socket to
 do
 this.

You can always exec a program. You can just call the wget or curl or eve=


 scp!

That just creates additional dependencies and these tools are not availab=

 on Windows. Creating a new process also takes additional resources, time,
 memory and so on.

Windows? Who programs for that thing? Just kidding. I know of the problems. My comment was meant as a obscure comment about that aspect of the Windows platform.
Jun 17 2011
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Holy shit look how complicated:
http://gnuwin32.sourceforge.net/packages/wget.htm
http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip
Jun 18 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
P.S. as you can see I'm a little jumpy today. Phobos bugs got to me a little. :)

On 6/18/11, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 Holy shit look how complicated:
 http://gnuwin32.sourceforge.net/packages/wget.htm
 http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip

Jun 18 2011
parent "Nick Sabalausky" <a a.a> writes:
Heh, well, my statement was perhaps a bit inaccurate.

"Andrej Mitrovic" <andrej.mitrovich gmail.com> wrote in message 
news:mailman.1002.1308435182.14074.digitalmars-d puremagic.com...
 P.S. as you can see I'm a little jumpy today. Phobos bugs got to me a 
 little. :)

 On 6/18/11, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 Holy shit look how complicated:
 http://gnuwin32.sourceforge.net/packages/wget.htm
 http://downloads.sourceforge.net/gnuwin32/wget-1.11.4-1-bin.zip
 


Jun 18 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-06-10 16:14, Andrei Alexandrescu wrote:
 We need a script to update all of dmd's paraphernalia. By this I'm
 putting the task up for grabs and am describing it.

 The script would require git to be installed, and would orchestrate git
 commands to freshen the installation. Either a tag can be given, or the
 script would automatically detect the latest "semistable" release, or
 even the script can be told to get the latest and greatest. The exact
 tagging convention is to be defined later.

You don't need git to just do a simple clone. Have you seen the big "Downloads" button? You can download HEAD, tags and (I assume) branches. -- /Jacob Carlborg
Jun 17 2011