www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Make IN Dlang

reply =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
Dear dlang-folk,

one of the tools I always return to is rake 
(https://ruby.github.io/rake/). For those that do not know it, its a 
little like make in the
sense that you describe your build as a graph of tasks with dependencies
between them, but in contrast to make the definition is written in
a normal programming language (in this case ruby) with all features of
it.

Dlang is also quite expressive, so I thought why not define the build in 
Dlang.

The result is mind (https://gitlab.com/gizmomogwai/mind). An example 
mindfile looks like this:
```

/+ dub.sdl:
    name "mindfile"
    dependency "mind" version="~master"
    ...further dependencies...
  +/
import mind;
import std.stdio : writeln;
import std.format : format;
import std.range : iota;
import std.algorithm : map;
import std.array : array;
import core.thread.osthread : Thread;
import core.time : msecs;

int main(string[] args)
{
     description("Do all");
     auto all = task("all", null, (t) {});

     for (int i=0; i<1000; ++i)
     {
         auto fileName = "out/file-%s.txt".format(i);
         all.enhance(fileName);
         description(fileName);
         file(fileName, null, (t)
              {
                  Thread.sleep(100.msecs);
                  sh("touch %s".format(t.name));
              },
         );
     }

     return mindMain(args);
}

```

This uses dub's single file feature
(https://dub.pm/advanced_usage#single-file) to get the helper library
and "execution engine" (mind).

The main functions defines a bunch of tasks or file-tasks and finally
forwards to the main function of the library for parallel (if possible)
execution of the tasks.

At the moment this is still in an experimental stage, but has nice
features such as:
- colorized --help (thanks to argparse)
- --tasks to show all defined (and commented tasks)
- parallel execution


I am still trying to find answers to the following questions:
1. Is it somehow possible to get rid of the dub single file scheme, and
    e.g. interpret a full dlang script at runtime?
2. How can the program as is made nicer/shorter? (one could just import
    std, or make use of https://code.dlang.org/packages/scriptlike)?
3. Does this make sense at all, or should we just use make/rake? (dub
    also uses a custom build.d file).


Kind regards,
Christian
Nov 01 2022
next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
I don't have specific answers to your questions but your goal 
sounds similar to Atila's reggae project so it might be good for 
you to take a look at:

https://code.dlang.org/packages/reggae
Nov 01 2022
parent =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
On 02.11.22 00:51, Adam D Ruppe wrote:

 I don't have specific answers to your questions but your goal sounds 
 similar to Atila's reggae project so it might be good for you to take a 
 look at:
 
 https://code.dlang.org/packages/reggae


Hi Adam,

thanks for the pointer. I forgot about reggae ;-)
 From the documentation it looks more high level and already defines
things like libraries and so on. It's also very ambitious in that it
tries to support different backends!

Mind is more low level (although some language specific things could be
added on top of it.

I like reggaes approach in reading in the "source" and creating 
something new from it (perhaps with dub as library) this could get rid 
of some of the boilerplate of what I have now.

Kind regards,
Christian
Nov 02 2022
prev sibling next sibling parent reply Tejas <notrealemail gmail.com> writes:
On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin 
wrote:

 Dear dlang-folk,

 one of the tools I always return to is rake 
 (https://ruby.github.io/rake/). For those that do not know it, 
 its a little like make in the
 sense that you describe your build as a graph of tasks with 
 dependencies
 between them, but in contrast to make the definition is written 
 in
 a normal programming language (in this case ruby) with all 
 features of
 it.

 [...]


Sounds pretty similar to [tup](https://gittup.org/tup/)

Reggae, the build system mentioned by Adam, supports tup as a 
backend, so you could use that as well
Nov 01 2022
parent =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
On 02.11.22 03:25, Tejas wrote:

 On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin wrote:

 Dear dlang-folk,

 one of the tools I always return to is rake 
 (https://ruby.github.io/rake/). For those that do not know it, its a 
 little like make in the
 sense that you describe your build as a graph of tasks with dependencies
 between them, but in contrast to make the definition is written in
 a normal programming language (in this case ruby) with all features of
 it.

 [...]


 
 Sounds pretty similar to [tup](https://gittup.org/tup/)


I am a great admirer of tup (especially because they incorporated as the 
first system (that I know of) a filesystem monitor outside of IDEs).
Its language is make like I would say, for that I do not like it that much.


 Reggae, the build system mentioned by Adam, supports tup as a backend, 
 so you could use that as well


+1

Kind regards,
Christian
Nov 02 2022
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Something to consider:

dub can be used as a library.

You can add your own logic in main to allow using your build 
specification to generate a dub file (either in memory or in file system).
Nov 01 2022
parent =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
On 02.11.22 04:07, rikki cattermole wrote:

 Something to consider:
 
 dub can be used as a library.
 
 You can add your own logic in main to allow using your build 
 specification to generate a dub file (either in memory or in file system).


Nice ... I will perhaps give that a try!

Kind regards,
Christian
Nov 02 2022
prev sibling next sibling parent reply JN <666total wp.pl> writes:
On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin 
wrote:

                  sh("touch %s".format(t.name));


One of the problems of many Make-like tools is that they offer 
lots of freedom, especially when allowing you to launch arbitrary 
shell commands. But this also comes with drawbacks, because this 
touch command will instantly break Windows builds, might also be 
a problem on some non-Linux platforms like macOS. Declarative 
approach from tools like dub might be restrictive, but it also 
lets me as a user know that I can download an arbitrary dub 
project and 99% chance it will just compile out of the box on 
Windows.
Nov 02 2022
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Wed, Nov 02, 2022 at 03:08:36PM +0000, JN via Digitalmars-d-learn wrote:

 On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin wrote:

                  sh("touch %s".format(t.name));


 
 One of the problems of many Make-like tools is that they offer lots of
 freedom, especially when allowing you to launch arbitrary shell
 commands. But this also comes with drawbacks, because this touch
 command will instantly break Windows builds, might also be a problem
 on some non-Linux platforms like macOS. Declarative approach from
 tools like dub might be restrictive, but it also lets me as a user
 know that I can download an arbitrary dub project and 99% chance it
 will just compile out of the box on Windows.


IMO, the ideal situation is a hybrid situation: the underlying build
mechanism should be purely declarative, because otherwise the complexity
just goes out of control and you end up with non-portability,
non-reproducible builds, intractibility of static analysis (e.g., for an
external tool to understand what the build does).  However, quite often
in a large project you need to perform some complex tasks, and doing
this declaratively can be too cumbersome.  So what you want is a
procedural element to the build description that *generates* the
underlying declarative elements of the build.  The procedural part does
not perform any build actions; its job is to generate the declarative
build description.  The actual build is carried out based on this
declarative build description.


T

-- 
Life would be easier if I had the source code. -- YHL
Nov 02 2022
parent reply =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
On 02.11.22 20:16, H. S. Teoh wrote:

 On Wed, Nov 02, 2022 at 03:08:36PM +0000, JN via Digitalmars-d-learn wrote:

 On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin wrote:

                   sh("touch %s".format(t.name));


 One of the problems of many Make-like tools is that they offer lots of
 freedom, especially when allowing you to launch arbitrary shell
 commands. But this also comes with drawbacks, because this touch
 command will instantly break Windows builds, might also be a problem
 on some non-Linux platforms like macOS. Declarative approach from
 tools like dub might be restrictive, but it also lets me as a user
 know that I can download an arbitrary dub project and 99% chance it
 will just compile out of the box on Windows.


 
 IMO, the ideal situation is a hybrid situation: the underlying build
 mechanism should be purely declarative, because otherwise the complexity
 just goes out of control and you end up with non-portability,
 non-reproducible builds, intractibility of static analysis (e.g., for an
 external tool to understand what the build does).  However, quite often
 in a large project you need to perform some complex tasks, and doing
 this declaratively can be too cumbersome.  So what you want is a
 procedural element to the build description that *generates* the
 underlying declarative elements of the build.  The procedural part does
 not perform any build actions; its job is to generate the declarative
 build description.  The actual build is carried out based on this
 declarative build description.
 
 
 T
 


Thats an interesting approach. Reggae goes a little bit in that 
direction, right?

Kind regards,
Christian
Nov 02 2022
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Wed, Nov 02, 2022 at 09:16:22PM +0100, Christian Köstlin via
Digitalmars-d-learn wrote:

 On 02.11.22 20:16, H. S. Teoh wrote:


[...]

 IMO, the ideal situation is a hybrid situation: the underlying build
 mechanism should be purely declarative, because otherwise the
 complexity just goes out of control and you end up with
 non-portability, non-reproducible builds, intractibility of static
 analysis (e.g., for an external tool to understand what the build
 does).  However, quite often in a large project you need to perform
 some complex tasks, and doing this declaratively can be too
 cumbersome.  So what you want is a procedural element to the build
 description that *generates* the underlying declarative elements of
 the build.  The procedural part does not perform any build actions;
 its job is to generate the declarative build description.  The
 actual build is carried out based on this declarative build
 description.




[...]

 Thats an interesting approach. Reggae goes a little bit in that
 direction, right?


[...]

I haven't actually used reggae myself, but a cursory look suggests that
this is probably the case.


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
Nov 03 2022
prev sibling next sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 1 November 2022 at 23:40:22 UTC, Christian Köstlin 
wrote:

 I am still trying to find answers to the following questions:
 1. Is it somehow possible to get rid of the dub single file 
 scheme, and
    e.g. interpret a full dlang script at runtime?


If there was an interpreter like
```

...code
```
maybe it could run dub with right options and thus won't need a 
build script.
Nov 02 2022
next sibling parent Kagamin <spam here.lot> writes:
But embedded sdl is likely to be dwarfed by the actual code 
anyway.
Nov 02 2022
prev sibling parent reply Kagamin <spam here.lot> writes:
Another idea is to separate the script and interpreter then 
compile them together.
```
--- interp.d ---
import script;
import ...more stuff
...boilerplate code
int main()
{
   interpret(script.All);
   return 0;
}

--- script.d ---

module script;
import mind;

auto All=Task(...);
...more declarative tasks

--- run ---
dmd /usr/local/interp.d /path/to/script.d
```
Nov 02 2022
parent =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
On 02.11.22 17:24, Kagamin wrote:

 Another idea is to separate the script and interpreter then compile them 
 together.
 ```
 --- interp.d ---
 import script;
 import ...more stuff
 ...boilerplate code
 int main()
 {
    interpret(script.All);
    return 0;
 }
 
 --- script.d ---

 module script;
 import mind;
 
 auto All=Task(...);
 ...more declarative tasks
 
 --- run ---
 dmd /usr/local/interp.d /path/to/script.d
 ```


Thanks, have to think a little about that :)

Kind regards,
Christian
Nov 02 2022
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
Happened to stumble across this today, which I thought is a relevant, if
sadly humorous, take on build systems:

	https://pozorvlak.dreamwidth.org/174323.html


T

-- 
ASCII stupid question, getty stupid ANSI.
Nov 04 2022