www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Mixins and compile-time coding

reply janderson <askme me.com> writes:
Here's a suggestion that I actually sent a couple of of years ago that 
actually make sense now with the new mixin stuff.  What if you could 
plugin dlls into DMD that would be run at compile time.  The requirement 
of use would be that the input parameters would be constant.


ie

importplugin("perl");

mixin(Perl("perl code"));

//In the Perl dll
char[] Perl(char[] source)
{
  ...
}

Of course the other option would be to compile the Perl code on the fly. 
    Parhaps you could have "char[] str = static func()" or something (ie 
prefix calls to these functions with static).

of course templates could be used to represent almost anything but I 
find them hard to read when they start to try to do something weird.

The only downside I see is that people could write really slow dlls 
which would slow down compile time.  It probably would be a good idea 
for DMD to actually printout how much time is spent in each on of these 
processes (ie point its finger).

If this technique where to be enabled, we would be able to take all the 
string handling routines in the standard lib and use them now.

-Joel
Feb 06 2007
next sibling parent reply Walter Bright <newshound digitalmars.com> writes:
janderson wrote:

 Here's a suggestion that I actually sent a couple of of years ago that 
 actually make sense now with the new mixin stuff.  What if you could 
 plugin dlls into DMD that would be run at compile time.  The requirement 
 of use would be that the input parameters would be constant.



The main problem with this is the malware author who puts some source 
code up, and when you try to compile it, it would install a rootkit.
Feb 06 2007
parent reply janderson <askme me.com> writes:
Walter Bright wrote:

 janderson wrote:

 Here's a suggestion that I actually sent a couple of of years ago that 
 actually make sense now with the new mixin stuff.  What if you could 
 plugin dlls into DMD that would be run at compile time.  The 
 requirement of use would be that the input parameters would be constant.



 The main problem with this is the malware author who puts some source 
 code up, and when you try to compile it, it would install a rootkit.



Good point.  What about the static version.  ie

//This would be compiled on the fly. It could be put into its own object 
file so it doesn't need to be re-built everytime.
mixin char[] stateMachine(char[] input)
{
  ...
}

void main()
{
     stateMachine("ect...");
}

or

plugin char[] stateMachine(char[] input)
{
  ...
}

void main()
{
      mixin(stateMachine("ect..."));
}

or

char[] stateMachine(char[] input)
{
  ...
}

void main()
{
      mixin(plugin stateMachine("ect..."));
}


Something like that.


I guess malware may still be possible, but you can do that with unit 
tests now.  Essentially these are not much more then unit tests.

-Joel
Feb 06 2007
parent janderson <askme me.com> writes:
janderson wrote:

 Walter Bright wrote:

 janderson wrote:

 Here's a suggestion that I actually sent a couple of of years ago 
 that actually make sense now with the new mixin stuff.  What if you 
 could plugin dlls into DMD that would be run at compile time.  The 
 requirement of use would be that the input parameters would be constant.



 The main problem with this is the malware author who puts some source 
 code up, and when you try to compile it, it would install a rootkit.



 Good point.  What about the static version.  ie
 
 //This would be compiled on the fly. It could be put into its own object 
 file so it doesn't need to be re-built everytime.
 mixin char[] stateMachine(char[] input)
 {
  ...
 }
 
 void main()
 {
     stateMachine("ect...");
 }
 
 or
 
 plugin char[] stateMachine(char[] input)
 {
  ...
 }
 
 void main()
 {
      mixin(stateMachine("ect..."));
 }
 
 or
 
 char[] stateMachine(char[] input)
 {
  ...
 }
 
 void main()
 {
      mixin(plugin stateMachine("ect..."));
 }
 
 
 Something like that.
 
 
 I guess malware may still be possible, but you can do that with unit 
 tests now.  Essentially these are not much more then unit tests.
 
 -Joel



Actually you could go one step better with this.  No calls to functions 
outside of D.  That way you'd have access to all of D's 
string/int/float/array operations but you wouldn't be able to make 
systems calls at all.   You could also disabled asm calls if you wanted 
to get really restrictive although I don't see that as necessary.

The only way you'd be able to hack the system is to do some sort of 
funky memory accessing.

-JOel
Feb 06 2007
prev sibling next sibling parent reply Daniel919 <Daniel919 web.de> writes:
Hi, why not use D instead of perl ?
Since "sh style script syntax" is supported, I would like the following to be
possible:

import std.stdio, std.string, std.metastrings;

void foo(A...)(A a)
{
    writefln(format(typeid(typeof(a[1]))) ~ " " ~ a[0] ~ " = \"" ~ a[1] ~ "\"");
}

void main (char[][] args)
{
    debug(templates)
        foo("var", "whatever");

    foo!("var", "whatever");
    /* This will call dmd -run on this file. But instead of starting at main(),
     * it starts by invoking foo() with the tuple ("var", "whatever").
     * Mixed into the context will be the console output of the
     * dmd -run call, in this case:
     * char[8] var = "whatever";
    */

    writefln(var);
}

Wouldn't this be very neat and powerful ?

Walter, about your concerns on malware autors:
Of course script interpretation within the compile-process could be used for
malware. But also the malware could be put into the compiled code itself and
get's started if the app is run. So I think it doesn't make any difference.

Best regards,
Daniel
Feb 06 2007
next sibling parent Walter Bright <newshound digitalmars.com> writes:
Daniel919 wrote:

 Hi, why not use D instead of perl ?



I agree. I don't see any reason to use perl, unless you're using a 
pre-existing perl script.



 Walter, about your concerns on malware autors:
 Of course script interpretation within the compile-process could be used for
malware.
 But also the malware could be put into the compiled code itself and 




 the app is run. So I think it doesn't make any difference.



At a technical level, you're right. But people don't expect this to be 
possible when just the compiler is run, and won't be looking out for it. 
I also don't want D compilers to be maligned as vectors for malware, 
even if such maligning is undeserved. I don't want D compilers to be 
blocked by security conscious administrators.
Feb 06 2007
prev sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Daniel919 wrote:

 Hi, why not use D instead of perl ?
 Since "sh style script syntax" is supported, I would like the following to be
possible:
 
 import std.stdio, std.string, std.metastrings;
 
 void foo(A...)(A a)
 {
     writefln(format(typeid(typeof(a[1]))) ~ " " ~ a[0] ~ " = \"" ~ a[1] ~
"\"");
 }
 
 void main (char[][] args)
 {
     debug(templates)
         foo("var", "whatever");
 
     foo!("var", "whatever");
     /* This will call dmd -run on this file. But instead of starting at main(),
      * it starts by invoking foo() with the tuple ("var", "whatever").
      * Mixed into the context will be the console output of the
      * dmd -run call, in this case:
      * char[8] var = "whatever";
     */
 
     writefln(var);
 }
 
 Wouldn't this be very neat and powerful ?
 
 Walter, about your concerns on malware autors:
 Of course script interpretation within the compile-process could be used for
malware. But also the malware could be put into the compiled code itself and
get's started if the app is run. So I think it doesn't make any difference.
 
 Best regards,
 Daniel



Also, people download and run makefiles to build their software all the 
time. Who knows what that makefile is doing.  There could be an rm -rf / 
buried in there somewhere in all that tab-sensitive gobbledy-gook.

--bb
Feb 06 2007
next sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
John Reimer wrote:

 On Wed, 07 Feb 2007 05:13:06 +0900, Bill Baxter wrote:
 

 Daniel919 wrote:

 Hi, why not use D instead of perl ?
 Since "sh style script syntax" is supported, I would like the following to be
possible:

 import std.stdio, std.string, std.metastrings;

 void foo(A...)(A a)
 {
     writefln(format(typeid(typeof(a[1]))) ~ " " ~ a[0] ~ " = \"" ~ a[1] ~
"\"");
 }

 void main (char[][] args)
 {
     debug(templates)
         foo("var", "whatever");

     foo!("var", "whatever");
     /* This will call dmd -run on this file. But instead of starting at main(),
      * it starts by invoking foo() with the tuple ("var", "whatever").
      * Mixed into the context will be the console output of the
      * dmd -run call, in this case:
      * char[8] var = "whatever";
     */

     writefln(var);
 }

 Wouldn't this be very neat and powerful ?

 Walter, about your concerns on malware autors:
 Of course script interpretation within the compile-process could be used for
malware. But also the malware could be put into the compiled code itself and
get's started if the app is run. So I think it doesn't make any difference.

 Best regards,
 Daniel



 time. Who knows what that makefile is doing.  There could be an rm -rf / 
 buried in there somewhere in all that tab-sensitive gobbledy-gook.

 --bb



 
 Very good point.
 
 Any auxillary tool in the compilation process could could do the same.  I
 have difficult recognizing the validity of Walter's concern in this context.
 
 -JJR



I think the concern is that an awful lot of people who notice this capability
will also 
/not/ notice the context|precedant.  And perhaps also a bit of good
conscience... but 
that's no real obstacle in and of itself.

-- Chris Nicholson-Sauls
Feb 06 2007
prev sibling parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Wed, 07 Feb 2007 02:06:44 +0200, John Reimer <terminal.node gmail.com> wrote:


 On Wed, 07 Feb 2007 05:13:06 +0900, Bill Baxter wrote:


 Also, people download and run makefiles to build their software all the
 time. Who knows what that makefile is doing.  There could be an rm -rf /
 buried in there somewhere in all that tab-sensitive gobbledy-gook.



 Very good point.

 Any auxillary tool in the compilation process could could do the same.  I
 have difficult recognizing the validity of Walter's concern in this context.



I believe there must be a strict distinction between tools which just work with
data or code, and tools which run code. The sole definition of a compiler is
transform human-readable code to machine code - not run any of those outside
the purpose of generating other code or data.

Unit tests are a separate issue, because to run them the user must specify a
switch on the compiler's command-line - in which case he is well-aware that
code will be run. Thus, if any such feature does get implemented, the compiler
should not execute any code and give off an error instead, and force the user
to specify a switch which allows this, to prevent any naive mistakes and
annoyances.

That, or write in big letters on the DMD download page... "This compiler may
run some of the compiled code" in big red letters. But that'll just scare more
people off...

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Feb 06 2007
parent reply "Vladimir Panteleev" <thecybershadow gmail.com> writes:
On Wed, 07 Feb 2007 07:44:06 +0200, John Reimer <terminal.node gmail.com> wrote:


 On Wed, 07 Feb 2007 07:36:23 +0200, Vladimir Panteleev wrote:


 I believe there must be a strict distinction between tools which just work
with data or code, and tools which run code. The sole definition of a compiler
is transform human-readable code to machine code - not run any of those outside
the purpose of generating other code or data.

 Unit tests are a separate issue, because to run them the user must specify a
switch on the compiler's command-line - in which case he is well-aware that
code will be run. Thus, if any such feature does get implemented, the compiler
should not execute any code and give off an error instead, and force the user
to specify a switch which allows this, to prevent any naive mistakes and
annoyances.

 That, or write in big letters on the DMD download page... "This compiler may
run some of the compiled code" in big red letters. But that'll just scare more
people off...



 Hmm... so is not possible that rdmd.exe (rdmd on linux) might commit the
 same sort of security breech?  It's even packaged with the compiler?  Is
 it safe?



rdmd is on the other side of the "line" which I described - its base purpose is
to run code, while the compiler's purpose (unless explicitly specified by the
user) is to manipulate code. If we are to continue thinking this way, why not
eliminate the "executable" attribute on Unix filesystems and assume all files
are executable? The user surely knows what is he doing. It's simply common
safety practice.

Either way, allowing the compiler to run potentially dangerous code should at
least be optional (however [my] common sense would say that it should be
disallowed unless manually enabled). Imagine the problems this would cause with
remote compiling systems, for example at ACM contests.

-- 
Best regards,
  Vladimir                          mailto:thecybershadow gmail.com
Feb 06 2007
next sibling parent janderson <askme me.com> writes:
Vladimir Panteleev wrote:

 On Wed, 07 Feb 2007 07:44:06 +0200, John Reimer <terminal.node gmail.com>
wrote:
 

 On Wed, 07 Feb 2007 07:36:23 +0200, Vladimir Panteleev wrote:


 I believe there must be a strict distinction between tools which just work
with data or code, and tools which run code. The sole definition of a compiler
is transform human-readable code to machine code - not run any of those outside
the purpose of generating other code or data.

 Unit tests are a separate issue, because to run them the user must specify a
switch on the compiler's command-line - in which case he is well-aware that
code will be run. Thus, if any such feature does get implemented, the compiler
should not execute any code and give off an error instead, and force the user
to specify a switch which allows this, to prevent any naive mistakes and
annoyances.

 That, or write in big letters on the DMD download page... "This compiler may
run some of the compiled code" in big red letters. But that'll just scare more
people off...



 same sort of security breech?  It's even packaged with the compiler?  Is
 it safe?



 rdmd is on the other side of the "line" which I described - its base purpose
is to run code, while the compiler's purpose (unless explicitly specified by
the user) is to manipulate code. If we are to continue thinking this way, why
not eliminate the "executable" attribute on Unix filesystems and assume all
files are executable? The user surely knows what is he doing. It's simply
common safety practice.
 
 Either way, allowing the compiler to run potentially dangerous code should at
least be optional (however [my] common sense would say that it should be
disallowed unless manually enabled). Imagine the problems this would cause with
remote compiling systems, for example at ACM contests.
 



Some good points.  With great power comes great responsibility.  Its a 
good idea to be able to turn off the responsibility when you want it.

I've got nothing against a switch however I think others may feel that 
once an extremely powerful feature like this gained a foothold, half of 
D wouldn't work if you disabled the option.

-Joel
Feb 06 2007
prev sibling parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Vladimir Panteleev wrote:

 Either way, allowing the compiler to run potentially dangerous code should at
least be optional (however [my] common sense would say that it should be
disallowed unless manually enabled). Imagine the problems this would cause with
remote compiling systems, for example at ACM contests.



I'm with you on that point.

Take something like DSP: you have an embedded D compiler that is invoked in
response to a change in a web-script written 
in D. Without being able to specify where these literal imports are coming
from, the results could be disastrous.

There should be an option to disable the behavior, and the path searching
behavior of the import expression needs to be 
well defined.  Ideally, this would involve a separate switch for specifying
import paths as distinct from the module 
include path (so it can be routed to /dev/null if need be). The last thing we
want is people hijacking a server, just 
because a compiler was provided under httpd's process (with it's permissions).


-- 
- EricAnderton at yahoo
Feb 07 2007
next sibling parent Walter Bright <newshound digitalmars.com> writes:
Pragma wrote:

 Take something like DSP: you have an embedded D compiler that is invoked 
 in response to a change in a web-script written in D. Without being able 
 to specify where these literal imports are coming from, the results 
 could be disastrous.
 
 There should be an option to disable the behavior, and the path 
 searching behavior of the import expression needs to be well defined.  
 Ideally, this would involve a separate switch for specifying import 
 paths as distinct from the module include path (so it can be routed to 
 /dev/null if need be). The last thing we want is people hijacking a 
 server, just because a compiler was provided under httpd's process (with 
 it's permissions).



I think you're right.
Feb 07 2007
prev sibling parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Pragma wrote:

 Vladimir Panteleev wrote:

 Either way, allowing the compiler to run potentially dangerous code 
 should at least be optional (however [my] common sense would say that 
 it should be disallowed unless manually enabled). Imagine the problems 
 this would cause with remote compiling systems, for example at ACM 
 contests.



 I'm with you on that point.
 
 Take something like DSP: you have an embedded D compiler that is invoked 
 in response to a change in a web-script written in D. Without being able 
 to specify where these literal imports are coming from, the results 
 could be disastrous.



But isn't DSP also going to *run* the resulting compiled code?  If so 
then you still have the exact same security issues.

--bb
Feb 07 2007
prev sibling next sibling parent John Reimer <terminal.node gmail.com> writes:
On Wed, 07 Feb 2007 05:13:06 +0900, Bill Baxter wrote:


 Daniel919 wrote:

 Hi, why not use D instead of perl ?
 Since "sh style script syntax" is supported, I would like the following to be
possible:
 
 import std.stdio, std.string, std.metastrings;
 
 void foo(A...)(A a)
 {
     writefln(format(typeid(typeof(a[1]))) ~ " " ~ a[0] ~ " = \"" ~ a[1] ~
"\"");
 }
 
 void main (char[][] args)
 {
     debug(templates)
         foo("var", "whatever");
 
     foo!("var", "whatever");
     /* This will call dmd -run on this file. But instead of starting at main(),
      * it starts by invoking foo() with the tuple ("var", "whatever").
      * Mixed into the context will be the console output of the
      * dmd -run call, in this case:
      * char[8] var = "whatever";
     */
 
     writefln(var);
 }
 
 Wouldn't this be very neat and powerful ?
 
 Walter, about your concerns on malware autors:
 Of course script interpretation within the compile-process could be used for
malware. But also the malware could be put into the compiled code itself and
get's started if the app is run. So I think it doesn't make any difference.
 
 Best regards,
 Daniel



 Also, people download and run makefiles to build their software all the 
 time. Who knows what that makefile is doing.  There could be an rm -rf / 
 buried in there somewhere in all that tab-sensitive gobbledy-gook.
 
 --bb




Very good point.

Any auxillary tool in the compilation process could could do the same.  I
have difficult recognizing the validity of Walter's concern in this context.

-JJR
Feb 06 2007
prev sibling next sibling parent John Reimer <terminal.node gmail.com> writes:
On Wed, 07 Feb 2007 07:36:23 +0200, Vladimir Panteleev wrote:


 On Wed, 07 Feb 2007 02:06:44 +0200, John Reimer <terminal.node gmail.com>
wrote:
 

 On Wed, 07 Feb 2007 05:13:06 +0900, Bill Baxter wrote:


 Also, people download and run makefiles to build their software all the
 time. Who knows what that makefile is doing.  There could be an rm -rf /
 buried in there somewhere in all that tab-sensitive gobbledy-gook.



 Very good point.

 Any auxillary tool in the compilation process could could do the same.  I
 have difficult recognizing the validity of Walter's concern in this context.



 I believe there must be a strict distinction between tools which just work
with data or code, and tools which run code. The sole definition of a compiler
is transform human-readable code to machine code - not run any of those outside
the purpose of generating other code or data.
 
 Unit tests are a separate issue, because to run them the user must specify a
switch on the compiler's command-line - in which case he is well-aware that
code will be run. Thus, if any such feature does get implemented, the compiler
should not execute any code and give off an error instead, and force the user
to specify a switch which allows this, to prevent any naive mistakes and
annoyances.
 
 That, or write in big letters on the DMD download page... "This compiler may
run some of the compiled code" in big red letters. But that'll just scare more
people off...




Hmm... so is not possible that rdmd.exe (rdmd on linux) might commit the
same sort of security breech?  It's even packaged with the compiler?  Is
it safe?

-JJR
Feb 06 2007
prev sibling parent John Reimer <terminal.node gmail.com> writes:
On Wed, 07 Feb 2007 07:53:20 +0200, Vladimir Panteleev wrote:


 Hmm... so is not possible that rdmd.exe (rdmd on linux) might commit the
 same sort of security breech?  It's even packaged with the compiler?  Is
 it safe?



 rdmd is on the other side of the "line" which I described - its base purpose
is to run code, while the compiler's purpose (unless explicitly specified by
the user) is to manipulate code. If we are to continue thinking this way, why
not eliminate the "executable" attribute on Unix filesystems and assume all
files are executable? The user surely knows what is he doing. It's simply
common safety practice.
 
 Either way, allowing the compiler to run potentially dangerous code should at
least be optional (however [my] common sense would say that it should be
disallowed unless manually enabled). Imagine the problems this would cause with
remote compiling systems, for example at ACM contests.




Good points.  Your argument make sense.

-JJR
Feb 06 2007