www.digitalmars.com         C & C++   DMDScript  

D - Compiling Phobos

reply Burton Radons <loth users.sourceforge.net> writes:
I'm working on listdir and stat functions for the file module in
Phobos, which naturally requires compiling Phobos once I'm done.  The
makefile is out of date, however, and the directories are all wrong.

There's more.  array, memset, obj, and switcherr are not included in
the distribution but symbols from them are used.  interface.c,
intrinsic.d, iunknown.d, and time.d are not touched by the makefile.
None of them provide the necessary symbols, so I'm guessing there's
files missing.

Finally, sc compiles the .asm files incorrectly (it adds a semicolon
to the filename, bizarrely enough), and the version of masm and link I
got don't like the options used.  It'd be nice if the obj file were
just stuck in the distribution.

Now, onto listdir, statdir, and stat.  This code is in the public
domain blah blah blah and it follows.  I tried to follow Walter's
coding style, except for the no-indent-for-trivial-blocks, which I've
never seen before and hopefully will never see again.

This requires "ERROR_NO_MORE_FILES = 18," in the appropriate place in
windows.d and FileTimeToSystemTime using "export BOOL
FileTimeToSystemTime(FILETIME* lpFileTime, SYSTEMTIME*
lpSystemTime);".

-- /* For the string module */ --

/***************************************************
 * Determine whether the string ends with another.
 */

bit endswith(char[] string, char[] match)
{
    if (string.length < match.length)
        return 0;
    if (memcmp(&string [string.length - match.length], &match [0],
match.length))
        return 0;
    return 1;
}

/***************************************************
 * Determine whether the string begins with another.
 */

bit startswith(char[] string, char[] match)
{
    if (string.length < match.length)
        return 0;
    if (memcmp(&string [0], &match [0], match.length))
        return 0;
    return 1;
}

-- /* For the file module */ --

import date;
import path;

/***************************************************
 * Structure of a file stat.
 */

class Stat
{
    char[] filename; /* Filename plus full path */
    char[] filebase; /* Filename without path */
    char[] alternate; /* Alternate (8.3) filename, no path */
    d_time creation; /* When this file was created */
    d_time access; /* When this file was last used (this will probably
be writeTime in Linux) */
    d_time write; /* When this file was last written */
    long filesize; /* File size in bytes */

    bit archive; /* Is an archive file */
    bit directory; /* Is a directory */
    bit hidden; /* Is hidden (Linux should probably flag this if the
file starts with .) */
    bit readonly; /* Is read-only (Linux should set this if the
process can't write this file) */
    bit system; /* Is a system file */

    this (char[] path, WIN32_FIND_DATA finder)
    {
        SYSTEMTIME systemtime;
        FILETIME filetime;

        /* Get the filename and file size */
        filebase = (char []) finder.cFileName;
        filename = path ~ filebase;
        alternate = ((char []) finder.cAlternateFileName).dup;
        filesize = (long) finder.nFileSizeHigh * (long) 0x100000000 +
finder.nFileSizeLow;

        /* Get the attributes */
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_ARCHIVE)
            archive = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
            directory = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_HIDDEN)
            hidden = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_READONLY)
            readonly = 1;
        if (finder.dwFileAttributes & FILE_ATTRIBUTE_SYSTEM)
            system = 1;

        /* Get the times */

        filetime = finder.ftCreationTime;
        if (filetime.dwLowDateTime == 0 && filetime.dwHighDateTime ==
0)
            filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        creation = SYSTEMTIME2d_time(&systemtime, 0);

        filetime = finder.ftLastAccessTime;
        if (filetime.dwLowDateTime == 0 && filetime.dwHighDateTime ==
0)
            filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        access = SYSTEMTIME2d_time(&systemtime, 0);

        filetime = finder.ftLastWriteTime;
        if (!FileTimeToSystemTime(&filetime, &systemtime))
            throw new FileError(filename, GetLastError());
        write = SYSTEMTIME2d_time(&systemtime, 0);
    }
}

/***************************************************
 * Collate all files in a directory, returning an
 * array of strings.  The strings do not contain the
 * path.  Throws FileError if the directory does not
 * exist, if there was some file problem during the
 * collation, or if the path is malformed (generally
 * trying to use nonportable features).
 */

char[][] listdir(char[] path)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    char[][] list;
    bit done = 0;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (path, (char) "*") >= 0 || find (path, (char) "?") >= 0)
        throw new FileError(path, "Path contains invalid characters");

    /* Form the search path for Windows */
    if (endswith(path, "/") || endswith(path, "\\"))
        path ~= "*";
    else
        path ~= "/*";

    /* Start the file search */
    handle = FindFirstFileA(toStringz(path ~ "*"), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(path, GetLastError());
    
    while (!done)
    {
        char[] filename;

        /* Handle this file in our own inimitable fashion */
        filename = (char[]) finder.cFileName;
        list ~= filename.dup;

        /* Go to the next file or bust */
        if (!FindNextFileA(handle, &finder))
        {
            if (GetLastError() == ERROR_NO_MORE_FILES)
                done = 1;
            else
            {
                FindClose(handle);
                throw new FileError(path, GetLastError());
            }
        }
    }

    /* Finished, clean up */
    if (!FindClose(handle))
        throw new FileError(path, GetLastError());

    return list;
}

/***************************************************
 * Collate all files in a directory, returning an
 * array of stats.  Throws FileError if the
 * directory does not exist, if there was some file
 * problem during the collation, or if the path is
 * malformed (generally trying to use nonportable
 * features).
 */

Stat[] statdir(char[] path)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    Stat[] list;
    bit done = 0;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (path, (char) "*") >= 0 || find (path, (char) "?") >= 0)
        throw new FileError(path, "Path contains invalid characters");

    /* Form the search path for Windows */
    if (!endswith(path, "/") && !endswith(path, "\\"))
        path ~= "/";

    /* Start the file search */
    handle = FindFirstFileA(toStringz(path ~ "*"), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(path, GetLastError());
    
    while (!done)
    {
        /* Handle this file in our own inimitable fashion */
        list ~= new Stat (path, finder);

        /* Go to the next file or bust */
        if (!FindNextFileA(handle, &finder))
        {
            if (GetLastError() == ERROR_NO_MORE_FILES)
                done = 1;
            else
            {
                FindClose(handle);
                throw new FileError(path, GetLastError());
            }
        }
    }

    /* Finished, clean up */
    if (!FindClose(handle))
        throw new FileError(path, GetLastError());

    return list;
}

/***************************************************
 * Attempt to stat a single file, returning the
 * file details.  This will throw FileError when the
 * file cannot be found or when the path is badly
 * formed.
 */
 
Stat stat(char[] filename)
{
    WIN32_FIND_DATA finder;
    HANDLE handle;
    char[] path;
    Stat stat;

    /* Since we want to flatten out platform differences,
       first we cry if the user tried to use Windows. */

    if (find (filename, (char) "*") >= 0 || find (filename, (char)
"?") >= 0)
        throw new FileError(filename, "Path contains invalid
characters");

    /* Start the file search */
    handle = FindFirstFileA(toStringz(filename), &finder);
    if (handle == (HANDLE)-1)
        throw new FileError(filename, GetLastError());

    /* Construct the path */
    if (filename [0] != "/" && filename [0] != "\\")
        path = curdir;
    else
        path = getDirName(filename);

    /* Statify the sucker */
    stat = new Stat(path, finder);

    /* Finish up */
    if (!FindClose(handle))
        throw new FileError(path ~ pathsep, GetLastError());

    return stat;
}
Apr 26 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
Not really on the same subject, but I've put up a pre-alpha version of
a quick program maker for D; most Deed files (as the program is
called) are a single line, but have inherited options, dependencies
checking, and multitargets.  You can grab it at:

http://prdownloads.sourceforge.net/willow/deed.zip

The only extra symbol I needed was system:

"extern (C) { export int system(char* string); }
int system(char[] string) { return system ((char *) &(string ~ "\0")
[0]); }"

There apparently aren't many solid programs using D, so here's my
observations.

It's about as easy to prototype in D as it is in Python, if a little
easier (and that's saying something)... particularly now that I have
Deed, of course.

Array slicing should use default arguments; [ .. ] refers to the full
array, [x .. ] refers to x to the end of the array, [ .. x] refers to
the start of the array to x.  It's convenient.  I also like Python's
negative array indexing, but I can see many reasons to oppose it.

I want operator overloading; something like:

operator a + b (qbert a, int b) { qbert (a.x + b); }

Clean and unambiguous; C++'s method is way too confusing.

I want control of the parser and compiler at runtime.  I also want to
be able to take a class instance and reconstruct its class definition
perfectly.  I then want to be able to pick up dynamic modules and link
them into the running program myself.  This is all for my never-ending
quest for a scripting language that can act as a system language.

Having runtime compiler control is not only useful for modularization
and scripting, but also for dynamic code generation of OpenGL shaders
and utility parsers that always work with the language.

Otherwise things are either working good or are on a clear path to
getting good.  Nothing has made me feel that such is the wrong way to
go, although I'm opposed to templates.
Apr 26 2002
parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...

 It's about as easy to prototype in D as it is in Python, if a little
 easier (and that's saying something)... particularly now that I have
 Deed, of course.

Agreed. D is definitely much better in this area than C or C++, and this is a great achievement.
 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.

Default arguments for slicing would be very nice as well. It's nothing more than syntactic sugar, but it's convenient, why not add it?
 I want control of the parser and compiler at runtime.  I also want to
 be able to take a class instance and reconstruct its class definition
 perfectly.  I then want to be able to pick up dynamic modules and link
 them into the running program myself.  This is all for my never-ending
 quest for a scripting language that can act as a system language.

Wouldn't it bloat the resulting programs too much? I can understand the need for RTTI and dynamic construction of objects of unknown classes. But how are you going to control PARSER at run-time??? And linking modules dynamically... I think it's a bit too much... DLLs are there, isn't it enough?
Apr 27 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 11:06:00 +0400, "Pavel Minayev" <evilone omen.ru>
wrote:

"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...

 It's about as easy to prototype in D as it is in Python, if a little
 easier (and that's saying something)... particularly now that I have
 Deed, of course.

Agreed. D is definitely much better in this area than C or C++, and this is a great achievement.
 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.

Default arguments for slicing would be very nice as well. It's nothing more than syntactic sugar, but it's convenient, why not add it?

Sugar makes string processing go round. Out of 14 string slices in Deed, 13 use either 0 or the string length on the appropriate side, pretty typical.
 I want control of the parser and compiler at runtime.  I also want to
 be able to take a class instance and reconstruct its class definition
 perfectly.  I then want to be able to pick up dynamic modules and link
 them into the running program myself.  This is all for my never-ending
 quest for a scripting language that can act as a system language.

Wouldn't it bloat the resulting programs too much? I can understand the need for RTTI and dynamic construction of objects of unknown classes. But how are you going to control PARSER at run-time??? And linking modules dynamically... I think it's a bit too much... DLLs are there, isn't it enough?

What do you mean by bloat? If the parser's not linked in, it takes no space. The parser part of the compiler shouldn't take much space at all even when it is linked in. A parser class would mostly create tokens from an input stream, as well as have functions for reading in declarations and expressions and whatnot, plus a list of keywords. It doesn't have to be a particularly complex interface, just consistent with the language. So I could quickly pass through expressions and find any import statements no matter what future changes occur. Good tools to parse source code are awful hard to make correct; it'd be nice if the language itself standardized this. The compiler would produce Module, Class, or Function objects from a stream. The common expression compile should be done by building functions. It's for scripting. I'm sick and/or tired of having to use two languages in a project; the reason I'm here is as part of my exploration to get out of the paradigm. C/Python was the best I've ever tried, but getting the two languages to work together was a lot of work, and I had to put a lot of Python-applicable code in C merely because Python was so slow. Now that I've thought through it some more, I can see a way to make D behave for the most part, so dynamic linking control is not necessary. The only thing I need from here is a compiler and parser, actually, for in-game entities with code. But I'll let it simmer until I'm actually at the point that I could use it. Oh, Walter: Trying to compile mfile.d in Deed using -O causes an "Internal error: ..\ztc\gloop.c 1299". It appears to be in the listdir function, specifically the line "filename = (char[]) finder.cFileName;" Commenting this out removes the internal error, although a doppleganger of this line appears earlier in Stat's constructor.
Apr 27 2002
next sibling parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...

 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.

Parser of what? The program is already compiled to machine code!
 A parser class would mostly create tokens from an input stream, as
 well as have functions for reading in declarations and expressions and
 whatnot, plus a list of keywords.  It doesn't have to be a
 particularly complex interface, just consistent with the language.  So
 I could quickly pass through expressions and find any import
 statements no matter what future changes occur.  Good tools to parse
 source code are awful hard to make correct; it'd be nice if the
 language itself standardized this.

So, you propose to integrate a D interpreter into each program?
 The compiler would produce Module, Class, or Function objects from a
 stream.  The common expression compile should be done by building
 functions.  It's for scripting.  I'm sick and/or tired of having to
 use two languages in a project; the reason I'm here is as part of my
 exploration to get out of the paradigm.  C/Python was the best I've
 ever tried, but getting the two languages to work together was a lot
 of work, and I had to put a lot of Python-applicable code in C merely
 because Python was so slow.

I guess it's better to have an external D interpreter module (DScript or something), available as a part of Phobos. Not something built into the language itself though.
 The only thing I need from here is a compiler and parser, actually,
 for in-game entities with code.  But I'll let it simmer until I'm
 actually at the point that I could use it.

Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea. ... wanna try?
Apr 27 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone omen.ru>
wrote:

"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...

 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.

Parser of what? The program is already compiled to machine code!

Parser of D. You're missing what this would provide, simply: put the parser and compiler in a library and provide access to it. That's it, nothing weird about the language or anything, just a refactoring and an interface on the D side. [snip]
 The only thing I need from here is a compiler and parser, actually,
 for in-game entities with code.  But I'll let it simmer until I'm
 actually at the point that I could use it.

Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed a standard interpreter module could be done. Seems like a worthy idea.

Well, let's see. Of the 1930kb of C code in my current project, 1230kb is actual implementation and 700.39kb is devoted to nothing more than interfacing Python; that's 36% of the body. The Python code itself sums up to only 375kb. The balance is all wacky, definitely not acceptable, but very typical. With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.
... wanna try?

Been there, done that, have many, many T-shirts. If I were to do this I would write a compiler to machine code; apart from floating point they're not particularly hard to do dumbly and are easier to interact with the rest of the system. But then there's a lot of duplication of effort; it'd be like using a Tonka truck to dig a swimming pool when you have the keys to the backhoe in your pocket.
Apr 29 2002
next sibling parent "Pavel Minayev" <evilone omen.ru> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com...

Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.

Well, let's see. Of the 1930kb of C code in my current project, 1230kb is actual implementation and 700.39kb is devoted to nothing more than interfacing Python; that's 36% of the body. The Python code itself sums up to only 375kb. The balance is all wacky, definitely not acceptable, but very typical. With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.

This sounds like a special kind of computer language, quite different by the very concept from traditional all-purpose languages. Personally, I wouldn't like D to turn into something like that. It might be a useful tool, but not for everybody; not for me, at least.
Apr 29 2002
prev sibling next sibling parent reply "Richard Krehbiel" <rich kastle.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com...
 On Sat, 27 Apr 2002 20:58:32 +0400, "Pavel Minayev" <evilone omen.ru>
 wrote:

Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
a standard interpreter module could be done. Seems like a worthy
idea.

With runtime access to the D compiler and full introspection, on the other hand, getting the compiler to interact is pretty trivial, and anything I do will come with the guarantee that it will run as fast as my machine and compiler can make it.

Script languages usually have a set of problems to solve that differ from the problems of compiled languages. For in-game code, there is an additional difficulty: keeping the game player safe. Quake C and UnrealScript solve this problem. It's not possible to harm the gamer's computer with these languages. He/she can download mods (or have them automatically downloaded during on-line play) with a high level of comfort. If in-game code is in a native-compiled language, then you've got an opportunity to spread worms and viruses and cause real harm. Do you *really* want to subject your audience to malware like that? Just so that you have the convenience of coding the core game engine and logic in the same language? And I might even suggest another alternative: Java. The Java sandbox works, and JIT compilation works. You can use a Java native-code compiler for game engine performance, and load Java .class files for mods, which are then JIT-compiled. (Having said that, I know id Software experimented using Java for Quake 3 game logic, and performance considerations caused them to shelve it. Instead, Quake 3 builds in a C compiler; game mods are programmed in a safe C subset, which is compiled at load time to actual native machine code.) -- Richard Krehbiel, Arlington, VA, USA rich kastle.com (work) or krehbiel3 comcast.net (personal)
Apr 29 2002
parent reply "Walter" <walter digitalmars.com> writes:
"Richard Krehbiel" <rich kastle.com> wrote in message
news:aajmkt$qtu$1 digitaldaemon.com...
 (Having said that, I know id Software experimented using Java for Quake 3
 game logic, and performance considerations caused them to shelve it.
 Instead, Quake 3 builds in a C compiler; game mods are programmed in a

 C subset, which is compiled at load time to actual native machine code.)

That's an interesting idea. Does this C subset not allow pointers?
Apr 29 2002
parent reply Russell Borogove <kaleja estarcion.com> writes:
Walter wrote:
 "Richard Krehbiel" <rich kastle.com> wrote in message
 news:aajmkt$qtu$1 digitaldaemon.com...
 
(Having said that, I know id Software experimented using Java for Quake 3
game logic, and performance considerations caused them to shelve it.
Instead, Quake 3 builds in a C compiler; game mods are programmed in a

safe
C subset, which is compiled at load time to actual native machine code.)

That's an interesting idea. Does this C subset not allow pointers?

Quake 1 used QuakeC, which didn't allow pointers (and actually didn't allow a lot of things -- I wouldn't call it a C subset in fact; it was a vaguely C looking scripting language). If I remember rightly it was compiled to a tokenized or "bytecode"[1] form at game build time (or maybe at load time, it's been a while). Quake 2 used native DLLs. According to 5 minutes of Google research, it looks to me like Quake 3 also uses native DLLs: http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml I believe they decided to leave security as a "caveat player" thing in favor of performance and the ready availability of tools. -Russell B [1] Anyone else hate this term as much as I do?
Apr 29 2002
next sibling parent reply "Richard Krehbiel" <rich kastle.com> writes:
"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...
 Walter wrote:
 "Richard Krehbiel" <rich kastle.com> wrote in message
 news:aajmkt$qtu$1 digitaldaemon.com...

(Having said that, I know id Software experimented using Java for Quake



game logic, and performance considerations caused them to shelve it.
Instead, Quake 3 builds in a C compiler; game mods are programmed in a

safe
C subset, which is compiled at load time to actual native machine code.)

That's an interesting idea. Does this C subset not allow pointers?

Quake 1 used QuakeC, which didn't allow pointers (and actually didn't allow a lot of things -- I wouldn't call it a C subset in fact; it was a vaguely C looking scripting language). If I remember rightly it was compiled to a tokenized or "bytecode"[1] form at game build time (or maybe at load time, it's been a while). Quake 2 used native DLLs. According to 5 minutes of Google research, it looks to me like Quake 3 also uses native DLLs: http://www.planetquake.com/code3arena/tutorials/tutorial2.shtml

Um, I had forgotten that Quake 3 also allows game mods using .DLLs. But read the very bottom where it talks about QVMs.
 I believe they decided to leave security as a "caveat
 player" thing in favor of performance and the ready
 availability of tools.

Quake 2 was definitely programmed that way; game modes *are* done in native-code .DLLs. Here's a link that goes into more detail on Quake 3's mod system: http://firingsquad.gamers.com/hardware/q3adrivers/default.asp (I dimly recall John Carmack describing his QVM to native code JIT-compiler, but I can't find any links...) -- Richard Krehbiel, Arlington, VA, USA rich kastle.com (work) or krehbiel3 comcast.net (personal)
Apr 29 2002
parent Burton Radons <loth users.sourceforge.net> writes:
On Mon, 29 Apr 2002 15:03:15 -0400, "Richard Krehbiel"
<rich kastle.com> wrote:

"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...

 I believe they decided to leave security as a "caveat
 player" thing in favor of performance and the ready
 availability of tools.

Quake 2 was definitely programmed that way; game modes *are* done in native-code .DLLs. Here's a link that goes into more detail on Quake 3's mod system: http://firingsquad.gamers.com/hardware/q3adrivers/default.asp (I dimly recall John Carmack describing his QVM to native code JIT-compiler, but I can't find any links...)

It's in his .plan records, try (finger.quakeworld.com); I can't recall the date. However, it is entirely optional to do this. You can go through the full machine code route purely at your own option. Accountability and peer review pretty much eliminates the danger there. Can anyone think of one instance where this hole was exploited by a cracker? One? Otherwise understand if I'm dubious about this being a real problem... or whether it is relevant to us in any case. We have the compiler; we can define restricted compilation; it can be as secure as restricted execution in interpreted languages; it is no more difficult to make it secure than it is to make an interpreted language secure. If restricted compilation is not enough, uh, use an interpreted language. Better yet, convert your simplified scripting language to D and compile that. I haven't claimed a panacea here.
Apr 29 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...
 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?

LOL. I've written interpreters, and while bytecode minimizes object code size, it impairs runtime performance. Better to make your interpreter instructions word size and align them.
Apr 29 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Mon, 29 Apr 2002 14:54:06 -0700, "Walter" <walter digitalmars.com>
wrote:

"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...
 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?

LOL. I've written interpreters, and while bytecode minimizes object code size, it impairs runtime performance. Better to make your interpreter instructions word size and align them.

The FORTH way is to make these function pointers to call, then in the functions tail into the next instruction using jmp (gforth does this). Very fast stuff and smaller than native code, but bound to be eclipsed in the future by optimising native code compilers (bigforth does this).
Apr 29 2002
parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:bisrcu4ng7qcqartp3cec51mb851bqviep 4ax.com...
 On Mon, 29 Apr 2002 14:54:06 -0700, "Walter" <walter digitalmars.com>
 wrote:
"Russell Borogove" <kaleja estarcion.com> wrote in message
news:3CCD7F65.20705 estarcion.com...
 scripting language). If I remember rightly it was
 compiled to a tokenized or "bytecode"[1] form at game
 build time (or maybe at load time, it's been a while).
 [1] Anyone else hate this term as much as I do?

size, it impairs runtime performance. Better to make your interpreter instructions word size and align them.

functions tail into the next instruction using jmp (gforth does this). Very fast stuff and smaller than native code, but bound to be eclipsed in the future by optimising native code compilers (bigforth does this).

Ah yes, threaded code.
Apr 30 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
Burton Radons <loth users.sourceforge.net> wrote in 
news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

[snip]
 Parser of D.  You're missing what this would provide, simply: put the
 parser and compiler in a library and provide access to it.  That's it,
 nothing weird about the language or anything, just a refactoring and
 an interface on the D side.

You could use this to build a scripting engine into you program and it would be very fast. However there is a security issue with it depending on what you want to use it for. For example with games you may want to auto download game mods much like Unreal Tournament does. How do you prevent the mod writer from doing something malicious? You can certainly filter the functions that downloaded script can call but the main issue is that D still allows pointer manipulation. Given a pointer a clever coder can gain access to system level functions. So you need a more sandbox friendly version of D which ends up being a different language anyway. I do like the idea of having the compiler modules as libraries. I talked about something similar in a different thread here but I had different reasons for wanting it. Building the compiler in a modular way allows user defined processing to done between the compilation stages. [snip]
 Well, let's see.  Of the 1930kb of C code in my current project,
 1230kb is actual implementation and 700.39kb is devoted to nothing
 more than interfacing Python; that's 36% of the body.  The Python code
 itself sums up to only 375kb.  The balance is all wacky, definitely
 not acceptable, but very typical.

I've done a lot of C++/Python integration too. It's true that code to hook the two together is a pain. I eventually invented my own IDL language. I would run it through a script that would auto generate 99.9% of the hook code for me. Of course the runtime bulk is still there.
 Been there, done that, have many, many T-shirts.  If I were to do this
 I would write a compiler to machine code; apart from floating point
 they're not particularly hard to do dumbly and are easier to interact
 with the rest of the system.  But then there's a lot of duplication of
 effort; it'd be like using a Tonka truck to dig a swimming pool when
 you have the keys to the backhoe in your pocket.

Still I think a separate system is needed for this type of application. What I've really looking for is an embedable JIT compiler that also takes care of sandbox issues.
Apr 29 2002
next sibling parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:

 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

 [snip]
 Parser of D.  You're missing what this would provide, simply: put the
 parser and compiler in a library and provide access to it.  That's it,
 nothing weird about the language or anything, just a refactoring and
 an interface on the D side.

You could use this to build a scripting engine into you program and it would be very fast. However there is a security issue with it depending on what you want to use it for. For example with games you may want to auto download game mods much like Unreal Tournament does. How do you prevent the mod writer from doing something malicious? You can certainly filter the functions that downloaded script can call but the main issue is that D still allows pointer manipulation. Given a pointer a clever coder can gain access to system level functions. So you need a more sandbox friendly version of D which ends up being a different language anyway.

Hey, it can be D-- :) D, without pointers, without asm, with only access to a very carefully limited subset of the library, could potentially be a very good safe scripting language. I proposed (something very like) this back on 8/20/01 in "A Java-style sandbox without a Virtual Machine?", but there was a lot of confusion on what I meant by it :( I did get a pointer to Bochs from that thread, which was a real win :) Even if Walter doesn't feel like supporting it...we could do it. -- The Villagers are Online! villagersonline.com .[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ] .[ (a version.of(English).(precise.more)) is(possible) ] ?[ you want.to(help(develop(it))) ]
Apr 29 2002
next sibling parent reply Patrick Down <pat codemoon.com> writes:
Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
news:3CCD9200.64D5035D deming-os.org: 

 Patrick Down wrote:
 
 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:

 [snip]
 Parser of D.  You're missing what this would provide, simply: put
 the parser and compiler in a library and provide access to it. 
 That's it, nothing weird about the language or anything, just a
 refactoring and an interface on the D side.

You could use this to build a scripting engine into you program and it would be very fast. However there is a security issue with it depending on what you want to use it for. For example with games you may want to auto download game mods much like Unreal Tournament does. How do you prevent the mod writer from doing something malicious? You can certainly filter the functions that downloaded script can call but the main issue is that D still allows pointer manipulation. Given a pointer a clever coder can gain access to system level functions. So you need a more sandbox friendly version of D which ends up being a different language anyway.

Hey, it can be D-- :) D, without pointers, without asm, with only access to a very carefully limited subset of the library, could potentially be a very good safe scripting language. I proposed (something very like) this back on 8/20/01 in "A Java-style sandbox without a Virtual Machine?", but there was a lot of confusion on what I meant by it :( I did get a pointer to Bochs from that thread, which was a real win :) Even if Walter doesn't feel like supporting it...we could do it.

I would be cool to do this with D but it would require some restrictions. 1. Heavy control over imports 2. No pointers 3. No asm 4. Array bounds checking is always turned on.
Apr 29 2002
parent C.R.Chafer (DrWhat?) <blackmarlin nospam.asean-mail.com> writes:
Patrick Down wrote:

 
 I would be cool to do this with D but it would require some
 restrictions.
 
 1. Heavy control over imports
 2. No pointers
 3. No asm
 4. Array bounds checking is always turned on.
 

For a more theoritical discriptions of restrictions required you could try www.tunes.org . I believe one of their goals is the production of a HLL which could be checked at compile time disallowing access to privledged code - their site has been updated since last time I looked so I no longer have the link - it probably can be found under subprojects / HLL. C 2002/4/29
Apr 29 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
news:3CCD9200.64D5035D deming-os.org...
 Even if Walter doesn't feel like supporting it...we could do it.

It's not a matter of feeling like it, it's just that I'm supporting 4 compilers simultaneously, and I get stretched a little thin. People ask me why I don't play chess, etc. After working on compilers all night, I prefer mindless entertainment <g>. Anyhow, source to the D front end is now part of the distribution, and if anyone wants to morph that into an interpretted version, that would be a cool addition to D. Also, any gcc experts out there want to create a gcc version of the D compiler?
May 01 2002
parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Walter wrote:

 "Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
 news:3CCD9200.64D5035D deming-os.org...
 Even if Walter doesn't feel like supporting it...we could do it.

It's not a matter of feeling like it, it's just that I'm supporting 4 compilers simultaneously, and I get stretched a little thin. People ask me why I don't play chess, etc. After working on compilers all night, I prefer mindless entertainment <g>.

Sorry, I very definitely did NOT mean that to sound like you were lazy, random, or arbitrary. I am VERY happy that you put all this work into D. Like I said when I first arrived at this newsgroup...I'd been working on a spec very much like your D spec for a year or more, but I had exactly 0 lines of code actually written for it. (gets up on soapbox) Hear ye! Hear ye! Walter Bright is a good, hard working man! Let all gather together with me and express their great appreciation for his long hours and his willingness to give (at least some of it) away for free! Hear ye! Hear ye! (gets down, laughs at self)
 Anyhow, source to the D front end is now part of the distribution, and if
 anyone wants to morph that into an interpretted version, that would be a
 cool addition to D.

I'm going to have to look at the code you've written for that...is it architected where somebody could remove your code and drop in an alternative version, or would it be a pretty involved port? In a perfect world, there would be a parser module that turns D code into a recursive set of structures that could then be passed to any arbitrary backend: an interpreter, a GCC frontend, a dfront translator, or even a D-- compiler. -- The Villagers are Online! villagersonline.com .[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ] .[ (a version.of(English).(precise.more)) is(possible) ] ?[ you want.to(help(develop(it))) ]
May 01 2002
next sibling parent "Walter" <walter digitalmars.com> writes:
"Russ Lewis" <spamhole-2001-07-16 deming-os.org> wrote in message
news:3CCFF42C.A7C4906E deming-os.org...
 Sorry, I very definitely did NOT mean that to sound like you were lazy,

 or arbitrary.  I am VERY happy that you put all this work into D.  Like I

 when I first arrived at this newsgroup...I'd been working on a spec very

 like your D spec for a year or more, but I had exactly 0 lines of code

 written for it.

Ok!
 (gets up on soapbox)

 Hear ye!  Hear ye!  Walter Bright is a good, hard working man!  Let all

 together with me and express their great appreciation for his long hours

 his willingness to give (at least some of it) away for free!  Hear ye!

 ye!

 (gets down, laughs at self)

Let's not get carried away here!
 I'm going to have to look at the code you've written for that...is it
 architected where somebody could remove your code and drop in an

 version, or would it be a pretty involved port?

The source is for the front end only, so an alternative wouldn't make much sense.
 In a perfect world, there would be a parser module that turns D code into

 recursive set of structures that could then be passed to any arbitrary

 an interpreter, a GCC frontend, a dfront translator, or even a D--

May 01 2002
prev sibling parent Burton Radons <loth users.sourceforge.net> writes:
On Wed, 01 May 2002 06:57:01 -0700, Russ Lewis
<spamhole-2001-07-16 deming-os.org> wrote:

Walter wrote:

 Anyhow, source to the D front end is now part of the distribution, and if
 anyone wants to morph that into an interpretted version, that would be a
 cool addition to D.

I'm going to have to look at the code you've written for that...is it architected where somebody could remove your code and drop in an alternative version, or would it be a pretty involved port? In a perfect world, there would be a parser module that turns D code into a recursive set of structures that could then be passed to any arbitrary backend: an interpreter, a GCC frontend, a dfront translator, or even a D-- compiler.

Another speaks of the devil. Working on it now; lexer is done, expression parsing appears done, statement parsing is halfway done, still have to do class/struct/union/function parsing, then cleaning, then testing it to hell, then documentation. Few days perhaps for a beta? D in, uh, restricted compilation mode, doesn't even warrant a variant from what's been discussed. It could be done by making a kind of lint that confirms a file's validity under the stricter rules, then tells the caller whether it's all right to compile the code.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF8288ECF6Fpatcodemooncom 63.105.9.61...
 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:
 I've done a lot of C++/Python integration too.  It's true that code to
 hook the two together is a pain.  I eventually invented my own IDL
 language.
 I would run it through a script that would auto generate 99.9% of the
 hook
 code for me.  Of course the runtime bulk is still there.

What would it take to implement in D: extern (Python) int some_python_function(int foo); basically, make D really easy to interface to Python?
May 01 2002
next sibling parent reply Burton Radons <loth users.sourceforge.net> writes:
On Wed, 1 May 2002 03:42:54 -0700, "Walter" <walter digitalmars.com>
wrote:

"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF8288ECF6Fpatcodemooncom 63.105.9.61...
 Burton Radons <loth users.sourceforge.net> wrote in
 news:br5qcuo879plqcnr60avcraeamqu318phr 4ax.com:
 I've done a lot of C++/Python integration too.  It's true that code to
 hook the two together is a pain.  I eventually invented my own IDL
 language.
 I would run it through a script that would auto generate 99.9% of the
 hook
 code for me.  Of course the runtime bulk is still there.

What would it take to implement in D: extern (Python) int some_python_function(int foo); basically, make D really easy to interface to Python?

The more common direction, by a gigantic majority, is calling X from Python rather than Python from X. In my code the number of such cases can be counted on a hand that has fought with a wood chipper: three times, out of perhaps a thousand functions and methods. For all three times I don't see any way that they could be supported automatically; each has really specialized circumstances. Similarly classes are far more common than functions (although that will vary with the project, of course). Until recently you couldn't subclass builtin types in Python, and that would be easier to support. You could support Python subclassed types, however, by preprocessing a proxy class, and that would even allow calling Python-overloaded methods from D. So I think this should be alotted to a preprocessor that takes a D file and constructs an interface to Python, like SWIG, but more automatic. There's no magic bullet, that's for sure. D does help make it easier, but it's a lot of work in any case.
May 01 2002
parent Keith Ray <k1e2i3t4h5r6a7y 1m2a3c4.5c6o7m> writes:
In article <7oovcukrmtcvnvvlbuhh9s0vup7b9mk9nd 4ax.com>,
 Burton Radons <loth users.sourceforge.net> wrote:
[...]
 The more common direction, by a gigantic majority, is calling X from
 Python rather than Python from X.  In my code the number of such cases
 can be counted on a hand that has fought with a wood chipper: three
 times, out of perhaps a thousand functions and methods.
 
 For all three times I don't see any way that they could be supported
 automatically; each has really specialized circumstances.
 
 Similarly classes are far more common than functions (although that
 will vary with the project, of course).  Until recently you couldn't
 subclass builtin types in Python, and that would be easier to support.
 You could support Python subclassed types, however, by preprocessing a
 proxy class, and that would even allow calling Python-overloaded
 methods from D.
 
 So I think this should be alotted to a preprocessor that takes a D
 file and constructs an interface to Python, like SWIG, but more
 automatic.  There's no magic bullet, that's for sure.  D does help
 make it easier, but it's a lot of work in any case.

Interesting thing about Objective-C is how well it can work with Java, Ruby, and Python without needing pre-processing of code... The thing those four languages have in common is good reflection/introspection. For calling any of those languages from ObjectiveC, you can take advantage of the fact that polymorphism in ObjectiveC does NOT require inheritance. You can call any method on (that is, "send any message to") any object. If that object doesn't understand that method/message, then the run-time code will invoke the method "forwardInvocation:" on the object (if it exists), passing in as the argument information about the originally intended message and arguments. In that object's implementation of "forwardInvocation:", it can use functions in Java (via JNI), Python, Ruby, etc., to forward that call and arguments to that other language. The same technique is used for 'distributed objects' -- forwarding calls through a network to a remote computer. More information about this is in: <http://developer.apple.com/techpubs/macosx/Cocoa/ObjectiveC/5RunTime/For warding.html> <http://developer.apple.com/techpubs/macosx/Cocoa/ObjectiveC/4MoreObjC/iR emote_Messaging.html> You might ask, if Objective C is so wonderful, why don't more people use it. I think three reasons: One, early implementations of Objective C were proprietary and expensive, so it lost mindshare to C++, which was being given away (in the form of cfront) to universities by AT&T. Two, the 'object-oriented' portions of Objective C use a syntax similar to Smalltalk, which is looks alien to C programmers. Three, the current maintainer of the Objective C FAQ wrote his own 'non-standard' Objective C implementation, and hates Apple/Next. Since most questions on the objective C newsgroup are asked by people using Apple/Cocoa or NextStep, those message threads turn into flames-fests regularly. (Is there any other newsgroup where the FAQ maintainer is also the resident troll?) Today, Objective C is available in gcc, so the only thing preventing people from using it is their lack of familiarity (few environments require Objective C, so few people take the time to learn it) and the lack of good objective-c run-time library on the user's platform. For example, on MacOS 9, I'm not aware of any useful implementation of the Objective-c run-time library. On other platforms, the run-time library is there (provided by gnu), but it may not have the full breadth of capabilities of the one NeXT developed and that is now in Apple's MacOS X -- for example, the gnu implementation of objective-c run-time library may not have distributed (over networks) objects support. Even though the Objective C syntax is 'alien', it is simple to learn -- much simpler than C++. Of course, it doesn't have templates and operator-overloading, but that is what makes it easy to learn. People who rely on Objective C love the syntax, since it basically names each argument. [anInvocation invokeWithTarget:someOtherObject]; is equivalent to C++ anInvocation->invokeWithTarget( someOtherObject ); Now imagine that there are more arguments... [anObject setWidth: 12.0 height: 13.0 colorMode: 'gray' taxes: 'irs' filmMaker: 'woo' star: 'sirius' ]; is equivalent to anObject->setLotsOfStuff(12.0, 13.0, 'gray', 'irs', 'sirius', 'woo'); (or was that supposed to be: anObject->setLotsOfStuff(12.0, 13.0, 'gray', 'irs', 'woo', 'sirius'); ?) Of course, many IDEs these days let you look up the calling docs for a function or method easily, so this advantage is not as big a deal as it used to be. To get back to my point. I think the ideal language is simplified C++ with Objective C's runtime support, including the ability to "send a message" to any object and allow a class to implement 'forwardInvocation:' in order to catch unimplemented method-calls and handle them appropriately. This ability to catch unimplemented method-calls makes interfacing with dynamic languages like Python, Ruby, Java easy. -- C. Keith Ray <http://homepage.mac.com/keithray/xpminifaq.html>
May 01 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
"Walter" <walter digitalmars.com> wrote in news:aaogn4$2pau$1
 digitaldaemon.com:
 
 What would it take to implement in D:
 
     extern (Python) int some_python_function(int foo);
 
 basically, make D really easy to interface to Python?
 

First off you need classes too. extern(Python) class some_python_class { long some_python_method(long foo); } The way I usually handle this is to generate a helper class like the following. class some_python_class : public CPyObj { public: some_python_class() : CPyObj() {} some_python_class(some_python_class& rO) : CPyObj(rO) {} int some_python_method(long p1) { CPyObj args; args.CreateTuple(1); args.SetTupleItem(1,PyInt_FromLong(p1)); CPyObj meth = GetAttr("some_python_method"); if(!meth.IsValid()) { MethodNotFoundError("some_python_method"); return 0; } CPyObj rtn = meth.Call(args); return PyInt_AsLong(rtn); } }; CPyObj contains the handle to the real Python object and automatically deals with reference counting and other utility functions. However this is all trivial compared to calling C code from Python. You need to build tables mapping string names to function pointers. You need to fill out other data structures that define things like constructors, destructors, and where to fine attribute and method tables. You need code to type check and unpack parameters from Python's variant type and call the C function with them. Then type need to pack up the return value into a python object and return it. Are you letting Python control the lifetime of the object? Then you need code to track the reference count and delete it. "Programming Python" is a good reference it has a couple of chapters devoted to Python/C interfacing. I will append an example to the end of this post. In D you would need the equivalent reverse syntax. Something this might cover it. export(Python) int Func(int a, char[] b) { return a; } export(Python) class CFoo { export(Python) int Method(int a) { return a; } } However I will still go with my opinion that a tool outside the compiler is a better option. But I will say supporting such tools with some user defined attribute syntax like C# has might be a good thing. In C# custom attributes are actually classes derived from System.Attribute that can be discovered via reflection. But perhaps we don't need anything as complicated. defattr scriptable; scriptable int Func(int a,char[] b) { return a; } scriptable("constructor=no") class Foo { } The D front end parser would know how to parse the attribute syntax but wouldn't do anything with the information. A third party tool could use the D front end to create a code generator for interfacing the D module to a scripting engine, Python or anything else. ######################### Excerpts from a Python to C example. The example shown connects to a C++ class called CListControl derived from CBaseControl. class CListControl { public: virtual long AddString(const char*); virtual long InsertString(long,const char*); virtual long AddStringData(const char*,CPyObj); // Cut virtual long Get_SelectedIndex(); virtual void Set_SelectedIndex(long); virtual long Get_ListSize(); virtual void Set_ListSize(long); virtual long Get_Enabled(); virtual void Set_Enabled(long); // Cut }; This is rather old stuff from Python 2.0 days. 2.2 is a little bit easier to handle. The real cpp file is over 4000 lines long so I've cut out sections. Forgive me if it's rather ugly. :-) ######################### // GUIFramePy.h generated by pyintgen.py // Wed Sep 19 17:02:33 2001 // Cut some stuff here #include <PyObj.h> // Cut some stuff here struct CListControl_PyObj { PyObject_HEAD CListControl* m_pObj; }; // Cut some stuff here ######################## // GUIFramePy.cpp generated by pyintgen.py // Wed Sep 19 17:02:33 2001 // Cut some stuff here #include "listctrl.h" #include "GUIFramePy.h" static PyObject* pyErrorObj; // Cut some stuff here PyObject* CListControl_AddString(CListControl_PyObj* pObj, PyObject* pArgs) { const char* p1; if(!PyArg_ParseTuple(pArgs,"z" , &p1)) return 0; long rtn = pObj->m_pObj->AddString(p1); return PyInt_FromLong(rtn); } PyObject* CListControl_InsertString(CListControl_PyObj* pObj, PyObject* pArgs) { long p1; const char* p2; if(!PyArg_ParseTuple(pArgs,"iz" , &p1, &p2)) return 0; long rtn = pObj->m_pObj->InsertString(p1,p2); return PyInt_FromLong(rtn); } PyObject* CListControl_AddStringData(CListControl_PyObj* pObj, PyObject* pArgs) { const char* p1; CPyObj p2; if(!PyArg_ParseTuple(pArgs,"zO&" , &p1,Conv_PyObj, &p2)) return 0; long rtn = pObj->m_pObj->AddStringData(p1,p2); return PyInt_FromLong(rtn); } // Cut some stuff here PyMethodDef CListControl_methods[] = { { "AddString", (PyCFunction)CListControl_AddString, 1 }, { "InsertString", (PyCFunction)CListControl_InsertString, 1 }, { "AddStringData", (PyCFunction)CListControl_AddStringData, 1 }, { "InsertStringData", (PyCFunction)CListControl_InsertStringData, 1 }, { "DeleteItem", (PyCFunction)CListControl_DeleteItem, 1 }, { "DeleteAll", (PyCFunction)CListControl_DeleteAll, 1 }, { "SetData", (PyCFunction)CListControl_SetData, 1 }, { "GetData", (PyCFunction)CListControl_GetData, 1 }, { "GetText", (PyCFunction)CListControl_GetText, 1 }, { "LoadDirectory", (PyCFunction)CListControl_LoadDirectory, 1 }, { 0, 0, 0 } }; PyObject* CListControl_ObjAttr(CListControl_PyObj* self, char *pAttrName) { if(strcmp(pAttrName,"SelectedIndex") == 0) return PyInt_FromLong(self->m_pObj->Get_SelectedIndex()); else if(strcmp(pAttrName,"ListSize") == 0) return PyInt_FromLong(self->m_pObj->Get_ListSize()); else if(strcmp(pAttrName,"Enabled") == 0) return PyInt_FromLong(self->m_pObj->Get_Enabled()); else if(strcmp(pAttrName,"LD_READWRITE") == 0) return PyInt_FromLong(DDL_READWRITE); else if(strcmp(pAttrName,"LD_READONLY") == 0) return PyInt_FromLong(DDL_READONLY); else if(strcmp(pAttrName,"LD_HIDDEN") == 0) return PyInt_FromLong(DDL_HIDDEN); else if(strcmp(pAttrName,"LD_SYSTEM") == 0) return PyInt_FromLong(DDL_SYSTEM); else if(strcmp(pAttrName,"LD_DIRECTORY") == 0) return PyInt_FromLong(DDL_DIRECTORY); else if(strcmp(pAttrName,"LD_ARCHIVE") == 0) return PyInt_FromLong(DDL_ARCHIVE); else if(strcmp(pAttrName,"LD_DRIVES") == 0) return PyInt_FromLong(DDL_DRIVES); else if(strcmp(pAttrName,"LD_EXCLUSIVE") == 0) return PyInt_FromLong(DDL_EXCLUSIVE); return 0; } PyMethodChain CListControl_chain = { CListControl_methods, &CBaseControl_chain }; PyObject* CListControl_getattr(CListControl_PyObj* self, char *pAttrName) { PyObject* pyObjRtn = CListControl_ObjAttr(self,pAttrName); if(pyObjRtn) return pyObjRtn; // Really nasty solution to a multiple inheritance problem // Fix this someday CListControl* pTmp = self->m_pObj; ((CBaseControl_PyObj*)self)->m_pObj = (CBaseControl*)self->m_pObj; pyObjRtn = CBaseControl_ObjAttr((CBaseControl_PyObj*)self,pAttrName); self->m_pObj = pTmp; if(pyObjRtn) return pyObjRtn; return Py_FindMethodInChain(&CListControl_chain,(PyObject*) self,pAttrName); } int CListControl_setattr(CListControl_PyObj* self, char *pAttrName, PyObject* pyValue) { if(strcmp(pAttrName,"SelectedIndex") == 0) { self->m_pObj->Set_SelectedIndex(static_cast<long>(PyInt_AsLong (pyValue))); return 0; } else if(strcmp(pAttrName,"Enabled") == 0) { self->m_pObj->Set_Enabled(static_cast<long>(PyInt_AsLong(pyValue))); return 0; } return -1; } // Cut some stuff here void CListControl_dealloc(CListControl_PyObj* self) { MODTRACE("CListControl(%p) dereference\n",self); if(self->m_pObj) self->m_pObj->ResetPyObject(); #if 0 delete self->m_pObj; #endif PyMem_DEL(self); } PyTypeObject CListControl_type = { PyObject_HEAD_INIT(&PyType_Type) 0, "CListControl", sizeof(CListControl_type), 0, (destructor) CListControl_dealloc, 0, (getattrfunc) CListControl_getattr, (setattrfunc) CListControl_setattr, 0, 0, 0, 0, 0, 0, 0, 0 }; // Cut some stuff here int Conv_CListControl(PyObject* pyObj,CListControl** pVar) { if(pyObj->ob_type == &CListControl_type) { *pVar = ((CListControl_PyObj*)pyObj)->m_pObj; return 1; } else if(pyObj == Py_None) { *pVar = 0; return 1; } PyErr_SetString(pyErrorObj,"Function requires a CListControl"); return 0; } // Cut some stuff here // Module level functions PyMethodDef AppFrame_methods[] = { { "RunApp", (PyCFunction)RunApp_PyFunc, 1 }, { "ShowDialog", (PyCFunction)ShowDialog_PyFunc, 1 }, { "FileBrowse", (PyCFunction)FileBrowse_PyFunc, 1 }, { "ShellExec", (PyCFunction)ShellExec_PyFunc, 1 }, {0, 0, 0} }; void initAppFrame() { PyObject* pMod; PyObject* pDict; pMod = Py_InitModule("AppFrame", AppFrame_methods); pDict = PyModule_GetDict(pMod); pyErrorObj = Py_BuildValue("s","AppFrame.error"); PyDict_SetItemString(pDict,"error",pyErrorObj); if(PyErr_Occurred()) Py_FatalError("Can't initialize module AppFrame"); else InstallDataStruct(); }
May 01 2002
parent "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns9201B2282EE82patcodemooncom 63.105.9.61...
 However this is all trivial compared to calling C code from
 Python.  You need to build tables mapping string names to
 function pointers.  You need to fill out other data structures that
 define things like constructors, destructors, and where to fine
 attribute and method tables.  You need code to type check and unpack
 parameters from Python's variant type and call the C function with
 them.  Then type need to pack up the return value into a
 python object and return it.  Are you letting Python control
 the lifetime of the object?  Then you need code to track the
 reference count and delete it.  "Programming Python" is a good
 reference it has a couple of chapters devoted to Python/C interfacing.
 I will append an example to the end of this post.

Oh darn. Sounds like a project for another time.
May 02 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in news:aael3n$18ql$1
 digitaldaemon.com:

 
 Oh, now I see. Sorta UnrealScript, right? Well, then, as I proposed
 a standard interpreter module could be done. Seems like a worthy
 idea.
 
 ... wanna try?

I like the idea of a standard interpreter module although as I said in a different thread here what I really want is a embeddable JIT compiler module. But in the mean time I'll take a regular old script interpreter. I've thought about doing this for D if I have some spare time. I've had a lot of success building applications that mix C++ and Python. I usually let the Python code handle the GUI stuff like dialogs and it also acts as a glue layer between the C++ modules. There are a couple of problems. 1) The amount of interface code that is needed to tie the interpreter to the C++ code. I have scripts that will automatically generate a lot of the code. But the interface code is still bulky. 2) Python's objects are garbage collected via reference counting. Any C++ code that references one of these objects had to manually deal with the reference count. And C++ object that is reference from Python has to supply a reference count for the interpreter. 3) This is less of a problem and more of a personal preference on my part. I like Python's power I still don't like it's syntax. I just can't handle using indention as the only indicator of code blocks. :-) Some of these problems could be solved with a scripting language interpreter written in D that uses extension modules written in D. Number 2 is easy because the interpreter will use D's garbage collector. All the memory management problems go away because the interpreter and D will use common memory management. Number 1 is not quite so easy to solve. Having the memory management issue solved does away with some of the interface code but there are still issues with marshalling function parameters between the interpreter and D code. There are two sides to this. Calling D functions from the script and calling script functions from D. Having a script language construct like the "native" directive in Unreal Script would allow the interpreter to automatically generate stack frames for calls from script into D. The interpreter could also use reflection to find the function pointers in a module. However, there are issues with controlling which functions you want exported to the script interpreter. If D allowed user defined attribute tagging of classes and functions like C# then this problem could be solved by adding a "scriptexport" attribute to things in D that you wanted to export to the interpreter. Another way is just to hand a list of exported functions and classes to the interpreter when it is initialized. Calling script functions from D is still messy but manageable with some helper functions.
Apr 29 2002
next sibling parent Nuno Nunes <nachn altagis.biz> writes:
 
 I like Python's power I still don't like it's syntax.  I just can't
 handle using indention as the only indicator of code blocks. :-)
 

I think Ruby tries to cover this bad feature (at least to me), and as a better integration with native code (C). But it as few (far few) support from libraries
Apr 29 2002
prev sibling next sibling parent reply Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:

 Number 1 is not quite so easy to solve.  Having the memory management
 issue solved does away with some of the interface code but there are
 still issues with marshalling function parameters between the interpreter
 and D code.  There are two sides to this.  Calling D functions from the
 script and calling script functions from D.

I don't know Python, so this is just a shot in the dark, but...could you define a "Python" calling convention? D already supports multiple calling conventions, and this could be another: extern(Python) void MyFunc(...); Thus, you can take functions exported from Python and call them directly from D with no wrapper code (the compiler generates the wrapper); similarly, you can export function TO Python and have Python call your D code. -- The Villagers are Online! villagersonline.com .[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ] .[ (a version.of(English).(precise.more)) is(possible) ] ?[ you want.to(help(develop(it))) ]
Apr 29 2002
next sibling parent Nuno Nunes <nachn altagis.biz> writes:
 
 I don't know Python, so this is just a shot in the dark, but...could you
 define a "Python" calling convention?  D already supports multiple calling
 conventions, and this could be another:
 
     extern(Python) void MyFunc(...);
 

Apr 29 2002
prev sibling parent reply Patrick Down <pat codemoon.com> writes:
Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
news:3CCDB223.17897881 deming-os.org: 

 Patrick Down wrote:
  
 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:
 
     extern(Python) void MyFunc(...);

Yes but there are a lot of scripting languages out there. I wouldn't ask Walter to support all of them, even Python. :-) It's a task better left to an third party tool.
Apr 29 2002
next sibling parent Russ Lewis <spamhole-2001-07-16 deming-os.org> writes:
Patrick Down wrote:

 Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
 news:3CCDB223.17897881 deming-os.org:

 Patrick Down wrote:

 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:

     extern(Python) void MyFunc(...);

Yes but there are a lot of scripting languages out there. I wouldn't ask Walter to support all of them, even Python. :-) It's a task better left to an third party tool.

I'm not sure there isn't a way to integrate 3rd party plug-ins into a compiler...what if you could just download a plugin that would add support for the Python calling convention to your D compiler? -- The Villagers are Online! http://villagersonline.com .[ (the fox.(quick,brown)) jumped.over(the dog.lazy) ] .[ (a version.of(English).(precise.more)) is(possible) ] ?[ you want.to(help(develop(it))) ]
Apr 29 2002
prev sibling parent "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FFAE8EF54EFpatcodemooncom 63.105.9.61...
 Russ Lewis <spamhole-2001-07-16 deming-os.org> wrote in
 news:3CCDB223.17897881 deming-os.org:

 Patrick Down wrote:

 I don't know Python, so this is just a shot in the dark, but...could
 you define a "Python" calling convention?  D already supports multiple
 calling conventions, and this could be another:

     extern(Python) void MyFunc(...);

Yes but there are a lot of scripting languages out there. I wouldn't ask Walter to support all of them, even Python. :-) It's a task better left to an third party tool.

I've received many requests for supporting an interface to Python. Interestingly, people who find D appealing also find Python appealing for their scripting needs.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Patrick Down" <pat codemoon.com> wrote in message
news:Xns91FF9D202AFD6patcodemooncom 63.105.9.61...
 I like the idea of a standard interpreter module although as I said
 in a different thread here what I really want is a embeddable JIT
 compiler module.  But in the mean time I'll take a regular old script
 interpreter.  I've thought about doing this for D if I have some
 spare time.

What do people think of adding an ECMAScript interpreter as a Phobos library routine? (ECMAScript is the official name for javascript, jscript, etc.)
May 01 2002
parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Walter" <walter digitalmars.com> wrote in message
news:aaoh8b$2qe7$2 digitaldaemon.com...

 What do people think of adding an ECMAScript interpreter as a Phobos

 routine? (ECMAScript is the official name for javascript, jscript, etc.)

You mean, that one you've written? Anyhow, YES. Some standartized way to do scripting would be just great. It's nice to have such a handy tool as a part of distribution: no need to search for 3rd-party libs, try to compile them, then find out they crash every second call...
May 01 2002
parent reply "Walter" <walter digitalmars.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in message
news:aaovcd$t5r$1 digitaldaemon.com...
 "Walter" <walter digitalmars.com> wrote in message
 news:aaoh8b$2qe7$2 digitaldaemon.com...
 What do people think of adding an ECMAScript interpreter as a Phobos

 routine? (ECMAScript is the official name for javascript, jscript, etc.)


Yes. Gotta find some use for it <g>.
 Anyhow, YES. Some standartized way to do scripting would be just great.
 It's nice to have such a handy tool as a part of distribution: no need
 to search for 3rd-party libs, try to compile them, then find out they
 crash every second call...

The interpreter has been well tested, and is pretty solid.
May 01 2002
next sibling parent reply "Pavel Minayev" <evilone omen.ru> writes:
"Walter" <walter digitalmars.com> wrote in message
news:aapcpr$1mt7$1 digitaldaemon.com...

 routine? (ECMAScript is the official name for javascript, jscript,



 You mean, that one you've written?

Yes. Gotta find some use for it <g>.

Well, you have to port it to D first... =)
 Anyhow, YES. Some standartized way to do scripting would be just great.
 It's nice to have such a handy tool as a part of distribution: no need
 to search for 3rd-party libs, try to compile them, then find out they
 crash every second call...

The interpreter has been well tested, and is pretty solid.

That's what I mean. Just "import script.ecma" - nothing more needed. I like it! =)
May 01 2002
parent "Walter" <walter digitalmars.com> writes:
"Pavel Minayev" <evilone omen.ru> wrote in message
news:aapeo8$1pes$1 digitaldaemon.com...
 "Walter" <walter digitalmars.com> wrote in message
 news:aapcpr$1mt7$1 digitaldaemon.com...

 routine? (ECMAScript is the official name for javascript, jscript,



 You mean, that one you've written?

Yes. Gotta find some use for it <g>.

Well, you have to port it to D first... =)

It would be some work to interface it to D, but I am just not up to rewriting it.
May 02 2002
prev sibling parent Patrick Down <pat codemoon.com> writes:
"Walter" <walter digitalmars.com> wrote in
news:aapcpr$1mt7$1 digitaldaemon.com: 

 
 "Pavel Minayev" <evilone omen.ru> wrote in message
 news:aaovcd$t5r$1 digitaldaemon.com...
 "Walter" <walter digitalmars.com> wrote in message
 news:aaoh8b$2qe7$2 digitaldaemon.com...
 What do people think of adding an ECMAScript interpreter as a
 Phobos 

 routine? (ECMAScript is the official name for javascript, jscript,
 etc.) 


Yes. Gotta find some use for it <g>.

This would be really cool.
May 01 2002
prev sibling parent reply "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...
 On Sat, 27 Apr 2002 11:06:00 +0400, "Pavel Minayev" <evilone omen.ru>
 wrote:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:4kekcu4p1pbtp790rvugkbatvln5b3ra2l 4ax.com...
 Array slicing should use default arguments; [ .. ] refers to the full
 array, [x .. ] refers to x to the end of the array, [ .. x] refers to
 the start of the array to x.  It's convenient.  I also like Python's
 negative array indexing, but I can see many reasons to oppose it.

more than syntactic sugar, but it's convenient, why not add it?

Deed, 13 use either 0 or the string length on the appropriate side, pretty typical.

This is a good idea.
 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.

Are you thinking of something like javascript's exec function?
 Oh, Walter: Trying to compile mfile.d in Deed using -O causes an
 "Internal error: ..\ztc\gloop.c 1299".  It appears to be in the
 listdir function, specifically the line "filename = (char[])
 finder.cFileName;"  Commenting this out removes the internal error,
 although a doppleganger of this line appears earlier in Stat's
 constructor.

Ok, could you email me please a reproducible example?
Apr 27 2002
parent reply Burton Radons <loth users.sourceforge.net> writes:
On Sat, 27 Apr 2002 09:50:25 -0700, "Walter" <walter digitalmars.com>
wrote:

"Burton Radons" <loth users.sourceforge.net> wrote in message
news:q5tkcus6dleacm81j2v6jl9h45fri6qbgg 4ax.com...

 What do you mean by bloat?  If the parser's not linked in, it takes no
 space.  The parser part of the compiler shouldn't take much space at
 all even when it is linked in.

Are you thinking of something like javascript's exec function?

You mean eval? No - t'would be a pain to get the local scope into the dynamic code (and not really be desirable) and the type system would interfere. Better would be to construct a function in text, compile that, then execute it. For the parser, I just mean the ability to get tokens and parse declarations, nothing more. Just a few judicious links into the frontend as it stands, with the frontend refactored into a library. So, for a minimal example: class DLexer { this (wchar[] text); int LineNo (); DToken Token (); } enum DTokenType { ... } class DToken { DTokenType type; DIdentifier ident; } class DIdentifier { int value; wchar[] string; } With additional features added later. I'm not proposing suddenly jumping into a full-fledged feature-complete parser, but just a start that I can add things to. The only part of the work I can't do is putting the frontend into a library that the backend uses. I'm volunteering my time for the rest.
 Oh, Walter: Trying to compile mfile.d in Deed using -O causes an
 "Internal error: ..\ztc\gloop.c 1299".  It appears to be in the
 listdir function, specifically the line "filename = (char[])
 finder.cFileName;"  Commenting this out removes the internal error,
 although a doppleganger of this line appears earlier in Stat's
 constructor.

Ok, could you email me please a reproducible example?

This is the smallest fragment I could get it into: void listdir() { char[14] foo; char[] bar; while (1) bar = foo; } Using "\dmd\bin\dmd -I/dmd/src/phobos file.d -O" to compile, version 0.28.
Apr 27 2002
parent "Walter" <walter digitalmars.com> writes:
Thanks, I can reproduce the error. I'll see about a fix.
Apr 27 2002
prev sibling next sibling parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:tg9jcu418udak1jp1g1d98g82fqjfk9or3 4ax.com...
 There's more.  array, memset, obj, and switcherr are not included in
 the distribution but symbols from them are used.

That's a mistake. I'll put them in.
  interface.c,

Obsolete, I'll remove it.
 intrinsic.d,

Just declarations. No need to compile it.
 iunknown.d,

Needs to be added to the library, I'll fix it.
 and time.d


Just declarations, no need to compile it.
 are not touched by the makefile.

 Finally, sc compiles the .asm files incorrectly (it adds a semicolon
 to the filename, bizarrely enough), and the version of masm and link I
 got don't like the options used.

Unfortunately, Microsoft keeps changing how MASM works. Older versions did require the ;. It's a constant problem, and one reason why D has its own inline assembler.
 It'd be nice if the obj file were
 just stuck in the distribution.

Good idea. I'll do it.
Apr 27 2002
prev sibling parent "Walter" <walter digitalmars.com> writes:
"Burton Radons" <loth users.sourceforge.net> wrote in message
news:tg9jcu418udak1jp1g1d98g82fqjfk9or3 4ax.com...
 This requires "ERROR_NO_MORE_FILES = 18," in the appropriate place in
 windows.d and FileTimeToSystemTime using "export BOOL
 FileTimeToSystemTime(FILETIME* lpFileTime, SYSTEMTIME*
 lpSystemTime);".

Done.
Apr 27 2002