www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Python integration going well

reply David Rushby <David_member pathlink.com> writes:
D's a breeze.  To fuse link-level C compatibility and high performance into a
reasonably high-level language *without descending into a cesspit of complexity*
is a major accomplishment.

I'm grateful to Deja for his/her translation of the Python header files; that
saved me a lot of work.  Having said that, the translation was clearly in a
pre-alpha state.

Initally, I couldn't even create a Python extension module containing a function
that "did nothing", because the global variables that Python expects extensions
to manipulate weren't exposed to D.  The initial translation had fallen into the
trap of trying to provide access to global variables in a C library, while in
fact creating a distinct set of variables with the same names in D.  I tried the
"place the declarations in a secondary D module and import that module, but
don't link it" workaround suggested in numerous past threads on this forum, but
it didn't work.  The D pointers were initialized to *something* non-null, but
Python segfaulted upon any attempt to use them.  Eventually I gave up and wrote
code in the python.d module's static initializer that traverses the Python
module structure via the Python/C API and extracts pointers to the requisite
global variables.  That works fine, though it'll be a bit of a maintenance
burden.

Another problem with the initial translation was that it assumed that Python's
internal Unicode representation is UCS-2.  In fact, Python can be compiled in
any of the following configs:
1) without any Unicode support
2) using UCS-2 as the internal repr
3) using UCS-4 as the internal repr

The most common config, including that of the standard Windows binaries and many
Linux distributions, is #2.  However, RedHat-derived Linux distributions
typically use #3.  This code will need to be conditionalized.

Also, the initial translation used the D type int in areas of the Python/C API
that expect a C long.  That's fine on x86, and on x86_64-Windows, but is not
portable to most other 64-bit operating systems, including x86_64-Linux.  This
problem should be easy to fix with a conditional typedef and minor tweaks.

---

All things considered, I'm astonished at how easy it was to go from near-total
ignorance of D to a working--albeit trivial--Python extension written in D.

I plan to address the issues I outlined above, then have a go at writing Python
distutils support for extension modules written in D.  (distutils is the
standard Python infrastructure for the compilation and
distributable-package-creation of extension modules.)  After that, I plan to
write a tutorial on the subject.  In the not-too-distant future, I expect to
have polished the infrastructural aspects of writing Python extension modules in
D nearly to the extent that they are today for C (ugh, what a sentence... need
sleep).

Thanks to Deja Augustine for performing the initial translation of the Python
header files to D, to Justin Calvarese for digging up that translation after it
had disappeared from the web, and most of all to Walter for designing and
implementing D.  The language has great potential in general, and in particular,
as an extension language for Python.

Just in case I get run over by a truck tomorrow, here's a snapshot of what I
have so far:
http://kinterbasdb.sourceforge.net/other/d/python-extension-written-in-d-2005_10_20_00_38.zip
Oct 19 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
This is great stuff! Thanks for taking charge of this and getting it done.
Oct 19 2005
parent reply David Rushby <David_member pathlink.com> writes:
Walter Bright wrote:
 This is great stuff! Thanks for taking charge of this and 
 getting it done.

You're welcome. I think it's wise of you to be so supportive of efforts to integrate D with the so-called "scripting languages". Although a few of those languages (Python and Ruby foremost among them) provide good support for the construction of non-trivial programs, they're all plagued by slow execution speed, which creates strong demand for high-performance extensions. Folks who're already using a so-called scripting language for serious programming have recognized the value of most of the features that differentiate D from C++, so I'd expect them to be receptive to D. Furthermore, those of us who're using a so-called scripting language as an alternative to Java or C# have a considerable degree of control over our choice of languages, so there are fewer bureaucratic barriers to D adoption among the "scripting language" community. It's hard to tell whether D can be made approachable by Python programmers who've never written a C extension. They still have to use a C API to interact with Python, after all. But Pyrex (http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ ), a language that "lets you write code that mixes Python and C data types any way you want, and compiles it into a C extension for Python", seems to be popular among Python programmers who don't know C. Since I know C and the Python/C API well, I'll probably settle for finishing (and maintaining) the header translation, writing distutils (build infrastructure) support, and creating a basic tutorial. It'd be interesting to see whether a creative person could make it comfortable for Python programmers to write extensions in D without using the Python/C API at all. --- Here's what I've done since my last message on the 20th: - Annotated each section of the python.d header translation to make it easy to trace the D header code back to the original Python C header file. This will make maintenance of the D header file easier. - Updated the translation to reflect additions made to the Python/C API in Python 2.4, which is the current version of Python. The initial translation was based on Python 2.3. - The following header files were missing from the initial translation, but have now been translated: - cStringIO.h - datetime.h - eval.h - genobject.h - marshal.h - pystrtod.h - pythread.h - setobject.h - structmember.h - structseq.h As far as I know, all Python/C API header content that could conceivably be useful in D programmer has now been translated. Some areas of the API are not covered because it seems likely that any extensions that would use these advanced facilities would be written in C. For example, a non-Python-core program that creates and manipulates parse trees for Python code would almost certainly be written in Python itself; only the Python core is likely to implement such functionality in a native extension, and that would be in C, not D. - Used D's mixin feature instead of manually expanding the field header macros _PyObject_HEAD_EXTRA, PyObject_HEAD, PyObject_VAR_HEAD, and PyDescr_COMMON. In addition to easing maintenance, this will make it possible for extensions written in D to run under a debug build of the Python interpreter, which adds extra fields to every PyObject for diagnostic purposes. - Applied a consistent renaming scheme where it was necessary to circumvent D keywords (I used the renaming convention that's most popular in the Python community, which is to add a trailing underscore). - Fixed all problems with 64-bit portability. - Implemented dynamic type checking macros as D functions. These tend to be used heavily in Python/C API code, but the intial translation omitted them. - Fixed numerous other omissions and a few errors. The following issues remain unresolved, though I plan to deal with them in due course (after I've written distutils support for D, so that there's a build framework within which to operate): - Need to conditionalize UCS2-versus-UCS4 Python Unicode API functions based on Python distutils control over the D compilation process. - During this step, need to review Python's unicodeobject.h to ensure that no members were omitted from the Deja's initial translation. - In a build process controlled by Python distutils, need to detect whether the Python interpreter was built in debug build mode, and if so, make the appropriate adjustments to the header mixins. I'll continue to post snapshots at http://kinterbasdb.sourceforge.net/other/d/ in case I get run over by the proverbial truck.
Oct 23 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...
 I'll continue to post snapshots at

 in case I get run over by the proverbial truck.

Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(
Oct 25 2005
parent reply David Rushby <David_member pathlink.com> writes:
In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says...
"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...
 I'll continue to post snapshots at

 in case I get run over by the proverbial truck.

Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(

The only reason I'm currently placing the files at http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that web space. kinterbasdb is not related to D in any way. I believe in the value of documentation, and have always planned to document Python/D integration when I am confident that the infrastructure is sufficiently mature. (I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English. That is atypical and quite refreshing.) Before I reach the stage where I consider Python/D integration worth documenting or publicizing, though, I need to: 1) Make sure there are no fundamental stumbling blocks (such as digitalmars.D/29399 ). 2) Have implemented the infrastructure that Python programmers expect (chiefly distutils support). #1 will be something of an ongoing process, but I've already encountered stumbling blocks. I couldn't seriously advocate D as an extension language for Python when there's no (reliable) way to use it as such under Linux. I'm willing to put forth effort myself to correct this, assuming that it doesn't require months of preparatory study. #2 I'm currently working on.
Oct 25 2005
next sibling parent John Demme <me teqdruid.com> writes:
In regards to #1 (the shared libraries), I'm really looking forward to
this D integration as well as shared library support (keeping in mind that
I don't even have a copy of Windows on my computer).

Walter, since you seem interested in this project, this should be a good
motivation to get the shared library support working under Linux.  Right? 
BTW, did you read Georg's recent "Policy opinion" post?  I agree with it,
although I'm pretty biased.

~John Demme

On Tue, 25 Oct 2005 19:35:05 +0000, David Rushby wrote:

 In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says...
"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...
 I'll continue to post snapshots at

 in case I get run over by the proverbial truck.

Your post is full of great information. When you are ready, could you distill it all into a "How To" document and tutorial, and then we can put it on the D web site? That would give it much more exposure, as a random project on sourceforge isn't going to be run across by the people who need it :-(

The only reason I'm currently placing the files at http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that web space. kinterbasdb is not related to D in any way. I believe in the value of documentation, and have always planned to document Python/D integration when I am confident that the infrastructure is sufficiently mature. (I'd like to compliment you on the quality of D's documentation. Not only is it broad in its coverage, but it's written in fluent English. That is atypical and quite refreshing.) Before I reach the stage where I consider Python/D integration worth documenting or publicizing, though, I need to: 1) Make sure there are no fundamental stumbling blocks (such as digitalmars.D/29399 ). 2) Have implemented the infrastructure that Python programmers expect (chiefly distutils support). #1 will be something of an ongoing process, but I've already encountered stumbling blocks. I couldn't seriously advocate D as an extension language for Python when there's no (reliable) way to use it as such under Linux. I'm willing to put forth effort myself to correct this, assuming that it doesn't require months of preparatory study. #2 I'm currently working on.

Oct 25 2005
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...
 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.

 atypical and quite refreshing.)

Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.
 Before I reach the stage where I consider Python/D integration worth

 or publicizing, though, I need to:

 1) Make sure there are no fundamental stumbling blocks (such as
 digitalmars.D/29399 ).

 2) Have implemented the infrastructure that Python programmers expect

 distutils support).


 #1 will be something of an ongoing process, but I've already encountered
 stumbling blocks.  I couldn't seriously advocate D as an extension

 Python when there's no (reliable) way to use it as such under Linux.  I'm
 willing to put forth effort myself to correct this, assuming that it

 require months of preparatory study.

 #2 I'm currently working on.

I don't know how much work #1 will be, but it's time to look into it.
Oct 25 2005
parent reply J C Calvarese <technocrat7 gmail.com> writes:
In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says...
"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...
 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.

 atypical and quite refreshing.)

Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.

I don't recall anyone saying it was bad. Sometimes people contain there's just not enough of it. ;) But I guess Ddoc should definitely help with that. I think having Ddoc built into the compiler is smart and it's going to be a terrific boon for D and D users. I suppose you might be thinking of when people complained about the formating of the web pages, but you'll probably have a lot fewer complaints now. I think they look pretty slick these days! Thanks for fixing them up. jcc7
Oct 25 2005
parent J C Calvarese <technocrat7 gmail.com> writes:
In article <djn7ug$8h8$1 digitaldaemon.com>, J C Calvarese says...
In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says...
"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...
 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.

 atypical and quite refreshing.)

Thanks for the kind words, most people say the documentation stinks <g>. I actually agree with them, and making the documentation better was the inspiration for ddoc.

I don't recall anyone saying it was bad. Sometimes people contain there's just

I guess I meant "complain" rather than "contain". Perhaps it's time for me to go to bed? jcc7
Oct 26 2005