www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Python integration going well

reply David Rushby <David_member pathlink.com> writes:
D's a breeze.  To fuse link-level C compatibility and high performance into a
reasonably high-level language *without descending into a cesspit of complexity*
is a major accomplishment.

I'm grateful to Deja for his/her translation of the Python header files; that
saved me a lot of work.  Having said that, the translation was clearly in a
pre-alpha state.

Initally, I couldn't even create a Python extension module containing a function
that "did nothing", because the global variables that Python expects extensions
to manipulate weren't exposed to D.  The initial translation had fallen into the
trap of trying to provide access to global variables in a C library, while in
fact creating a distinct set of variables with the same names in D.  I tried the
"place the declarations in a secondary D module and import that module, but
don't link it" workaround suggested in numerous past threads on this forum, but
it didn't work.  The D pointers were initialized to *something* non-null, but
Python segfaulted upon any attempt to use them.  Eventually I gave up and wrote
code in the python.d module's static initializer that traverses the Python
module structure via the Python/C API and extracts pointers to the requisite
global variables.  That works fine, though it'll be a bit of a maintenance
burden.

Another problem with the initial translation was that it assumed that Python's
internal Unicode representation is UCS-2.  In fact, Python can be compiled in
any of the following configs:
1) without any Unicode support
2) using UCS-2 as the internal repr
3) using UCS-4 as the internal repr

The most common config, including that of the standard Windows binaries and many



Also, the initial translation used the D type int in areas of the Python/C API
that expect a C long.  That's fine on x86, and on x86_64-Windows, but is not
portable to most other 64-bit operating systems, including x86_64-Linux.  This
problem should be easy to fix with a conditional typedef and minor tweaks.

---

All things considered, I'm astonished at how easy it was to go from near-total
ignorance of D to a working--albeit trivial--Python extension written in D.

I plan to address the issues I outlined above, then have a go at writing Python
distutils support for extension modules written in D.  (distutils is the
standard Python infrastructure for the compilation and
distributable-package-creation of extension modules.)  After that, I plan to
write a tutorial on the subject.  In the not-too-distant future, I expect to
have polished the infrastructural aspects of writing Python extension modules in
D nearly to the extent that they are today for C (ugh, what a sentence... need
sleep).

Thanks to Deja Augustine for performing the initial translation of the Python
header files to D, to Justin Calvarese for digging up that translation after it
had disappeared from the web, and most of all to Walter for designing and
implementing D.  The language has great potential in general, and in particular,
as an extension language for Python.

Just in case I get run over by a truck tomorrow, here's a snapshot of what I
have so far:
http://kinterbasdb.sourceforge.net/other/d/python-extension-written-in-d-2005_10_20_00_38.zip
Oct 19 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
This is great stuff! Thanks for taking charge of this and getting it done.
Oct 19 2005
parent reply David Rushby <David_member pathlink.com> writes:
Walter Bright wrote:

 This is great stuff! Thanks for taking charge of this and 
 getting it done.


You're welcome.

I think it's wise of you to be so supportive of efforts to integrate D with the
so-called "scripting languages".  Although a few of those languages (Python and
Ruby foremost among them) provide good support for the construction of
non-trivial programs, they're all plagued by slow execution speed, which creates
strong demand for high-performance extensions.  Folks who're already using a
so-called scripting language for serious programming have recognized the value
of most of the features that differentiate D from C++, so I'd expect them to be
receptive to D.  Furthermore, those of us who're using a so-called scripting

over our choice of languages, so there are fewer bureaucratic barriers to D
adoption among the "scripting language" community.

It's hard to tell whether D can be made approachable by Python programmers
who've never written a C extension.  They still have to use a C API to interact
with Python, after all.  But Pyrex
(http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ ), a language that "lets
you write code that mixes Python and C data types any way you want, and compiles
it into a C extension for Python", seems to be popular among Python programmers
who don't know C.

Since I know C and the Python/C API well, I'll probably settle for finishing
(and maintaining) the header translation, writing distutils (build
infrastructure) support, and creating a basic tutorial.  It'd be interesting to
see whether a creative person could make it comfortable for Python programmers
to write extensions in D without using the Python/C API at all.

---

Here's what I've done since my last message on the 20th:

- Annotated each section of the python.d header translation to make it easy to
trace the D header code back to the original Python C header file.  This will
make maintenance of the D header file easier.

- Updated the translation to reflect additions made to the Python/C API in
Python 2.4, which is the current version of Python.  The initial translation was
based on Python 2.3.

- The following header files were missing from the initial translation, but
have now been translated:
- cStringIO.h
- datetime.h
- eval.h
- genobject.h
- marshal.h
- pystrtod.h
- pythread.h
- setobject.h
- structmember.h
- structseq.h

As far as I know, all Python/C API header content that could conceivably be
useful in D programmer has now been translated.

Some areas of the API are not covered because it seems likely that any
extensions that would use these advanced facilities would be written in C.  For
example, a non-Python-core program that creates and manipulates parse trees for
Python code would almost certainly be written in Python itself; only the Python
core is likely to implement such functionality in a native extension, and that
would be in C, not D.

- Used D's mixin feature instead of manually expanding the field header macros
_PyObject_HEAD_EXTRA, PyObject_HEAD, PyObject_VAR_HEAD, and PyDescr_COMMON.  In
addition to easing maintenance, this will make it possible for extensions
written in D to run under a debug build of the Python interpreter, which adds
extra fields to every PyObject for diagnostic purposes.

- Applied a consistent renaming scheme where it was necessary to circumvent D
keywords (I used the renaming convention that's most popular in the Python
community, which is to add a trailing underscore).

- Fixed all problems with 64-bit portability.

- Implemented dynamic type checking macros as D functions.  These tend to be
used heavily in Python/C API code, but the intial translation omitted them.

- Fixed numerous other omissions and a few errors.


The following issues remain unresolved, though I plan to deal with them in due
course (after I've written distutils support for D, so that there's a build
framework within which to operate):

- Need to conditionalize UCS2-versus-UCS4 Python Unicode API functions based on
Python distutils control over the D compilation process.
- During this step, need to review Python's unicodeobject.h to ensure that no
members were omitted from the Deja's initial translation.

- In a build process controlled by Python distutils, need to detect whether the
Python interpreter was built in debug build mode, and if so, make the
appropriate adjustments to the header mixins.


I'll continue to post snapshots at http://kinterbasdb.sourceforge.net/other/d/
in case I get run over by the proverbial truck.
Oct 23 2005
parent reply "Walter Bright" <newshound digitalmars.com> writes:
"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...

 I'll continue to post snapshots at


http://kinterbasdb.sourceforge.net/other/d/

 in case I get run over by the proverbial truck.


Your post is full of great information. When you are ready, could you
distill it all into a "How To" document and tutorial, and then we can put it
on the D web site? That would give it much more exposure, as a random
project on sourceforge isn't going to be run across by the people who need
it :-(
Oct 25 2005
parent reply David Rushby <David_member pathlink.com> writes:
In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says...

"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...

 I'll continue to post snapshots at


http://kinterbasdb.sourceforge.net/other/d/

 in case I get run over by the proverbial truck.


Your post is full of great information. When you are ready, could you
distill it all into a "How To" document and tutorial, and then we can put it
on the D web site? That would give it much more exposure, as a random
project on sourceforge isn't going to be run across by the people who need
it :-(


The only reason I'm currently placing the files at
http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that
web space.  kinterbasdb is not related to D in any way.

I believe in the value of documentation, and have always planned to document
Python/D integration when I am confident that the infrastructure is sufficiently
mature.  (I'd like to compliment you on the quality of D's documentation.  Not
only is it broad in its coverage, but it's written in fluent English.  That is
atypical and quite refreshing.)

Before I reach the stage where I consider Python/D integration worth documenting
or publicizing, though, I need to:

1) Make sure there are no fundamental stumbling blocks (such as
http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ).

2) Have implemented the infrastructure that Python programmers expect (chiefly
distutils support).



stumbling blocks.  I couldn't seriously advocate D as an extension language for
Python when there's no (reliable) way to use it as such under Linux.  I'm
willing to put forth effort myself to correct this, assuming that it doesn't
require months of preparatory study.
Oct 25 2005
next sibling parent John Demme <me teqdruid.com> writes:

this D integration as well as shared library support (keeping in mind that
I don't even have a copy of Windows on my computer).

Walter, since you seem interested in this project, this should be a good
motivation to get the shared library support working under Linux.  Right? 
BTW, did you read Georg's recent "Policy opinion" post?  I agree with it,
although I'm pretty biased.

~John Demme

On Tue, 25 Oct 2005 19:35:05 +0000, David Rushby wrote:


 In article <djlu53$agr$7 digitaldaemon.com>, Walter Bright says...

"David Rushby" <David_member pathlink.com> wrote in message
news:djhir0$2uuk$1 digitaldaemon.com...

 I'll continue to post snapshots at


http://kinterbasdb.sourceforge.net/other/d/

 in case I get run over by the proverbial truck.


Your post is full of great information. When you are ready, could you
distill it all into a "How To" document and tutorial, and then we can put it
on the D web site? That would give it much more exposure, as a random
project on sourceforge isn't going to be run across by the people who need
it :-(


 
 The only reason I'm currently placing the files at
 http://kinterbasdb.sourceforge.net/other/d/ is that I have ready access to that
 web space.  kinterbasdb is not related to D in any way.
 
 I believe in the value of documentation, and have always planned to document
 Python/D integration when I am confident that the infrastructure is
sufficiently
 mature.  (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.  That is
 atypical and quite refreshing.)
 
 Before I reach the stage where I consider Python/D integration worth
documenting
 or publicizing, though, I need to:
 
 1) Make sure there are no fundamental stumbling blocks (such as
 http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ).
 
 2) Have implemented the infrastructure that Python programmers expect (chiefly
 distutils support).
 
 

 stumbling blocks.  I couldn't seriously advocate D as an extension language for
 Python when there's no (reliable) way to use it as such under Linux.  I'm
 willing to put forth effort myself to correct this, assuming that it doesn't
 require months of preparatory study.
Oct 25 2005
prev sibling parent reply "Walter Bright" <newshound digitalmars.com> writes:
"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...

 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.


That is

 atypical and quite refreshing.)


Thanks for the kind words, most people say the documentation stinks <g>. I
actually agree with them, and making the documentation better was the
inspiration for ddoc.



 Before I reach the stage where I consider Python/D integration worth


documenting

 or publicizing, though, I need to:

 1) Make sure there are no fundamental stumbling blocks (such as
 http://www.digitalmars.com/drn-bin/wwwnews?digitalmars.D/29399 ).

 2) Have implemented the infrastructure that Python programmers expect


(chiefly

 distutils support).



 stumbling blocks.  I couldn't seriously advocate D as an extension


language for

 Python when there's no (reliable) way to use it as such under Linux.  I'm
 willing to put forth effort myself to correct this, assuming that it


doesn't

 require months of preparatory study.
Oct 25 2005
parent reply J C Calvarese <technocrat7 gmail.com> writes:
In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says...

"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...

 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.


That is

 atypical and quite refreshing.)


Thanks for the kind words, most people say the documentation stinks <g>. I
actually agree with them, and making the documentation better was the
inspiration for ddoc.


I don't recall anyone saying it was bad. Sometimes people contain there's just
not enough of it. ;) But I guess Ddoc should definitely help with that. I think
having Ddoc built into the compiler is smart and it's going to be a terrific
boon for D and D users.

I suppose you might be thinking of when people complained about the formating of
the web pages, but you'll probably have a lot fewer complaints now. I think they
look pretty slick these days! Thanks for fixing them up.

jcc7
Oct 25 2005
parent J C Calvarese <technocrat7 gmail.com> writes:
In article <djn7ug$8h8$1 digitaldaemon.com>, J C Calvarese says...

In article <djn1el$2v5n$2 digitaldaemon.com>, Walter Bright says...

"David Rushby" <David_member pathlink.com> wrote in message
news:djm1d9$nq7$1 digitaldaemon.com...

 (I'd like to compliment you on the quality of D's documentation.  Not
 only is it broad in its coverage, but it's written in fluent English.


That is

 atypical and quite refreshing.)


Thanks for the kind words, most people say the documentation stinks <g>. I
actually agree with them, and making the documentation better was the
inspiration for ddoc.


I don't recall anyone saying it was bad. Sometimes people contain there's just


I guess I meant "complain" rather than "contain". Perhaps it's time for me to go
to bed?

jcc7
Oct 26 2005