www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Wrapping Python - A function wrapping template

reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
In wrapping Python with D, I have just made a large step:

[testdll.d]
import python;
import pyd.pyd; // My "Python/D" package

int str_len(char[] str) {
     return str.length;
}

extern (C) {

PyMethodDef testdll_GlobalMethods[] = [
     { "str_len", cast(PyCFunction)&func_wrap!(str_len).func, 
METH_VARARGS, "" },
     { null, null, 0, null }
];

export void inittestdll() {
     _loadPythonSupport();

     PyObject* m = Py_InitModule("testdll", testdll_GlobalMethods);
}

} /* end extern (C) */
// EOF

(In a Python session:)
 import testdll
 testdll.str_len("This")



Note the "func_wrap" template. This takes a regular D function and wraps it with something that can be exposed to Python. There are some limitations, as yet: The arguments and return type of the function must be ones it knows how to convert to and from Python, and there is not yet a mechanism for adding types to this list. It also does not yet support default arguments. It would also be nice to wrap all of the business with the method definition array. This can all be done. I'd like to acknowledge the work of David Rushby, whose "celerid" project and updated Python header were invaluable, as well as Deja Augustine, who first took a crack at the header. Daniel Keep and Tom S also proivded some useful function property-deriving templates. This library, as it exists now, is more a proof-of-concept than a useful library, so I'm not quite prepared to release it yet. -Kirk McDonald
Jun 13 2006
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Kirk McDonald wrote:
 In wrapping Python with D, I have just made a large step:
 
 [testdll.d]
 import python;
 import pyd.pyd; // My "Python/D" package
 
 int str_len(char[] str) {
     return str.length;
 }
 
 extern (C) {
 
 PyMethodDef testdll_GlobalMethods[] = [
     { "str_len", cast(PyCFunction)&func_wrap!(str_len).func,
 METH_VARARGS, "" },
     { null, null, 0, null }
 ];
 
 export void inittestdll() {
     _loadPythonSupport();
 
     PyObject* m = Py_InitModule("testdll", testdll_GlobalMethods);
 }
 
 } /* end extern (C) */
 // EOF
 
 (In a Python session:)
 import testdll
 testdll.str_len("This")



Note the "func_wrap" template. This takes a regular D function and wraps it with something that can be exposed to Python. There are some limitations, as yet: The arguments and return type of the function must be ones it knows how to convert to and from Python, and there is not yet a mechanism for adding types to this list. It also does not yet support default arguments. It would also be nice to wrap all of the business with the method definition array. This can all be done. I'd like to acknowledge the work of David Rushby, whose "celerid" project and updated Python header were invaluable, as well as Deja Augustine, who first took a crack at the header. Daniel Keep and Tom S also proivded some useful function property-deriving templates. This library, as it exists now, is more a proof-of-concept than a useful library, so I'm not quite prepared to release it yet. -Kirk McDonald

You stole my idea :P Seriously, congratulations. I know I didn't quite get that far. This is fantastic stuff; it won't be long now before we have our own version of Boost::Python :) Just a few questions: * How did you deal with the whole 8-bit with encodings <==> UTF-8 thing? I was seriously considering just forcing everything to go via UTF-16, or did you write code to do the transcoding manually? * Do you have any plans on how to wrap the method definition array? I tried using nested templates but D didn't like it :( In any case, fantastic work. -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 13 2006
next sibling parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Daniel Keep wrote:
 Kirk McDonald wrote:
This library, as it exists now, is more a proof-of-concept than a useful
library, so I'm not quite prepared to release it yet.

You stole my idea :P Seriously, congratulations. I know I didn't quite get that far. This is fantastic stuff; it won't be long now before we have our own version of Boost::Python :) Just a few questions: * How did you deal with the whole 8-bit with encodings <==> UTF-8 thing? I was seriously considering just forcing everything to go via UTF-16, or did you write code to do the transcoding manually?

I haven't. :-) That remains to be done. It just does ASCII at the moment.
 * Do you have any plans on how to wrap the method definition array?  I
 tried using nested templates but D didn't like it :(

My current instinct is to hide the array in the library, and add a "def" function (a la Boost::Python) that knows how to add entries to it. Haven't tried this yet, of course.
 
 In any case, fantastic work.

Thanks! -Kirk McDonald
Jun 13 2006
prev sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Daniel Keep wrote:
 * Do you have any plans on how to wrap the method definition array?  I
 tried using nested templates but D didn't like it :(

May I know which part of nested templates it didn't like ? -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Tom S wrote:
 Daniel Keep wrote:
 * Do you have any plans on how to wrap the method definition array?  I
 tried using nested templates but D didn't like it :(

May I know which part of nested templates it didn't like ?

I deleted the code a while back, but it was basically using these nested, recursive templates so that I could do this: alias StartModule!("Quxx").AddMethod!("foo", PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule; However, D complained of various things. Funny thing was, THIS worked: alias StartModule!("Quxx") t1; alias t1.AddMethod!("foo", PydWrap!(foo)) t2; alias t2.AddMethod!("bar", PydWrap!(bar)) t3; alias t3.EndModule!() QuxxModule; But I figured that was as much, if not more verbose then doing it "long-hand", so I gave up. -- Daniel P.S. Like so many things, I can't actually remember specifics of how I defined that thing, but that was the gist of it. -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 13 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Daniel Keep wrote:
 I deleted the code a while back, but it was basically using these
 nested, recursive templates so that I could do this:
 
 alias StartModule!("Quxx").AddMethod!("foo",
 PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule;
 
 However, D complained of various things.  Funny thing was, THIS worked:
 
 alias StartModule!("Quxx") t1;
 alias t1.AddMethod!("foo", PydWrap!(foo)) t2;
 alias t2.AddMethod!("bar", PydWrap!(bar)) t3;
 alias t3.EndModule!() QuxxModule;
 
 But I figured that was as much, if not more verbose then doing it
 "long-hand", so I gave up.

Could this: http://158.75.59.9/~h3/tmp/tmp4.d possibly help in any way ? -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
Tom S wrote:
 Daniel Keep wrote:
 I deleted the code a while back, but it was basically using these
 nested, recursive templates so that I could do this:

 alias StartModule!("Quxx").AddMethod!("foo",
 PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule;

 However, D complained of various things.  Funny thing was, THIS worked:

 alias StartModule!("Quxx") t1;
 alias t1.AddMethod!("foo", PydWrap!(foo)) t2;
 alias t2.AddMethod!("bar", PydWrap!(bar)) t3;
 alias t3.EndModule!() QuxxModule;

 But I figured that was as much, if not more verbose then doing it
 "long-hand", so I gave up.

Could this: http://158.75.59.9/~h3/tmp/tmp4.d possibly help in any way ?

It just might :) Thanks for that. -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 14 2006
prev sibling next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Kirk McDonald wrote:
 This library, as it exists now, is more a proof-of-concept than a useful 
 library, so I'm not quite prepared to release it yet.

Looks sweet ! :) Hmmm... considering the code: { "str_len", cast(PyCFunction)&func_wrap!(str_len).func, METH_VARARGS, "" }, what's the purpose of 'cast(PyCFunction)&func_wrap!(str_len).func' ? couldn't it be replaced by just a single 'pycFunction!(str_len)' or something like this ? Keep up the great work ! -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Tom S wrote:
 Kirk McDonald wrote:
 
 This library, as it exists now, is more a proof-of-concept than a 
 useful library, so I'm not quite prepared to release it yet.

Looks sweet ! :) Hmmm... considering the code: { "str_len", cast(PyCFunction)&func_wrap!(str_len).func, METH_VARARGS, "" }, what's the purpose of 'cast(PyCFunction)&func_wrap!(str_len).func' ? couldn't it be replaced by just a single 'pycFunction!(str_len)' or something like this ?

interface easily enough, and I have some other ideas of how to do that.
 Keep up the great work !
 
 

-Kirk McDonald
Jun 13 2006
prev sibling next sibling parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Kirk McDonald wrote:
 In wrapping Python with D, I have just made a large step:
 

I've just done two things with regard to this wrapper: First, I've implemented some basic exception wrapping. If a wrapped Python API function raises a Python exception, this is now captured and thrown as a D exception. If this exception is not caught by user code, it is caught by the function wrapping template, and translated back into the original Python exception. If a D exception is thrown and not caught by user code, the function wrapping template will translate it into a Python RuntimeError and pass along the message. On the "to do" list is creating a hierarchy of D exception classes that mirrors the hierarchy of Python exceptions, as at the moment there is a single "PythonException" class. It might also be nice to provide some Python exceptions that mirror various D exceptions, for when those escape into the interpreter. (Alternately, one might map them onto an appropriate existing Python exception.) Second, the function wrapper now supports void return types. Imagine my suprise when I realised it didn't support them before! (It was sort of a "duh" moment.) Adding this support roughly doubled the length of the template, which wasn't short to begin with. Each number of arguments the wrapped function can accept (from 0 to 10) has its own "static if" clause, and now each of these has been doubled, as functions with return types to capture must be called differently than those without. Anyway, functions with a void return type now return Py_None as per the usual Python idiom. -Kirk McDonald
Jun 14 2006
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
Kirk McDonald wrote:
 Kirk McDonald wrote:
 In wrapping Python with D, I have just made a large step:

I've just done two things with regard to this wrapper: First, I've implemented some basic exception wrapping. If a wrapped Python API function raises a Python exception, this is now captured and thrown as a D exception. If this exception is not caught by user code, it is caught by the function wrapping template, and translated back into the original Python exception. If a D exception is thrown and not caught by user code, the function wrapping template will translate it into a Python RuntimeError and pass along the message.

It's like you cracked open my head while I was asleep, scooped it out and then made some kind of brain-USB connector. Actually, that might explain the headache and neck pain...
 On the "to do" list is creating a hierarchy of D exception classes that
 mirrors the hierarchy of Python exceptions, as at the moment there is a
 single "PythonException" class. It might also be nice to provide some
 Python exceptions that mirror various D exceptions, for when those
 escape into the interpreter. (Alternately, one might map them onto an
 appropriate existing Python exception.)

Actually, I never considered doing this, because I set my sights a little higher: wrapping every single last host class. I reckon that if we could get RTTI, it could be done. Then you could just to something equivalent to:
 import host.object.Exception as DException



The trick would be to make them subclassable, which I think would require more intimate knowledge of the binary format of classes, and probably generating machine code shims. But you've got to admit, being able to do that would be "wow" :) Failing "real" RTTI, I'd want to write a program that takes a D module and spits out an RTTI module for it. I know there's one that already does this, but I'm not sure when it was last updated. Plus, I'm itching for a reason to play with Enki :P [1] <sigh> When you look at what I have planned for my game engine, the list of dependencies I have to write is huge. Oh well... more fun for me :)
 Second, the function wrapper now supports void return types. Imagine my
 suprise when I realised it didn't support them before! (It was sort of a
 "duh" moment.) Adding this support roughly doubled the length of the
 template, which wasn't short to begin with. Each number of arguments the
 wrapped function can accept (from 0 to 10) has its own "static if"
 clause, and now each of these has been doubled, as functions with return
 types to capture must be called differently than those without. Anyway,
 functions with a void return type now return Py_None as per the usual
 Python idiom.

Haha. Yeah, I got a bit annoyed when I realised that supporting void generally means duplicating everything. It would be so much easier if "void" could be used like a regular type, except that it takes up zero bytes and never pushes anything on to the stack.
 -Kirk McDonald

-- Daniel [1] Every time I read that name, I get the feeling like it should be in text similar to Yahoo!'s font, with an exclamation mark on the end. Enki! -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 14 2006
prev sibling parent reply Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Kirk McDonald wrote:
 In wrapping Python with D, I have just made a large step:
 

The testdll.d file now looks like this: [testdll.d] module testdll; import pyd.pyd; char[] foo(int i) { if (i > 10) { return "It's greater than 10!"; } else { return "It's less than 10!"; } } extern (C) export void inittestdll() { def!("foo", foo); module_init("testdll"); } // EOF Quite a lot of things remain to be done, but at least the API looks nice and clean. -Kirk McDonald
Jun 17 2006
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Kirk McDonald wrote:
 Kirk McDonald wrote:
 In wrapping Python with D, I have just made a large step:

The testdll.d file now looks like this: [testdll.d] module testdll; import pyd.pyd; char[] foo(int i) { if (i > 10) { return "It's greater than 10!"; } else { return "It's less than 10!"; } } extern (C) export void inittestdll() { def!("foo", foo); module_init("testdll"); }

VERY nice.
 // EOF
 
 Quite a lot of things remain to be done, but at least the API looks nice
 and clean.
 
 -Kirk McDonald

Indeed it does. It's somewhat hard to tell that Python's even involved anymore :) The only other two things I can think of are: 1. Being able to use def!(foo), then just parse out foo's original name would be cool :) 2. I wonder if you can somehow pinch the current module's name for the module_init() call. Not criticisms by any stretch of the imagination; just curious if it'd be possible. Can't wait to get my grubby mits on your code :) -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 18 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Daniel Keep wrote:
 Indeed it does.  It's somewhat hard to tell that Python's even involved
 anymore :)  The only other two things I can think of are:
 
 1. Being able to use def!(foo), then just parse out foo's original name
 would be cool :)
 
 2. I wonder if you can somehow pinch the current module's name for the
 module_init() call.
 
 Not criticisms by any stretch of the imagination; just curious if it'd
 be possible.  Can't wait to get my grubby mits on your code :)

I think this stuff might help now: http://dsource.org/projects/ddl/browser/trunk/meta -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 18 2006
parent Kirk McDonald <kirklin.mcdonald gmail.com> writes:
Tom S wrote:
 Daniel Keep wrote:
 
 Indeed it does.  It's somewhat hard to tell that Python's even involved
 anymore :)  The only other two things I can think of are:

 1. Being able to use def!(foo), then just parse out foo's original name
 would be cool :)

 2. I wonder if you can somehow pinch the current module's name for the
 module_init() call.

 Not criticisms by any stretch of the imagination; just curious if it'd
 be possible.  Can't wait to get my grubby mits on your code :)

I think this stuff might help now: http://dsource.org/projects/ddl/browser/trunk/meta

Wow! That nameof module is quite something. I can quite easily do number (1) above (while still providing the ability to explicitly enter the name, of course!), but (2) is harder. The module_init function is naturally in a different module than then call to module_init. In this example case, it is easy enough to pull the module name out of foo's fully qualified name, but of course foo might be in a different module than the call to module_init, also! (And, similarly, a project of any size would likely have functions across more than one module; which one would you use?) Any other approaches would probably impose more of a burden on the user than simply entering the module's name. Of course, I could always say "if you don't provide a name to module_init, the name of the module in which the first function in the list is found will be used." But that could easily lead to oddity; I'd rather have it be explicit. On the other hand, it might be enough for small projects like the example. In other news, I am currently refactoring my object wrapper into a single class, rather than a hierarchy of classes. This is how Boost.Python does it, and it turns out it actually is a better way than the way I was doing it. Not only is it easier to work with, but the code ends up being easier to understand, too. -Kirk McDonald
Jun 18 2006