digitalmars.D - Wrapping Python - A function wrapping template
- Kirk McDonald (36/38) Jun 13 2006 In wrapping Python with D, I have just made a large step:
- Daniel Keep (18/69) Jun 13 2006 You stole my idea :P
- Kirk McDonald (7/29) Jun 13 2006 My current instinct is to hide the array in the library, and add a "def"...
- Tom S (4/6) Jun 13 2006 May I know which part of nested templates it didn't like ?
- Daniel Keep (20/26) Jun 13 2006 I deleted the code a while back, but it was basically using these
- Tom S (4/19) Jun 13 2006 Could this: http://158.75.59.9/~h3/tmp/tmp4.d possibly help in any way ?
- Daniel Keep (8/28) Jun 14 2006 It just might :) Thanks for that.
- Tom S (10/12) Jun 13 2006 Looks sweet ! :)
- Kirk McDonald (5/27) Jun 13 2006 All of that will eventually be wrapped. I could indeed tighten up the
- Kirk McDonald (25/27) Jun 14 2006 I've just done two things with regard to this wrapper:
- Daniel Keep (30/60) Jun 14 2006 It's like you cracked open my head while I was asleep, scooped it out
- Kirk McDonald (21/23) Jun 17 2006 The testdll.d file now looks like this:
- Daniel Keep (16/47) Jun 18 2006 Indeed it does. It's somewhat hard to tell that Python's even involved
- Tom S (5/16) Jun 18 2006 I think this stuff might help now:
- Kirk McDonald (22/40) Jun 18 2006 Wow! That nameof module is quite something. I can quite easily do number...
In wrapping Python with D, I have just made a large step:
[testdll.d]
import python;
import pyd.pyd; // My "Python/D" package
int str_len(char[] str) {
return str.length;
}
extern (C) {
PyMethodDef testdll_GlobalMethods[] = [
{ "str_len", cast(PyCFunction)&func_wrap!(str_len).func,
METH_VARARGS, "" },
{ null, null, 0, null }
];
export void inittestdll() {
_loadPythonSupport();
PyObject* m = Py_InitModule("testdll", testdll_GlobalMethods);
}
} /* end extern (C) */
// EOF
(In a Python session:)
import testdll
testdll.str_len("This")
4
Note the "func_wrap" template. This takes a regular D function and wraps
it with something that can be exposed to Python. There are some
limitations, as yet: The arguments and return type of the function must
be ones it knows how to convert to and from Python, and there is not yet
a mechanism for adding types to this list. It also does not yet support
default arguments. It would also be nice to wrap all of the business
with the method definition array. This can all be done.
I'd like to acknowledge the work of David Rushby, whose "celerid"
project and updated Python header were invaluable, as well as Deja
Augustine, who first took a crack at the header.
Daniel Keep and Tom S also proivded some useful function
property-deriving templates.
This library, as it exists now, is more a proof-of-concept than a useful
library, so I'm not quite prepared to release it yet.
-Kirk McDonald
Jun 13 2006
Kirk McDonald wrote:
In wrapping Python with D, I have just made a large step:
[testdll.d]
import python;
import pyd.pyd; // My "Python/D" package
int str_len(char[] str) {
return str.length;
}
extern (C) {
PyMethodDef testdll_GlobalMethods[] = [
{ "str_len", cast(PyCFunction)&func_wrap!(str_len).func,
METH_VARARGS, "" },
{ null, null, 0, null }
];
export void inittestdll() {
_loadPythonSupport();
PyObject* m = Py_InitModule("testdll", testdll_GlobalMethods);
}
} /* end extern (C) */
// EOF
(In a Python session:)
import testdll
testdll.str_len("This")
4
Note the "func_wrap" template. This takes a regular D function and wraps
it with something that can be exposed to Python. There are some
limitations, as yet: The arguments and return type of the function must
be ones it knows how to convert to and from Python, and there is not yet
a mechanism for adding types to this list. It also does not yet support
default arguments. It would also be nice to wrap all of the business
with the method definition array. This can all be done.
I'd like to acknowledge the work of David Rushby, whose "celerid"
project and updated Python header were invaluable, as well as Deja
Augustine, who first took a crack at the header.
Daniel Keep and Tom S also proivded some useful function
property-deriving templates.
This library, as it exists now, is more a proof-of-concept than a useful
library, so I'm not quite prepared to release it yet.
-Kirk McDonald
You stole my idea :P
Seriously, congratulations. I know I didn't quite get that far. This
is fantastic stuff; it won't be long now before we have our own version
of Boost::Python :)
Just a few questions:
* How did you deal with the whole 8-bit with encodings <==> UTF-8 thing?
I was seriously considering just forcing everything to go via UTF-16,
or did you write code to do the transcoding manually?
* Do you have any plans on how to wrap the method definition array? I
tried using nested templates but D didn't like it :(
In any case, fantastic work.
-- Daniel
--
Unlike Knuth, I have neither proven or tried the above; it may not even
make sense.
v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 13 2006
Kirk McDonald <kirklin.mcdonald gmail.com> Daniel Keep wrote:Kirk McDonald wrote:I haven't. :-) That remains to be done. It just does ASCII at the moment.This library, as it exists now, is more a proof-of-concept than a useful library, so I'm not quite prepared to release it yet.You stole my idea :P Seriously, congratulations. I know I didn't quite get that far. This is fantastic stuff; it won't be long now before we have our own version of Boost::Python :) Just a few questions: * How did you deal with the whole 8-bit with encodings <==> UTF-8 thing? I was seriously considering just forcing everything to go via UTF-16, or did you write code to do the transcoding manually?* Do you have any plans on how to wrap the method definition array? I tried using nested templates but D didn't like it :(My current instinct is to hide the array in the library, and add a "def" function (a la Boost::Python) that knows how to add entries to it. Haven't tried this yet, of course.In any case, fantastic work.Thanks! -Kirk McDonald
Jun 13 2006
Daniel Keep wrote:* Do you have any plans on how to wrap the method definition array? I tried using nested templates but D didn't like it :(May I know which part of nested templates it didn't like ? -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
Tom S wrote:Daniel Keep wrote:I deleted the code a while back, but it was basically using these nested, recursive templates so that I could do this: alias StartModule!("Quxx").AddMethod!("foo", PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule; However, D complained of various things. Funny thing was, THIS worked: alias StartModule!("Quxx") t1; alias t1.AddMethod!("foo", PydWrap!(foo)) t2; alias t2.AddMethod!("bar", PydWrap!(bar)) t3; alias t3.EndModule!() QuxxModule; But I figured that was as much, if not more verbose then doing it "long-hand", so I gave up. -- Daniel P.S. Like so many things, I can't actually remember specifics of how I defined that thing, but that was the gist of it. -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/* Do you have any plans on how to wrap the method definition array? I tried using nested templates but D didn't like it :(May I know which part of nested templates it didn't like ?
Jun 13 2006
Daniel Keep wrote:
I deleted the code a while back, but it was basically using these
nested, recursive templates so that I could do this:
alias StartModule!("Quxx").AddMethod!("foo",
PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule;
However, D complained of various things. Funny thing was, THIS worked:
alias StartModule!("Quxx") t1;
alias t1.AddMethod!("foo", PydWrap!(foo)) t2;
alias t2.AddMethod!("bar", PydWrap!(bar)) t3;
alias t3.EndModule!() QuxxModule;
But I figured that was as much, if not more verbose then doing it
"long-hand", so I gave up.
Could this: http://158.75.59.9/~h3/tmp/tmp4.d possibly help in any way ?
--
Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
Tom S wrote:Daniel Keep wrote:It just might :) Thanks for that. -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/I deleted the code a while back, but it was basically using these nested, recursive templates so that I could do this: alias StartModule!("Quxx").AddMethod!("foo", PydWrap!(foo)).AddMethod!("bar", PydWrap!(bar)).EndModule!() QuxxModule; However, D complained of various things. Funny thing was, THIS worked: alias StartModule!("Quxx") t1; alias t1.AddMethod!("foo", PydWrap!(foo)) t2; alias t2.AddMethod!("bar", PydWrap!(bar)) t3; alias t3.EndModule!() QuxxModule; But I figured that was as much, if not more verbose then doing it "long-hand", so I gave up.Could this: http://158.75.59.9/~h3/tmp/tmp4.d possibly help in any way ?
Jun 14 2006
Kirk McDonald wrote:This library, as it exists now, is more a proof-of-concept than a useful library, so I'm not quite prepared to release it yet.Looks sweet ! :) Hmmm... considering the code: { "str_len", cast(PyCFunction)&func_wrap!(str_len).func, METH_VARARGS, "" }, what's the purpose of 'cast(PyCFunction)&func_wrap!(str_len).func' ? couldn't it be replaced by just a single 'pycFunction!(str_len)' or something like this ? Keep up the great work ! -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 13 2006
Tom S wrote:Kirk McDonald wrote:All of that will eventually be wrapped. I could indeed tighten up the interface easily enough, and I have some other ideas of how to do that.This library, as it exists now, is more a proof-of-concept than a useful library, so I'm not quite prepared to release it yet.Looks sweet ! :) Hmmm... considering the code: { "str_len", cast(PyCFunction)&func_wrap!(str_len).func, METH_VARARGS, "" }, what's the purpose of 'cast(PyCFunction)&func_wrap!(str_len).func' ? couldn't it be replaced by just a single 'pycFunction!(str_len)' or something like this ?Keep up the great work !Thanks! -Kirk McDonald
Jun 13 2006
Kirk McDonald wrote:In wrapping Python with D, I have just made a large step:I've just done two things with regard to this wrapper: First, I've implemented some basic exception wrapping. If a wrapped Python API function raises a Python exception, this is now captured and thrown as a D exception. If this exception is not caught by user code, it is caught by the function wrapping template, and translated back into the original Python exception. If a D exception is thrown and not caught by user code, the function wrapping template will translate it into a Python RuntimeError and pass along the message. On the "to do" list is creating a hierarchy of D exception classes that mirrors the hierarchy of Python exceptions, as at the moment there is a single "PythonException" class. It might also be nice to provide some Python exceptions that mirror various D exceptions, for when those escape into the interpreter. (Alternately, one might map them onto an appropriate existing Python exception.) Second, the function wrapper now supports void return types. Imagine my suprise when I realised it didn't support them before! (It was sort of a "duh" moment.) Adding this support roughly doubled the length of the template, which wasn't short to begin with. Each number of arguments the wrapped function can accept (from 0 to 10) has its own "static if" clause, and now each of these has been doubled, as functions with return types to capture must be called differently than those without. Anyway, functions with a void return type now return Py_None as per the usual Python idiom. -Kirk McDonald
Jun 14 2006
Kirk McDonald wrote:Kirk McDonald wrote:It's like you cracked open my head while I was asleep, scooped it out and then made some kind of brain-USB connector. Actually, that might explain the headache and neck pain...In wrapping Python with D, I have just made a large step:I've just done two things with regard to this wrapper: First, I've implemented some basic exception wrapping. If a wrapped Python API function raises a Python exception, this is now captured and thrown as a D exception. If this exception is not caught by user code, it is caught by the function wrapping template, and translated back into the original Python exception. If a D exception is thrown and not caught by user code, the function wrapping template will translate it into a Python RuntimeError and pass along the message.On the "to do" list is creating a hierarchy of D exception classes that mirrors the hierarchy of Python exceptions, as at the moment there is a single "PythonException" class. It might also be nice to provide some Python exceptions that mirror various D exceptions, for when those escape into the interpreter. (Alternately, one might map them onto an appropriate existing Python exception.)Actually, I never considered doing this, because I set my sights a little higher: wrapping every single last host class. I reckon that if we could get RTTI, it could be done. Then you could just to something equivalent to:The trick would be to make them subclassable, which I think would require more intimate knowledge of the binary format of classes, and probably generating machine code shims. But you've got to admit, being able to do that would be "wow" :) Failing "real" RTTI, I'd want to write a program that takes a D module and spits out an RTTI module for it. I know there's one that already does this, but I'm not sure when it was last updated. Plus, I'm itching for a reason to play with Enki :P [1] <sigh> When you look at what I have planned for my game engine, the list of dependencies I have to write is huge. Oh well... more fun for me :)import host.object.Exception as DExceptionSecond, the function wrapper now supports void return types. Imagine my suprise when I realised it didn't support them before! (It was sort of a "duh" moment.) Adding this support roughly doubled the length of the template, which wasn't short to begin with. Each number of arguments the wrapped function can accept (from 0 to 10) has its own "static if" clause, and now each of these has been doubled, as functions with return types to capture must be called differently than those without. Anyway, functions with a void return type now return Py_None as per the usual Python idiom.Haha. Yeah, I got a bit annoyed when I realised that supporting void generally means duplicating everything. It would be so much easier if "void" could be used like a regular type, except that it takes up zero bytes and never pushes anything on to the stack.-Kirk McDonald-- Daniel [1] Every time I read that name, I get the feeling like it should be in text similar to Yahoo!'s font, with an exclamation mark on the end. Enki! -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 14 2006
Kirk McDonald wrote:In wrapping Python with D, I have just made a large step:The testdll.d file now looks like this: [testdll.d] module testdll; import pyd.pyd; char[] foo(int i) { if (i > 10) { return "It's greater than 10!"; } else { return "It's less than 10!"; } } extern (C) export void inittestdll() { def!("foo", foo); module_init("testdll"); } // EOF Quite a lot of things remain to be done, but at least the API looks nice and clean. -Kirk McDonald
Jun 17 2006
Kirk McDonald wrote:Kirk McDonald wrote:VERY nice.In wrapping Python with D, I have just made a large step:The testdll.d file now looks like this: [testdll.d] module testdll; import pyd.pyd; char[] foo(int i) { if (i > 10) { return "It's greater than 10!"; } else { return "It's less than 10!"; } } extern (C) export void inittestdll() { def!("foo", foo); module_init("testdll"); }// EOF Quite a lot of things remain to be done, but at least the API looks nice and clean. -Kirk McDonaldIndeed it does. It's somewhat hard to tell that Python's even involved anymore :) The only other two things I can think of are: 1. Being able to use def!(foo), then just parse out foo's original name would be cool :) 2. I wonder if you can somehow pinch the current module's name for the module_init() call. Not criticisms by any stretch of the imagination; just curious if it'd be possible. Can't wait to get my grubby mits on your code :) -- Daniel -- Unlike Knuth, I have neither proven or tried the above; it may not even make sense. v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Jun 18 2006
Daniel Keep wrote:Indeed it does. It's somewhat hard to tell that Python's even involved anymore :) The only other two things I can think of are: 1. Being able to use def!(foo), then just parse out foo's original name would be cool :) 2. I wonder if you can somehow pinch the current module's name for the module_init() call. Not criticisms by any stretch of the imagination; just curious if it'd be possible. Can't wait to get my grubby mits on your code :)I think this stuff might help now: http://dsource.org/projects/ddl/browser/trunk/meta -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jun 18 2006
Tom S wrote:Daniel Keep wrote:Wow! That nameof module is quite something. I can quite easily do number (1) above (while still providing the ability to explicitly enter the name, of course!), but (2) is harder. The module_init function is naturally in a different module than then call to module_init. In this example case, it is easy enough to pull the module name out of foo's fully qualified name, but of course foo might be in a different module than the call to module_init, also! (And, similarly, a project of any size would likely have functions across more than one module; which one would you use?) Any other approaches would probably impose more of a burden on the user than simply entering the module's name. Of course, I could always say "if you don't provide a name to module_init, the name of the module in which the first function in the list is found will be used." But that could easily lead to oddity; I'd rather have it be explicit. On the other hand, it might be enough for small projects like the example. In other news, I am currently refactoring my object wrapper into a single class, rather than a hierarchy of classes. This is how Boost.Python does it, and it turns out it actually is a better way than the way I was doing it. Not only is it easier to work with, but the code ends up being easier to understand, too. -Kirk McDonaldIndeed it does. It's somewhat hard to tell that Python's even involved anymore :) The only other two things I can think of are: 1. Being able to use def!(foo), then just parse out foo's original name would be cool :) 2. I wonder if you can somehow pinch the current module's name for the module_init() call. Not criticisms by any stretch of the imagination; just curious if it'd be possible. Can't wait to get my grubby mits on your code :)I think this stuff might help now: http://dsource.org/projects/ddl/browser/trunk/meta
Jun 18 2006