www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - autowrap v0.0.1 - Automatically wrap existing D code for use in Python

reply Atila Neves <atila.neves gmail.com> writes:
http://code.dlang.org/packages/autowrap

This came out of the need at work to take existing D code and 
make it available for both Excel and Python.

Both pyd and excel-d make the reasonable assumption that one is 
using them to write code specifically for those environments. 
That breaks when there's existing production D code one wants to 
wrap.

The idea is to not "dirty" the existing code with dependencies on 
either pyd or excel-d and instead wrap them from outside the 
existing dub packages, whilst still using pyd and excel-d behind 
the scenes.

The end result is that if you have two D modules called 
`my.module1` and `my.module2` and you want to create a Python 
extension called `mylibrary`, then one dub.sdl and one 
source/app.d such as this is enough:


     import autowrap.python;
     mixin(
         wrapAll(
             LibraryName("mylibrary"),
             Modules("my.module1", "my.module2", /* ... */),
         )
     );

Seriously, that's it.* Well, other than the fact that dub will 
produce libmylibrary.so on Linux and what you need is 
mylibrary.so, so you'll have to rename the file.

The functions that are to be enabled for wrapping must be marked 
`export`, both for Python and Excel. It is a form of tagging and 
the production code must be changed to accomodate it, but at 
least it introduces no extra dependencies.

For Python, any struct used in an `export` function's parameters 
or return type is automatically wrapped, as well as any structs 
inside it.


Atila


* Except maybe for those pesky bug things I can't seem to get rid 
of.
Apr 18
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 18 April 2018 at 15:28:07 UTC, Atila Neves wrote:
 http://code.dlang.org/packages/autowrap

 This came out of the need at work to take existing D code and 
 make it available for both Excel and Python.

 [snip]
Cool. I bet something similar would work with embedr as well.
Apr 18
prev sibling next sibling parent karita <shigekikarita gmail.com> writes:
On Wednesday, 18 April 2018 at 15:28:07 UTC, Atila Neves wrote:
 http://code.dlang.org/packages/autowrap

 This came out of the need at work to take existing D code and 
 make it available for both Excel and Python.

 [...]
Awesome. I'm also working on numpy.ndarray <-> mir.ndslice automatic wrapper https://github.com/ShigekiKarita/mir-pybuffer
Apr 20
prev sibling next sibling parent Laeeth Isharc <laeeth laeeth.com> writes:
On Wednesday, 18 April 2018 at 15:28:07 UTC, Atila Neves wrote:
 http://code.dlang.org/packages/autowrap

 This came out of the need at work to take existing D code and 
 make it available for both Excel and Python.

 Both pyd and excel-d make the reasonable assumption that one is 
 using them to write code specifically for those environments. 
 That breaks when there's existing production D code one wants 
 to wrap.
https://www.reddit.com/r/programming/comments/8eb12m/autowrap_v001_automatically_wrap_existing_d_code/
Apr 23
prev sibling next sibling parent Nikos <nikos home.com> writes:
Interesting stuff.

In http://code.dlang.org/packages/autowrap it says:

"""
  Python versions

Since autowrap depends on PyD, the python version must be 
explicitly stated as a dub configuration and defaults to 3.6. To 
use another version, pass -c $CONFIG to dub where $CONFIG is one 
of:

     python27
     python34
     python35
     python36
"""

Could you please provide an example of how can I do this?

Thanks
May 10
prev sibling parent reply Nikos <nikos home.com> writes:
In my dub.sdl file I have

 configuration "python35" {
  subConfiguration "autowrap" "python35"
}
and I run
 dub build --config=python35
which still tries to find python36. Why doesn't it look for 3.5?
May 10
next sibling parent Laeeth Isharc <laeethnospam nospam.laeeth.com> writes:
On Thursday, 10 May 2018 at 19:50:40 UTC, Nikos wrote:
 In my dub.sdl file I have

 configuration "python35" {
  subConfiguration "autowrap" "python35"
}
and I run
 dub build --config=python35
which still tries to find python36. Why doesn't it look for 3.5?
Hi. On my phone so can't copy paste. Edit your dub.sdl under the python35 subconfiguration and change python 36 to python35.
May 10
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 10 May 2018 at 19:50:40 UTC, Nikos wrote:
 In my dub.sdl file I have

 configuration "python35" {
  subConfiguration "autowrap" "python35"
}
and I run
 dub build --config=python35
which still tries to find python36. Why doesn't it look for 3.5?
Copy + paste error, sorry. Fixed now. Atila
May 11
next sibling parent Nikos <nikos home.com> writes:
On Friday, 11 May 2018 at 14:16:15 UTC, Atila Neves wrote:
 On Thursday, 10 May 2018 at 19:50:40 UTC, Nikos wrote:
 In my dub.sdl file I have

 configuration "python35" {
  subConfiguration "autowrap" "python35"
}
and I run
 dub build --config=python35
which still tries to find python36. Why doesn't it look for 3.5?
Copy + paste error, sorry. Fixed now. Atila
That worked like a charm! Thanks man!
May 12
prev sibling parent reply Nikos <nikos home.com> writes:
I'm trying to wrap drepl 
(https://github.com/dlang-community/drepl)

My dub.sdl files is

import autowrap.python;
mixin(
    wrapAll(
        LibraryName("drepl"),
        Modules("drepl.interpreter"),
    )
);
I also flagged `export` the interpreter function
export Interpreter!Engine interpreter(Engine)(return scope 
Engine e) if (isEngine!Engine)
{
    // workaround Issue 18540
    return Interpreter!Engine(()  trusted { return move(e); }());
}
I build the library with python35, but when I import it from python idle, I cannot access the `interpreter` function at all. I have the feeling I miss something essential here, but I don't know what it is. Any ideas?
May 13
next sibling parent Laeeth Isharc <laeeth laeeth.com> writes:
On Sunday, 13 May 2018 at 16:23:49 UTC, Nikos wrote:
 I'm trying to wrap drepl 
 (https://github.com/dlang-community/drepl)

 My dub.sdl files is

import autowrap.python;
mixin(
    wrapAll(
        LibraryName("drepl"),
        Modules("drepl.interpreter"),
    )
);
I also flagged `export` the interpreter function
export Interpreter!Engine interpreter(Engine)(return scope 
Engine e) if (isEngine!Engine)
{
    // workaround Issue 18540
    return Interpreter!Engine(()  trusted { return move(e); 
 }());
}
I build the library with python35, but when I import it from python idle, I cannot access the `interpreter` function at all. I have the feeling I miss something essential here, but I don't know what it is. Any ideas?
It won't wrap a templated type or function. You will have to make a little shim that instantiates the templated function with a particular type.
May 13
prev sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Sunday, 13 May 2018 at 16:23:49 UTC, Nikos wrote:
 I'm trying to wrap drepl 
 (https://github.com/dlang-community/drepl)

 My dub.sdl files is

import autowrap.python;
mixin(
    wrapAll(
        LibraryName("drepl"),
        Modules("drepl.interpreter"),
    )
);
I also flagged `export` the interpreter function
export Interpreter!Engine interpreter(Engine)(return scope 
Engine e) if (isEngine!Engine)
{
    // workaround Issue 18540
    return Interpreter!Engine(()  trusted { return move(e); 
 }());
}
I build the library with python35, but when I import it from python idle, I cannot access the `interpreter` function at all. I have the feeling I miss something essential here, but I don't know what it is. Any ideas?
Eg turn this into a function and try wrapping this instead: auto intp = interpreter(dmdEngine());
May 13
next sibling parent Nikos <nikoskaragiannakis gmail.com> writes:
 Eg turn this into a function and try wrapping this instead:

     auto intp = interpreter(dmdEngine());
Thanks for your help. I'm doing this in my spare time which is, unfortunately, not much. I did what you said
 export {
     auto intp = interpreter(echoEngine);
 }
but when I import drepl from python idle I still can access intp.
Jun 24
prev sibling next sibling parent Nikos <nikoskaragiannakis gmail.com> writes:
Ok, I made a stupid mistake. It works now. Thanks a lot!
Jul 29
prev sibling parent reply Nikos <nikoskaragiannakis gmail.com> writes:
 Eg turn this into a function and try wrapping this instead:

     auto intp = interpreter(dmdEngine());
Actually, I manage to export the `interpret` method
 export:
     auto intp(char[] txt) {
         return interpreter(dmdEngine()).interpret(txt);
     }
and tested it in ipython successfully. But when I try to export the whole dmdEngine
 export:

    auto engine(char[] txt) {
            return interpreter(dmdEngine());
    }
it complains about copying Interpreter!(DMDEngine).Interpreter
 ../../../.dub/packages/pyd-master/pyd/infrastructure/pyd/ma
e_object.d(249,30): Error: struct
drepl.interpreter.Interpreter!(DMDEngine).Interpreter is not copyable because
it is annotated with  disable
I removed disable, but then complained about accessing the members `_engine` and `_incomplete` in Interpreter (https://github.com/dlang-community/drepl/blob/master/src/drepl/interpreter.d#L147-L148)
 ../../../.dub/packages/pyd-master/pyd/infrastructure/pyd/struct_wra
.d-mixin-56(56,15): Deprecation: std.array.Appender!(char[]).Appender._data is
not visible from module
../../../.dub/packages/pyd-master/pyd/infrastructure/pyd/struct_wra
.d-mixin-56(56,15): Error: struct std.array.Appender!(char[]).Appender member
_data is not accessible
After I made those public, it complained about `Appender`
 ../../../.dub/packages/pyd-master/pyd/infrastructure/pyd/struct_wra
.d-mixin-56(56,15): Deprecation: std.array.Appender!(char[]).Appender._data is
not visible from module
 ../../../.dub/packages/pyd-master/pyd/infrastructure/pyd/struct_wra
.d-mixin-56(56,15): Error: struct std.array.Appender!(char[]).Appender member
_data is not accessible
Is there something I can do here or would it better to talk to the Drepl guys? Thank you
Jul 29
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 29 July 2018 at 18:14:31 UTC, Nikos wrote:
 But when I try to export the whole dmdEngine

 export:

    auto engine(char[] txt) {
            return interpreter(dmdEngine());
    }
Can you export an instance of `interpreter(dmdEngine())`? e.g. __gshared auto dmdi = interpreter(dmdEngine()); export ref dmd() { return dmdi; } or if that doesn't work, proxy it __gshared auto dmdi = interpreter(dmdEngine()); struct Dmd { mixin Proxy!dmdi; } export auto dmd() { Dmd d; return d; } That is pretty much required if you want to maintain state across. Also I'm working on a D kernel for Jupyter notebook which should be done soon.
Jul 31
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 31 July 2018 at 09:09:11 UTC, Nicholas Wilson wrote:
 [snip]

 Also I'm working on a D kernel for Jupyter notebook which 
 should be done soon.
Excellent.
Jul 31
prev sibling parent reply Nikos <nikoskaragiannakis gmail.com> writes:
On Tuesday, 31 July 2018 at 09:09:11 UTC, Nicholas Wilson wrote:
 On Sunday, 29 July 2018 at 18:14:31 UTC, Nikos wrote:
 But when I try to export the whole dmdEngine

 export:

    auto engine(char[] txt) {
            return interpreter(dmdEngine());
    }
Can you export an instance of `interpreter(dmdEngine())`? e.g. __gshared auto dmdi = interpreter(dmdEngine()); export ref dmd() { return dmdi; } or if that doesn't work, proxy it __gshared auto dmdi = interpreter(dmdEngine()); struct Dmd { mixin Proxy!dmdi; } export auto dmd() { Dmd d; return d; } That is pretty much required if you want to maintain state across. Also I'm working on a D kernel for Jupyter notebook which should be done soon.
Thank you very much for your feedback. Unfortunately, none of the above worked. By the way, the reason I'm trying all this is to create a Jupyter notebook. I've already made a simple version of it some time ago (https://github.com/nikoskaragiannakis/d-jupyter-kernel). Since you are also working on a D kernel, maybe we could work together?
Aug 05
next sibling parent Laeeth Isharc <laeeth laeeth.com> writes:
On Sunday, 5 August 2018 at 20:01:22 UTC, Nikos wrote:
 On Tuesday, 31 July 2018 at 09:09:11 UTC, Nicholas Wilson wrote:
 On Sunday, 29 July 2018 at 18:14:31 UTC, Nikos wrote:
 But when I try to export the whole dmdEngine

 export:

    auto engine(char[] txt) {
            return interpreter(dmdEngine());
    }
Can you export an instance of `interpreter(dmdEngine())`? e.g. __gshared auto dmdi = interpreter(dmdEngine()); export ref dmd() { return dmdi; } or if that doesn't work, proxy it __gshared auto dmdi = interpreter(dmdEngine()); struct Dmd { mixin Proxy!dmdi; } export auto dmd() { Dmd d; return d; } That is pretty much required if you want to maintain state across. Also I'm working on a D kernel for Jupyter notebook which should be done soon.
Thank you very much for your feedback. Unfortunately, none of the above worked. By the way, the reason I'm trying all this is to create a Jupyter notebook. I've already made a simple version of it some time ago (https://github.com/nikoskaragiannakis/d-jupyter-kernel). Since you are also working on a D kernel, maybe we could work together?
Working example of Python calling D Repl is here. https://github.com/kaleidicassociates/pydrepl
Aug 05
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 5 August 2018 at 20:01:22 UTC, Nikos wrote:
 Thank you very much for your feedback. Unfortunately, none of 
 the above worked.
 By the way, the reason I'm trying all this is to create a 
 Jupyter notebook. I've already made a simple version of it some 
 time ago 
 (https://github.com/nikoskaragiannakis/d-jupyter-kernel). Since 
 you are also working on a D kernel, maybe we could work 
 together?
Sure https://github.com/thewilsonator/jupyterd There are 5 main objects: Channel, an abstraction over raw ZMQ sockets. Interpreter, abstracted for debugging and also other kernels (I work for Laeeth and they have an internal language they wan't to use from this.) Kernel, does the higher level network logic and calls the interpreter. Message and Wire message are high and low level representation of the packet layout. I'm currently trying to figure out why the message signing is bjorked (I swear it did use to work!), the IOPub channel fails intermittently (the worst kind) and the Shell/Control fails all of the time. Any effort pointing out where I done something stupid is much appreciated (and I do mean something stupid, the last major hurdle was trying to figure out why the packets were going nowhere: turns out I was missing the routing prefix). Are you on slack?
Aug 05
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 6 August 2018 at 00:47:12 UTC, Nicholas Wilson wrote:
 Sure https://github.com/thewilsonator/jupyterd
The echo interpreter now works, and the dmd one is there. I've got no idea if it works because I'm on OSX (yay no shared library support) if someone wants to test it that would be great.
Aug 09