www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - beginner's pyd question - exporting structs to python

reply "Laeeth Isharc" <laeeth laeeth.com> writes:
Hi there.

Brief introduction, and a beginner's question.

I just started playing with D a couple of weeks ago.  I have been 
programming in C on and off since the late 80s, but I do finance 
for a living and my programming skills grew rusty.  I have a bit 
more time now to catch up with developments, and D looks a very 
intriguing robust alternative to C when python won't cut the 
mustard.

I am trying to implement the Demark technical analysis indicators 
in D, and want to be able to call them from python.

So I have set up pyd okay, and the examples seem to work fine.  
Running on Fedora 20.  I am struggling a bit with the syntax on 
wrap_struct to export a struct and function returning a struct to 
python.  Here is what I have:

module hello2;

import pyd.pyd;
import std.stdio;
import std.conv;

struct t_mystruct {
   int i;
   string s;
};

t_mystruct hello(int[] inp) {
     int i;
     t_mystruct mystruct;

     mystruct.i=inp.length;
     mystruct.s="hello there";
     return mystruct;
}

extern(C) void PydMain() {
     def!(hello)();
     module_init();
     wrap_struct!(
       t_mystruct,
       "t_mystruct"
     );
}

Error:

def: hello
/root/anaconda/lib/python2.7/site-packages/celerid/infrastruc
ure/pyd/def.d(151): 
Error: static assert  "string parameters must be wrapped with 
Docstring, Mode, etc"
/root/anaconda/lib/python2.7/site-packages/celerid/infrastructure/pyd
class_wrap.d(1474): 
        instantiated from here: Args!("", "", "t_mystruct", "", 
"t_mystruct")
hellostruct.d(30):        instantiated from here: 
wrap_struct!(t_mystruct, "t_mystruct")
error: command 'dmd' failed with exit status 1
[root fedora demark]#

Apologies if it's in one of the examples - I did poke around, and 
couldn't see any obvious sample.

Many thanks.


Laeeth.
Aug 18 2014
next sibling parent reply "Laeeth Isharc" <laeeth laeeth.com> writes:
Thank to jwhear on irc who solved it for me despite claiming not 
to be a pyd guru.

In case it's of benefit, here is what works:

module hellostruct;

import pyd.pyd;
import std.stdio;
import std.conv;

struct t_mystruct {
   int i;
   string s;
};

t_mystruct hellostruct(int[] inp) {
     int i;
     t_mystruct mystruct;

     for (i=0;i<inp.length;i++)
     {
       writefln("inp " ~ to!string(i) ~ " is " ~ 
to!string(inp[i]));
     }
     writefln(" * ok, bye");
     mystruct.i=99;
     mystruct.s="hello there";
     return mystruct;
}

extern(C) void PydMain() {
     def!(hellostruct)();
     module_init();
     wrap_struct!(
       t_mystruct,
       Member!"s",
       Member!"i"
     )();
}

[root fedora demark]# cat test.py
import os.path, sys
import distutils.util

# Append the directory in which the binaries were placed to 
Python's sys.path,
# then import the D DLL.
libDir = os.path.join('build', 'lib.%s-%s' % (
     distutils.util.get_platform(),
     '.'.join(str(v) for v in sys.version_info[:2])
))
sys.path.append(os.path.abspath(libDir))

import hellostruct
inp=[1,2,3,4]
mystruct= hellostruct.hellostruct(inp)
print mystruct.i
print mystruct.s
Aug 18 2014
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 17:17 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
 Thank to jwhear on irc who solved it for me despite claiming not 
 to be a pyd guru.
:-) […]
      distutils.util.get_platform(),
[…] Does os.uname() not provide sufficient information? […]
 print mystruct.i
 print mystruct.s
Should use print as a function not as a statement. Use Python 3, or if you have to use Python 2 (which almost no-one does): from __future__ import print_function as the first statement. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
parent reply "Laeeth Isharc" <laeeth laeeth.com> writes:
On Monday, 18 August 2014 at 18:08:59 UTC, Russel Winder via 
Digitalmars-d-learn wrote:

 […]
      distutils.util.get_platform(),
[…] Does os.uname() not provide sufficient information?
This was boilerplate generated by pyd.
Aug 18 2014
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 19:00 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
 On Monday, 18 August 2014 at 18:08:59 UTC, Russel Winder via 
 Digitalmars-d-learn wrote:
 
 […]
      distutils.util.get_platform(),
[…] Does os.uname() not provide sufficient information?
This was boilerplate generated by pyd.
Hummm… if I get time I feel a PR in order. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
next sibling parent "Laeeth Isharc" <laeeth laeeth.com> writes:
On Monday, 18 August 2014 at 19:28:55 UTC, Russel Winder via 
Digitalmars-d-learn wrote:
 On Mon, 2014-08-18 at 19:00 +0000, Laeeth Isharc via 
 Digitalmars-d-learn
 wrote:
 On Monday, 18 August 2014 at 18:08:59 UTC, Russel Winder via 
 Digitalmars-d-learn wrote:
 
 […]
      distutils.util.get_platform(),
[…] Does os.uname() not provide sufficient information?
This was boilerplate generated by pyd.
Hummm… if I get time I feel a PR in order.
Perhaps I should rephrase. I copied the example as a base, so technically it wasn't generated. But no harm in tweaking the examples.
Aug 18 2014
prev sibling parent reply "Laeeth Isharc" <laeeth laeeth.com> writes:
 Dr Russel Winder
 41 Buckmaster Road
 London SW11 1EN, UK
Are there any D users groups/meetups in London? I see you are not far away (I am in Barnes). Laeeth
Aug 18 2014
parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 23:16 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:

 Are there any D users groups/meetups in London?  I see you are 
 not far away (I am in Barnes).
Crickey, that is just round the corner . :-) Yes there is a London D user group, it just hasn't met as yet. I investigated creating the London D User Group on meetup.com, but they want $30.00 per 6 months, which is sort of reasonable for them to ask, but doesn't make sense for me unless we get members, actually meet reasonably regularly and I can defray the cost somehow. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
prev sibling parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 16:39 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
 Hi there.
 
 Brief introduction, and a beginner's question.
 
 I just started playing with D a couple of weeks ago.  I have been 
 programming in C on and off since the late 80s, but I do finance 
 for a living and my programming skills grew rusty.  I have a bit 
 more time now to catch up with developments, and D looks a very 
 intriguing robust alternative to C when python won't cut the 
 mustard.
All the cool folk doing data analysis and visualization using Python no longer bother with hand written C (*) for when pure Python won't cut the mustard. If Numba can't do the job, then Cython gets used. I have all my computational pure Python source codes running as fast as C these days thanks to Numba. (And judicious profiling.) I would say that Python folk will now only be looking to C, C++, Fortran, D, for pre-written libraries in those language. Given all the codes are written in C, C++ or Fortran with none in D… On the other hand PyD does work for me, but I only pass primitive types, not C structs. (*) Nor C++, nor D. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
parent reply "Laeeth Isharc" <laeeth laeeth.com> writes:
 All the cool folk doing data analysis and visualization using 
 Python no longer bother with hand written C (*) for when pure 
 Python won't cut the mustard.  If Numba can't do the job, then 
 Cython gets used.

 I have all my computational pure Python source codes running as 
 fast as C these days thanks to Numba. (And judicious profiling.)

 I would say that Python folk will now only be looking to C, C++,
 Fortran, D, for pre-written libraries in those language. Given 
 all the codes are written in C, C++ or Fortran with none in D…
Thanks for the colour - I appreciate it. I have played with numba and pypy with numpy and it seems a powerful tool for some kinds of jobs. Perhaps it is my relative unfamiliarity with python, but for the time being I feel more comfortable with C type languages for other kinds of work. As a pragmatic idealist, one may as well use whatever tool seems to be pretty good generally and which one feels confident in wielding to accomplish the task at hand. Out of curiosity, what do you use D for given your views about the redundancy of C type languages for non-system programming? "Should use print as a function not as a statement. Use Python 3, or if you have to use Python 2 (which almost no-one does): from __future__ import print_function as the first statement." Thank you.
Aug 18 2014
parent reply Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 18:59 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
[…]
 Thanks for the colour - I appreciate it.  I have played with
 numba and pypy with numpy and it seems a powerful tool for some
 kinds of jobs.  Perhaps it is my relative unfamiliarity with
 python, but for the time being I feel more comfortable with C
 type languages for other kinds of work.  As a pragmatic idealist,
 one may as well use whatever tool seems to be pretty good
 generally and which one feels confident in wielding to accomplish
 the task at hand.
Whilst the hardcore Pythonistas remain Pythonistas, some of the periphery has jumped ship to Go. Sadly D did not capture these folk, it perhaps should have done. It would be easy to blame fadism, but I think the actual reasons are far less superficial. For me, NumPy has some serious problems despite being the accepted norm for computational work. The USP for Python/NumPy/SciPy/Matplotlib is that it costs an infinite amount less than Mathematica and Matlab. This is an non-trivial issue from which I am gaining a lot of business :-)
 Out of curiosity, what do you use D for given your views about
 the redundancy of C type languages for non-system programming?
In a sense I could rightly be labelled a D dilettante. I had a possible startup a couple of years ago where we would have used D, but it never started. A side-effect was that I gave some support to David Simcha creating std.parallelism, but for the last couple of years my income has come from Python or Groovy/GPars with no real D activity. […] -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
parent reply "Laeeth Isharc" <laeeth laeeth.com> writes:
 Whilst the hardcore Pythonistas remain Pythonistas, some of the
 periphery has jumped ship to Go. Sadly D did not capture these 
 folk, it perhaps should have done. It would be easy to blame 
 fadism, but I think the actual reasons are far less superficial.
So I gather that you agree that "what everyone is doing" may not be the best in this case (python vs D) if there are no direct network effects beyond libraries and getting help and you have the freedom to determine your own platform choices?
 For me, NumPy has some serious problems despite being the 
 accepted norm for computational work.
If not too offtopic, do you have a link describing, or would you briefly summarize these problems? I am intrigued. And what would you suggest in its place? Fortran?
 Out of curiosity, what do you use D for given your views about
 the redundancy of C type languages for non-system programming?
In a sense I could rightly be labelled a D dilettante. I had a possible startup a couple of years ago where we would have used D, but it never started. A side-effect was that I gave some support to David Simcha creating std.parallelism, but for the last couple of years my income has come from Python or Groovy/GPars with no real D activity.
Would you consider D stable enough/suitable for general financial market work with development initially by a small underresourced team? Not ultra high frequency execution - at most legging in and managing longer term positions. But I am more interested in sentiment analysis, producing technical analysis indicators that summarize market activity across many different securities, some bond arb stuff. C++ just seems so ugly, and I feel uncomfortable only having python in the toolbox. D seems so far to be quite suitable...
Aug 18 2014
next sibling parent reply "bachmeier" <no spam.net> writes:
On Monday, 18 August 2014 at 23:12:28 UTC, Laeeth Isharc wrote:
 For me, NumPy has some serious problems despite being the 
 accepted norm for computational work.
If not too offtopic, do you have a link describing, or would you briefly summarize these problems? I am intrigued. And what would you suggest in its place? Fortran?
I'm not sure which computational work he is referring to, but for statistical analysis, R dominates by a wide margin (although statistical analysis done in Silicon Valley, the type you read about on Hacker News, is often done using Python). I write functions in D, compile as a shared library, and call from R. The more statically typed code I write, the more I like it. I'm finishing up a blog post describing my usage.
Aug 18 2014
parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Tue, 2014-08-19 at 00:53 +0000, bachmeier via Digitalmars-d-learn
wrote:
[…]
 I'm not sure which computational work he is referring to, but for 
 statistical analysis, R dominates by a wide margin (although 
 statistical analysis done in Silicon Valley, the type you read 
 about on Hacker News, is often done using Python).
My own :-) I think Python is not making inroads in the world-wide R community, but it is in the Matlab and Mathematica ones. Whether Julia acts as a disruptive technology here we will see.
 I write functions in D, compile as a shared library, and call 
 from R. The more statically typed code I write, the more I like 
 it. I'm finishing up a blog post describing my usage.
This model also works for Python and D and does not require PyD (*). The issue here is that D, like C++, can provide shared objects (aka DLLs) with C linkage entry points and so Python extensions can be created. (*) Though arguably it is easier using PyD. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
prev sibling next sibling parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 23:12 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
 Whilst the hardcore Pythonistas remain Pythonistas, some of the
 periphery has jumped ship to Go. Sadly D did not capture these 
 folk, it perhaps should have done. It would be easy to blame 
 fadism, but I think the actual reasons are far less superficial.
So I gather that you agree that "what everyone is doing" may not be the best in this case (python vs D) if there are no direct network effects beyond libraries and getting help and you have the freedom to determine your own platform choices?
Go arrived with high marketing as something new, which it is. It is a stripped down, strongly typed C with memory management, objects and extension methods, and (most importantly) goroutines as a lightweight built-in concurrency and parallelism framework based on thread pools. The language itself is very simple (apart from the consequences of semi-colon ellision/re-adding), and so captured the imagination of many. Not unreasonably. I really quite like Go. However it has some, for me, serious irritants, most especially an obsessive hatred of exceptions, enforcing return codes and error handling at the point of call – though it does support this in a neat way. One effect of the "fanboi" element of using go has been very rapid evolution of the eco-system, especially strong because of the package approach and the use of DVCS as a tool for accessing packages. D is not a new language so doesn't have the "new shiny toy" feature, but it remains an rapidly evolving language. It is definitely a better C++ but C++ is evolving fast enough that C++ folks stay with C++. So in a sense D has failed to capture the market it aimed for when it started. Also it is still playing "catch up" in the eco-system stakes. D does however have a very neat model of abstraction that allows for a very functional programming approach. It isn't functional programming per se, but it gets very declarative. C++ is a long way behind in this, but there is little C++ → D transfer. The problem for Python folks looking for a native code language is that D is a big language and Go is a small language. And then there is Rust…
 For me, NumPy has some serious problems despite being the 
 accepted norm for computational work.
If not too offtopic, do you have a link describing, or would you briefly summarize these problems? I am intrigued. And what would you suggest in its place? Fortran?
I'll start a separate thread for this one. […]
 Would you consider D stable enough/suitable for general financial 
 market work with development initially by a small underresourced 
 team?  Not ultra high frequency execution - at most legging in 
 and managing longer term positions.  But I am more interested in 
 sentiment analysis, producing technical analysis indicators that 
 summarize market activity across many different securities, some 
 bond arb stuff.  C++ just seems so ugly, and I feel uncomfortable 
 only having python in the toolbox.  D seems so far to be quite 
 suitable...
The most important thing about any project using a programming language is that the development team, testing team and deployment team all approve of the language being used. It sounds like you approve of D so go for it. The language is not totally stable, so expect incompatibilities on upgrade. I am not a fan of all this "backward compatibility" obsession that has gripped Fortran, C++ and Java (though I can see why it is necessary), but with D you will need to budget for upgrade-related refactoring. Personally I was happy to take that hit. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014
prev sibling parent Russel Winder via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:
On Mon, 2014-08-18 at 23:12 +0000, Laeeth Isharc via Digitalmars-d-learn
wrote:
[…]
 For me, NumPy has some serious problems despite being the 
 accepted norm for computational work.
If not too offtopic, do you have a link describing, or would you briefly summarize these problems? I am intrigued. And what would you suggest in its place? Fortran?
[…] I have no benchmark experiment data as proof yet, just anecdotal evidence forming an hypothesis, but it seems that the underlying data parallelism model of NumPy has some serious overhead problems: speed-ups are not as high as they should be, and scaling is not as good as it should be. The finance people using Python in London, and indeed the general data analysis using Python folk (cf. PyData meetings around the world) all take NumPy as a given, and that it works well enough for them. I guess those for whom NumPy is not good enough are using Cython, C++ or Fortran (or even C) for the computationally intensive stuff. Or more likely they already had the native code in place and so are not using NumPy other than for data visualization and replacement of Matlab. I think Numba is a disruptive technology here. However the danger is that the opaque type approach of NumPy (which is good) is forgotten as a good abstraction in the face of Numba speeds, with people reverting to explicit rather than implicit iteration just because things go faster. Or maybe this won't happen because all the needed computationally intensive libraries already exists. I guess the PyData meetings are the place where al this will be played out. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 18 2014