www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D extensions to python, inline in an ipython/jupyter notebook

reply "John Colvin" <john.loughran.colvin gmail.com> writes:
Bare-bones at the moment, but hopefully of interest to some people

https://github.com/DlangScience/PydMagic

This allows you to write D extensions to python, as inline cells 
in an ipython/jupyter notebook.
Jun 14 2015
next sibling parent reply Russel Winder via Digitalmars-d-announce writes:
On Sun, 2015-06-14 at 15:51 +0000, John Colvin via Digitalmars-d
-announce wrote:
 Bare-bones at the moment, but hopefully of interest to some people
=20
 https://github.com/DlangScience/PydMagic
=20
 This allows you to write D extensions to python, as inline cells=20
 in an ipython/jupyter notebook.
I am doing a talk on using C++, D and Chapel in Python to the PyData London 2015 people, many I am sure will want to use IPython/Jupyter. If you can guide me to a short demo that has maximum impact for the least risk [*], I can certainly give it an airing. [*] As ever with my live demos, the demo gods always want a say ;-) --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jun 14 2015
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 14 June 2015 at 16:14:58 UTC, Russel Winder wrote:
 On Sun, 2015-06-14 at 15:51 +0000, John Colvin via 
 Digitalmars-d -announce wrote:
 Bare-bones at the moment, but hopefully of interest to some 
 people
 
 https://github.com/DlangScience/PydMagic
 
 This allows you to write D extensions to python, as inline 
 cells in an ipython/jupyter notebook.
I am doing a talk on using C++, D and Chapel in Python to the PyData London 2015 people, many I am sure will want to use IPython/Jupyter. If you can guide me to a short demo that has maximum impact for the least risk [*], I can certainly give it an airing. [*] As ever with my live demos, the demo gods always want a say ;-)
Ok, see examples/test.ipynb it's not amazing but it's a start.
Jun 14 2015
prev sibling next sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
 Bare-bones at the moment, but hopefully of interest to some 
 people

 https://github.com/DlangScience/PydMagic

 This allows you to write D extensions to python, as inline 
 cells in an ipython/jupyter notebook.
This was just what I was looking for - great work, and I think this will be very useful. Laeeth
Jun 14 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 15 June 2015 at 05:11:17 UTC, Laeeth Isharc wrote:
 On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
 Bare-bones at the moment, but hopefully of interest to some 
 people

 https://github.com/DlangScience/PydMagic

 This allows you to write D extensions to python, as inline 
 cells in an ipython/jupyter notebook.
This was just what I was looking for - great work, and I think this will be very useful. Laeeth
It's really only a beginning. Pyd's API is pretty clunky feeling by modern D standards, I hope to extend pdef to automatically work for most language constructs, amongst other sugar. I also aim to factor it out in to a library that doesn't depend on ipython, so people can just get a nicer Pyd. Ultimately maybe it could replace the upstream API. Any contributions are very much welcome, it's a fun thing to work on but I'm crazily busy at the moment. Anyone who's interested, get in touch :)
Jun 14 2015
parent reply "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
 It's really only a beginning. Pyd's API is pretty clunky 
 feeling by modern D standards, I hope to extend  pdef to 
 automatically work for most language constructs, amongst other 
 sugar.
Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen.
 Any contributions are very much welcome, it's a fun thing to 
 work on but I'm crazily busy at the moment. Anyone who's 
 interested, get in touch :)
Have some other constraints but will get in touch later if I can. I hope this wasn't premature to submit: http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/
Jun 15 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:
 On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
 [...]
Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen.
I haven't looked at torch much, is it up to the hype?
 [...]
Have some other constraints but will get in touch later if I can. I hope this wasn't premature to submit: http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/
Perhaps a little early but I'm sure it won't do any harm.
Jun 15 2015
next sibling parent reply "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote:
 On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:
 On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
 [...]
Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen.
I haven't looked at torch much, is it up to the hype?
To be direct: I don't know, as I haven't used it. I was just interested in seeing how it fitted with ipython and bokeh in particular. I liked your talk at dconf. Looks like progress is being made on planting the seeds of a matrix implementation and computation library on top. Having a way to use D code to explore data within an approach of rapid iteration will be useful too, and I think your work is a great start on this (seems like its value as it is is very much greater than you modestly suggest, and also I am amazed by how short the code is). At the moment I am using a combination of the python interface to Bokeh and D code to do the work. It would be great to port the bokeh server side stuff to D - I have made a start, but I haven't the space for a concerted effort. The funny thing is there is nothing to the server side - all you do is have a shallow layer that translates the cumulative state from API calls to JSON; the hard work of drawing stuff is done in Javascript on the client. Whether it's Bokeh or not, having some kind of visualisation solution that fits well with iteratively exploring data in D will be nice. (What you have done means it's already enough to be very usable, but there may be benefits to moving the charting itself to D). Perhaps not for scientific applications, but for finance: reporting is also something that is an important and rising theme in use cases. Not just for accounting, but for example to understand portfolio risk in a richer way. You have for a long time been able to hook up reportlab with D code using PyD, but this element of rapid iteration was not so easy - small frictions have large consequences when you don't have a big team. So in any case, I really appreciate your work. One more thing - any thoughts on best spreadsheet like interface either purely on browser or within ipython to display results and allow user to enter parameters? My friend, Giles Thomas, wrote Resolver but that's too big and aimed at something different, and I don't know if it is being maintained properly. Laeeth
Jun 15 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 15 June 2015 at 19:35:16 UTC, Laeeth Isharc wrote:
 On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote:
 On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:
 On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
 [...]
Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen.
I haven't looked at torch much, is it up to the hype?
To be direct: I don't know, as I haven't used it. I was just interested in seeing how it fitted with ipython and bokeh in particular. I liked your talk at dconf.
Thank you :)
 Looks like progress is being made on planting the seeds of a 
 matrix implementation and computation library on top.
Ilya Yaroshenko is doing great work here.
 Having a way to use D code to explore data within an approach 
 of rapid iteration will be useful too, and I think your work is 
 a great start on this (seems like its value as it is is very 
 much greater than you modestly suggest, and also I am amazed by 
 how short the code is).
Well, as i briefly mentioned in my talk, it's really very easy to do, Jupyter/IPython makes it simple. Of course, behind the scenes Pyd is doing the hard work.
 At the moment I am using a combination of the python interface 
 to Bokeh and D code to do the work.  It would be great to port 
 the bokeh server side stuff to D - I have made a start, but I 
 haven't the space for a concerted effort.  The funny thing is 
 there is nothing to the server side - all you do is have a 
 shallow layer that translates the cumulative state from API 
 calls to JSON; the hard work of drawing stuff is done in 
 Javascript on the client.

 Whether it's Bokeh or not, having some kind of visualisation 
 solution that fits well with iteratively exploring data in D 
 will be nice.  (What you have done means it's already enough to 
 be very usable, but there may be benefits to moving the 
 charting itself to D).
Visualisation is one of those things that no-one seems to have got right, despite the huge efforts that have gone in to it. I have ideas about D-based visualisation and already my own basic 3-D plotting library for huge datasets, but for now the link to python is key to get quick and easy access to the latest tools that the scientific and finance world are creating.
 Perhaps not for scientific applications, but for finance: 
 reporting is also something that is an important and rising 
 theme in use cases.  Not just for accounting, but for example 
 to understand portfolio risk in a richer way.  You have for a 
 long time been able to hook up reportlab with D code using PyD, 
 but this element of rapid iteration was not so easy - small 
 frictions have large consequences when you don't have a big 
 team.

 So in any case, I really appreciate your work.

 One more thing - any thoughts on best spreadsheet like 
 interface either purely on browser or within ipython to display 
 results and allow user to enter parameters?  My friend, Giles 
 Thomas, wrote Resolver but that's too big and aimed at 
 something different, and I don't know if it is being maintained 
 properly.
There is http://nbviewer.ipython.org/gist/rossant/9463955 (see https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which looks like a good starting point, but it's very basic. If https://github.com/quantopian/qgrid supported editing (perhaps it does already, but I can't see any mention of it) then that would probably be better.
Jun 16 2015
parent "Laeeth Isharc" <nospamlaeeth laeethnospam.laeeth.com> writes:
On Tuesday, 16 June 2015 at 13:11:51 UTC, John Colvin wrote:
 Looks like progress is being made on planting the seeds of a 
 matrix implementation and computation library on top.
Ilya Yaroshenko is doing great work here.
Yes - Ilya's work looks great, and Vlad Levenfeld has some interesting ideas too (less specifically on matrices, but relating to them).
 Well, as i briefly mentioned in my talk, it's really very easy 
 to do, Jupyter/IPython makes it simple. Of course, behind the 
 scenes Pyd is doing the hard work.
Yes - so often it's the last bit of integration that isn't so hard but removes frictions that makes it possible to realize the value created by all the foundational meta work. And solving one's own problems is often a route to completing that integration.
 Visualisation is one of those things that no-one seems to have 
 got right, despite the huge efforts that have gone in to it. I 
 have ideas about D-based visualisation and already my own basic 
 3-D plotting library for huge datasets, but for now the link to 
 python is key to get quick and easy access to the latest tools 
 that the scientific and finance world are creating.
Would love to see it when it is ready (although I haven't used 3d plots much myself). Agree about python, for better or for worse.
 There is http://nbviewer.ipython.org/gist/rossant/9463955 (see 
 https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which 
 looks like a good starting point, but it's very basic.

 If https://github.com/quantopian/qgrid supported editing 
 (perhaps it does already, but I can't see any mention of it) 
 then that would probably be better.
Thank you for these - I will take a look. Laeeth.
Jun 16 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/15 10:28 AM, John Colvin wrote:
 On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:
 On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
 [...]
Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen.
I haven't looked at torch much, is it up to the hype?
Yann LeCun recommended it; I think that's a good endorsement. -- Andrei
Jun 15 2015
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
 Bare-bones at the moment, but hopefully of interest to some 
 people

 https://github.com/DlangScience/PydMagic

 This allows you to write D extensions to python, as inline 
 cells in an ipython/jupyter notebook.
v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing. Highlights: Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system Extended the pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more. Many bugs fixed.
Jun 30 2015
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:
 On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
 [...]
v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing. Highlights: Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system Extended the pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more. Many bugs fixed.
Please report any bugs, suggestions, problems, anything. It all helps.
Jun 30 2015
prev sibling next sibling parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:
 On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
 Bare-bones at the moment, but hopefully of interest to some 
 people

 https://github.com/DlangScience/PydMagic

 This allows you to write D extensions to python, as inline 
 cells in an ipython/jupyter notebook.
v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing. Highlights: Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system Extended the pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more. Many bugs fixed.
This is great - looking forward to playing with it. One nice idea I have is to be able to flip between the notebook for charting and specifying reports and Excel for viewing and exploring the results. Not just for me, but some people are much more comfortable with Excel. (would use a backend server that is communicated with via nanomsg IPC so excel can see same data)
Jun 30 2015
prev sibling parent reply Russel Winder via Digitalmars-d-announce writes:
On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
-announce
 [=E2=80=A6]
=20
 Ditched distutils in favour of dub. This is easier for me to=20
 maintain and fits much better with the rest of the D ecosystem
I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jul 02 2015
next sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
 On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
 -announce
  […]
 
 Ditched distutils in favour of dub. This is easier for me to 
 maintain and fits much better with the rest of the D ecosystem
I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.
What is the benefit from using distutils for working with D in a notebook? There are two standards - the Python one, and the D one. The advantage of using dub is that it becomes wonderfully easy to pull in D projects from code.dlang.org and to compile your own work developed under dub. (And dub itself continues to improve). Linking to a D project of decent complexity via distutils is not my idea of fun. I do agree that for pyd itself it would be nice to retain the option of distutils (although I would love to see dub added also), One should look at this as an alpha, extremely promising project. It is not any rough edges that are important at this stage, but that it has been done at all (in a form that is already very valuable). The set of people that want to use computers to explore larger data sets than lately considered comfortable in an iterative manner is not small, even confining It just to finance. It's of real value to be able to hook in to a proper back end that manages market and static data, with also any data processing and analytics also in D, and then to have a pretty and friendly front end that can just talk to this code on the back with minimal messing around to get there. It's also very nice to have both Jupyter and an Excel spreadsheet at windows onto the data on your local machine or in the enterprise cloud. The frictions to starting to play with D in a notebook are much lower than going via the command line, so I really am not sure if we should be worried about making it marketable at this stage rather than useful for doing real work. It's potentially a nice debugging tool where you are trying to make sense of things that don't tidily fit in a debugging window, and where there is just too much stuff to do it via writefln or logging without putting lots of effort into the infrastructure to find events first. I do like the simpler syntax (than PyD). One question is whether you need to wrap every member of a struct. But it's a very useful start as it stands.
Jul 02 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 2 July 2015 at 19:51:19 UTC, Laeeth Isharc wrote:
 What is the benefit from using distutils for working with D in 
 a notebook?  There are two standards - the Python one, and the 
 D one.  The advantage of using dub is that it becomes 
 wonderfully easy to pull in D projects from code.dlang.org and 
 to compile your own work developed under dub.  (And dub itself 
 continues to improve).  Linking to a D project of decent 
 complexity via distutils is not my idea of fun.
This was my train of thought too.
 I do agree that for pyd itself it would be nice to retain the 
 option of distutils (although I would love to see dub added 
 also),
pyd does have rudimentary dub support, but it's only for embedding python in D, not the other way around. Extension support is on my TODO list, it's not complicated.
 One should look at this as an alpha, extremely promising 
 project.
  It is not any rough edges that are important at this stage, 
 but that it has been done at all (in a form that is already 
 very valuable).

 The set of people that want to use computers to explore larger 
 data sets than lately considered comfortable in an iterative 
 manner is not small, even confining It just to finance.  It's 
 of real value to be able to hook in to a proper back end that 
 manages market and static data, with also any data processing 
 and analytics also in D, and then to have a pretty and friendly 
 front end that can just talk to this code on the back with 
 minimal messing around to get there.  It's also very nice to 
 have both Jupyter and an Excel spreadsheet at windows onto the 
 data on your local machine or in the enterprise cloud.

 The frictions to starting to play with D in a notebook are much 
 lower than going via the command line, so I really am not sure 
 if we should be worried about making it marketable at this 
 stage rather than useful for doing real work.  It's potentially 
 a nice debugging tool where you are trying to make sense of 
 things that don't tidily fit in a debugging window, and where 
 there is just too much stuff to do it via writefln or logging 
 without putting lots of effort into the infrastructure to find 
 events first.

 I do like the simpler syntax (than PyD).  One question is 
 whether you need to wrap every member of a struct.
pdefRecursive!() or pdef!(Recursive.yes) or pdef!(Recursive) ? It's totally doable.
 But it's a very useful start as it stands.

 As an addendum: not many people prefer to do non system type 
 things in C than a higher level language when time is money.  
 So the mindset is you build a C extension to address the worst 
 bits where Python's true colours shine through (ie when you are 
 in a tight loop within Python interpreter and not spending most 
 of the time in its C libraries.  Or writing C glue to connect 
 Python to another library.

 In my view, D is different.  I would rather write in D than 
 Python, and to me it's much better Tor doing serious work.  
 Even for parsing a CSV I prefer it (although one could debate 
 the point).  In any case D is not especially slower to write in 
 than Python (particularly when you include time spent getting 
 the bugs out), and in a decent number of cases it may be more 
 productive.

 So why bother with Python at all ?  Better ecosystem for 
 charting and exploring data, and an interpreter is better 
 suited generally for certain kinds of tasks.  Also in some 
 areas more libraries, which can be helpful to get a quick 
 result, so that one can go back and do it properly later.
Libraries, libraries, libraries. In a perfect world we can reimplement everything in D and it'll be awesome, but in the "getting shit done" world that normal people inhabit they need it to work, today. Bridges to other languages gives us that. E.g. I have a 3D plotting library in D (still in stealth mode on that one) that uses matplotlib via pyd to generate tick labels using either the builtin TeX parser & renderer (OK, fast) or actually calling out to latex (perfect, slow). Could I have done that all myself in D? Sure. Would I? No, i would still have terribly rendered tick labels with no TeX support. I don't have time to write a TeX implementation, even one as rudimentary and limited as matplotlib's. In short, you're right.
 It would be v helpful to have a Datetime conversion from D.  
 Looks like there is a macro for converting from ymd in 
 datetime.h, so I guess one could just write some code against 
 this API in C, D, or Cython and link it in with D so one can 
 transfer data structures over more easily.
I know just enough about that topic to be very scared.
Jul 02 2015
parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
 It would be v helpful to have a Datetime conversion from D.  
 Looks like there is a macro for converting from ymd in 
 datetime.h, so I guess one could just write some code against 
 this API in C, D, or Cython and link it in with D so one can 
 transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I think, for D to Python. Not sure about Python to D, but maybe. https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d From D to Python datetime.datetime =================================== PyObject *PyDate_FromDate()(int year, int month, int day) { return PyDateTimeAPI.Date_FromDate(year, month, day, PyDateTimeAPI.DateType); } PyObject *PyDateTime_FromDateAndTime()(int year, int month, int day, int hour, int min, int sec, int usec) { return PyDateTimeAPI.DateTime_FromDateAndTime(year, month, day, hour, min, sec, usec, Py_None, PyDateTimeAPI.DateTimeType); } From Python datetime.datetime to D =================================== // D translations of C macros: int PyDateTime_GET_YEAR()(PyObject *o) { PyDateTime_Date *ot = cast(PyDateTime_Date *) o; return (ot.data[0] << 8) | ot.data[1]; } int PyDateTime_GET_MONTH()(PyObject *o) { PyDateTime_Date *ot = cast(PyDateTime_Date *) o; return ot.data[2]; } int PyDateTime_GET_DAY()(PyObject *o) { PyDateTime_Date *ot = cast(PyDateTime_Date *) o; return ot.data[3]; } int PyDateTime_DATE_GET_HOUR()(PyObject *o) { PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o; return ot.data[4]; } int PyDateTime_DATE_GET_MINUTE()(PyObject *o) { PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o; return ot.data[5]; } int PyDateTime_DATE_GET_SECOND()(PyObject *o) { PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o; return ot.data[6]; } int PyDateTime_DATE_GET_MICROSECOND()(PyObject *o) { PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o; return (ot.data[7] << 16) | (ot.data[8] << 8) | ot.data[9]; }
Jul 02 2015
parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
 It would be v helpful to have a Datetime conversion from D.  
 Looks like there is a macro for converting from ymd in 
 datetime.h, so I guess one could just write some code against 
 this API in C, D, or Cython and link it in with D so one can 
 transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I think, for D to Python. Not sure about Python to D, but maybe. https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh! It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function. So all that is needed is the following: module example; import std.datetime; import std.stdio; import pyd.pyd; import std.conv; import deimos.python.datetime; DateTime foo() { return DateTime(1999,1,1); } extern(C) void PydMain() { ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0)); def!foo(); module_init(); PyDateTime_IMPORT(); } Ideally, shouldn't PyD do this for you ? I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be. Thanks. Laeeth.
Jul 03 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
 On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
 It would be v helpful to have a Datetime conversion from D.  
 Looks like there is a macro for converting from ymd in 
 datetime.h, so I guess one could just write some code 
 against this API in C, D, or Cython and link it in with D so 
 one can transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I think, for D to Python. Not sure about Python to D, but maybe. https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh! It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function. So all that is needed is the following: module example; import std.datetime; import std.stdio; import pyd.pyd; import std.conv; import deimos.python.datetime; DateTime foo() { return DateTime(1999,1,1); } extern(C) void PydMain() { ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0)); def!foo(); module_init(); PyDateTime_IMPORT(); } Ideally, shouldn't PyD do this for you ? I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be. Thanks. Laeeth.
Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?
Jul 03 2015
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:
 Also, there would have to be some serious warning signs about 
 translating from python to D, in that the micro-seconds will be 
 truncated. Doesn't a lack of microseconds make it unusable for 
 tick data?
Woops, I means milliseconds.
Jul 03 2015
prev sibling next sibling parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:
 On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
 On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
 It would be v helpful to have a Datetime conversion from D.
  Looks like there is a macro for converting from ymd in 
 datetime.h, so I guess one could just write some code 
 against this API in C, D, or Cython and link it in with D 
 so one can transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I think, for D to Python. Not sure about Python to D, but maybe. https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh! It's already in deimos, but you need to call PyDateTime_IMPORT(); before calling the conversion function. So all that is needed is the following: module example; import std.datetime; import std.stdio; import pyd.pyd; import std.conv; import deimos.python.datetime; DateTime foo() { return DateTime(1999,1,1); } extern(C) void PydMain() { ex_d_to_python((DateTime dt) => PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0)); def!foo(); module_init(); PyDateTime_IMPORT(); } Ideally, shouldn't PyD do this for you ? I think the code is already there in make_object.d, but it doesn't work - maybe because PyDateTime_IMPORT() has not been called if it needs to be. Thanks. Laeeth.
Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data?
I just put 0 in the microseconds field as I was too tired to look up how D does that for now and it wasn't so important for my proof of concept. But python has microseconds (though not smaller, I think). Numpy and pandas have their own datetime64 concept - I have not so nice memories of using these, but I am sure it must be possible to convert at C API level. Worst case if there are no system calls to do the conversion, just hack up the current binary representation of the struct and deal with the mess when they change the version. But I wouldn't suggest you put that in PyD unless we can do it via official call! I do think regular datetime.datetime should be in there, though. I think it would probably only need a handful of lines added to the file I mentioned - I thought they were already there, but that was my own change! I am not sure where the init date call should go, and I am wary of tinkering with PyD when I don't yet know the codebase, nor the Python C API. I think someone who knows it could do it in a few minutes. Is it a problem not having sub micro second resolution? It might be for some people, but then it's a core problem, and they probably have the budget to do it themselves and might have their own date representation anyway (which nonetheless might most conveniently be mapped to datetime64 given all the python libraries). However, it's not my use case. (I guess those who need are scientists and investors who analyze data at a high-frequency level, which is not me - 1 minute bars are enough for now, and my holding period will be much longer). It's incrementally a big improvement to start by adding datetime.datetime... Laeeth.
Jul 03 2015
prev sibling parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
 Aren't there time-zone concerns? Or is this just a mapping 
 between D's std.datetime.DateTime and python's 
 datetime.datetime with tzinfo==None, i.e. a naive date?

 Also, there would have to be some serious warning signs about 
 translating from python to D, in that the micro-seconds will be 
 truncated. Doesn't a lack of microseconds make it unusable for 
 tick data?
https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free. There are libraries that implement datetimes with a timezone. So sensible options are: 1. datetime.datetime<->std.datetime.DateTime throwing away fractions of a second for python -> D. 2. datetime.datetime<->std.datetime.SysTime presuming python time is UTC and 2a) throwing away SysTime timezone and assuming it is UTC 2b) converting SysTime to a UTC time (and presuming python time is UTC). I would go for 1 as default and allowing 2b as an option. The problem is that if you implement the mapping in the pyd main code then user cannot override it in custom conversion code because it is only called if the inbuilt conversion fails. So maybe 2b is the only sensible feasible option.
Jul 03 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
 Aren't there time-zone concerns? Or is this just a mapping 
 between D's std.datetime.DateTime and python's 
 datetime.datetime with tzinfo==None, i.e. a naive date?

 Also, there would have to be some serious warning signs about 
 translating from python to D, in that the micro-seconds will 
 be truncated. Doesn't a lack of microseconds make it unusable 
 for tick data?
https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.
both datetime.datetime and numpy.datetime64 have timezone support.
Jul 03 2015
parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:
 On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
 Aren't there time-zone concerns? Or is this just a mapping 
 between D's std.datetime.DateTime and python's 
 datetime.datetime with tzinfo==None, i.e. a naive date?

 Also, there would have to be some serious warning signs about 
 translating from python to D, in that the micro-seconds will 
 be truncated. Doesn't a lack of microseconds make it unusable 
 for tick data?
https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.
both datetime.datetime and numpy.datetime64 have timezone support.
Great - in that case 2b makes sense.
Jul 03 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Saturday, 4 July 2015 at 00:14:51 UTC, Laeeth Isharc wrote:
 On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:
 On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
 Aren't there time-zone concerns? Or is this just a mapping 
 between D's std.datetime.DateTime and python's 
 datetime.datetime with tzinfo==None, i.e. a naive date?

 Also, there would have to be some serious warning signs 
 about translating from python to D, in that the 
 micro-seconds will be truncated. Doesn't a lack of 
 microseconds make it unusable for tick data?
https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free.
both datetime.datetime and numpy.datetime64 have timezone support.
Great - in that case 2b makes sense.
or rather, not 2b, but just mapping python + numpy to a SysTime
Jul 03 2015
prev sibling next sibling parent "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
 On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
 -announce
  […]
 
 Ditched distutils in favour of dub. This is easier for me to 
 maintain and fits much better with the rest of the D ecosystem
I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.
As an addendum: not many people prefer to do non system type things in C than a higher level language when time is money. So the mindset is you build a C extension to address the worst bits where Python's true colours shine through (ie when you are in a tight loop within Python interpreter and not spending most of the time in its C libraries. Or writing C glue to connect Python to another library. In my view, D is different. I would rather write in D than Python, and to me it's much better Tor doing serious work. Even for parsing a CSV I prefer it (although one could debate the point). In any case D is not especially slower to write in than Python (particularly when you include time spent getting the bugs out), and in a decent number of cases it may be more productive. So why bother with Python at all ? Better ecosystem for charting and exploring data, and an interpreter is better suited generally for certain kinds of tasks. Also in some areas more libraries, which can be helpful to get a quick result, so that one can go back and do it properly later. It would be v helpful to have a Datetime conversion from D. Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
Jul 02 2015
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:
 On Tue, 2015-06-30 at 18:20 +0000, John Colvin via Digitalmars-d
 -announce
  […]
 
 Ditched distutils in favour of dub. This is easier for me to 
 maintain and fits much better with the rest of the D ecosystem
I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one.
There are 2 seperate points where a build/install system is needed: 1) Installing the extension. It's just one python file, which makes it a good fit for %install_ext but of course I could wrap it in a trivial setup.py to make people feel more comfortable / make it fit in with easy_install/pip or related tools. 2) building the inline D code and sorting out linking to other libraries etc. The basics of this can be done with distutils (it's how pyd traditionally does it), but seeing as it's all D code from this point, dub is a better fit, otherwise people have to have manually install any D libraries they want to use. I originally modelled PydMagic on how the %%cython inline magic works. %%cython uses distutils under the scenes with minor leakage for the compilation options and PydMagic uses dub, with similar levels of leakage. The recently-spun-off-from-PydMagic work of making a nicer API for pyd ( github.com/John-Colvin/ppyd which PydMagic pulls in via dub) is independent of the build system, so you can use pyd's distutils extensions or dub. This is the project you would use if you want to write a proper python extension in D, as opposed to little snippets in a Jupyter notebook.
Jul 02 2015