www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D for project in computational chemistry

reply "Yura" <min_yura mail.ru> writes:
Dear D coders/developers,

I am just thinking on one project in computational chemistry, and 
it is sort of difficult for me to pick up the right language this 
project to be written. The project is going to deal with the 
generation of the molecular structures and will resemble to some 
extent some bio-informatic stuff. Personally I code in two 
languages - Python, and a little bit in C (just started to learn 
this language).

While it is easy to code in Python there are two things I do not 
like:

1) Python is slow for nested loops (much slower comparing to C)
2) Python is not compiled. However, I want to work with a code 
which can be compiled and distributed as binaries (at least at 
the beginning).

When it comes to C, it is very difficult to code (I am a chemist 
rather than computer scientist). The pointers, memory allocation, 
absence of the truly dynamically allocated arrays, etc, etc make 
the coding very long. C is too low level I believe.

I just wander how D would be suitable for my purpose? Please, 
correct me if I am wrong, but in D the need of pointers is 
minimal, there is a garbage collector, the arrays can be 
dynamically allocated, the arrays can be sliced, ~=, etc which 
makes it similar to python at some extent. I tried to write a 
little code in D and it was very much intuitive and similar to 
what I did both in Python and C.

Any hints/thoughts/advises?

With kind regards,
Yury
Aug 02 2015
next sibling parent "ZombineDev" <valid_email he.re> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, 
 and it is sort of difficult for me to pick up the right 
 language this project to be written. The project is going to 
 deal with the generation of the molecular structures and will 
 resemble to some extent some bio-informatic stuff. Personally I 
 code in two languages - Python, and a little bit in C (just 
 started to learn this language).

 While it is easy to code in Python there are two things I do 
 not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code 
 which can be compiled and distributed as binaries (at least at 
 the beginning).

 When it comes to C, it is very difficult to code (I am a 
 chemist rather than computer scientist). The pointers, memory 
 allocation, absence of the truly dynamically allocated arrays, 
 etc, etc make the coding very long. C is too low level I 
 believe.

 I just wander how D would be suitable for my purpose? Please, 
 correct me if I am wrong, but in D the need of pointers is 
 minimal, there is a garbage collector, the arrays can be 
 dynamically allocated, the arrays can be sliced, ~=, etc which 
 makes it similar to python at some extent. I tried to write a 
 little code in D and it was very much intuitive and similar to 
 what I did both in Python and C.

 Any hints/thoughts/advises?

 With kind regards,
 Yury
I'd say go for it. My experience with D is that you can use it both for fast (to write and execute) scripts and for large enterprise applications. You can certainly view it as a easier version of C, though it can offer a lot more if you need it. 90% of the syntax is the same as C, so there shouldn't be gotchas in the basic stuff. Recently at DConf [1] John Colvin gave a talk [2] about using D for science which will probably be interesting for you. Good luck :) [1]: http://dconf.org/2015/schedule/index.html [2]: https://www.youtube.com/watch?v=edjrSDjkfko D Is For Science
Aug 02 2015
prev sibling next sibling parent "Daniel N" <ufo orbiting.us> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 Any hints/thoughts/advises?

 With kind regards,
 Yury
Dear Yura, D is a perfect fit. For performance reasons, when releasing your binary make sure to use one of these compilers: GDC - GCC D Compiler (comparable performance to gcc) LDC - LLVM D Compiler (comparable performance to clang) Usually D is on par with C, if performance is less than expected it probably means you forgot one optimization switch, especially singleobj is non-intuitive for ldc2, but can have a dramatic impact. ex ldc2 -O5 -inline -release -singleobj -boundscheck=off Best Regards, Daniel N
Aug 02 2015
prev sibling next sibling parent "bachmeier" <no spam.net> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 I just wander how D would be suitable for my purpose? Please, 
 correct me if I am wrong, but in D the need of pointers is 
 minimal, there is a garbage collector, the arrays can be 
 dynamically allocated, the arrays can be sliced, ~=, etc which 
 makes it similar to python at some extent. I tried to write a 
 little code in D and it was very much intuitive and similar to 
 what I did both in Python and C.
If you can do it in C, you can do it in D. Full stop. You get a lot of nice additional features with D, and unlike C++, there are not a thousand ways to shoot yourself in the foot with a hundred line program. You can easily call C libraries from D so you lose nothing wrt to legacy C code. You can use pyd to interoperate with Python, so you can move as quickly or as slowly as desired from Python to D. You can even call D from C, so if D doesn't work out, you don't lose the code you've written. I primarily use R but have been moving a lot of code into D for speed and the nice language features for two years now, and I have no regrets. When I started with D, I read lots of comments about the compiler being buggy, but have yet to encounter a compiler bug. Thankfully that myth seems to be dying.
Aug 02 2015
prev sibling next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 3/08/2015 4:25 a.m., Yura wrote:
 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, and it is
 sort of difficult for me to pick up the right language this project to
 be written. The project is going to deal with the generation of the
 molecular structures and will resemble to some extent some
 bio-informatic stuff. Personally I code in two languages - Python, and a
 little bit in C (just started to learn this language).

 While it is easy to code in Python there are two things I do not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code which can
 be compiled and distributed as binaries (at least at the beginning).

 When it comes to C, it is very difficult to code (I am a chemist rather
 than computer scientist). The pointers, memory allocation, absence of
 the truly dynamically allocated arrays, etc, etc make the coding very
 long. C is too low level I believe.

 I just wander how D would be suitable for my purpose? Please, correct me
 if I am wrong, but in D the need of pointers is minimal, there is a
 garbage collector, the arrays can be dynamically allocated, the arrays
 can be sliced, ~=, etc which makes it similar to python at some extent.
 I tried to write a little code in D and it was very much intuitive and
 similar to what I did both in Python and C.

 Any hints/thoughts/advises?

 With kind regards,
 Yury
Everyone else seems to be focusing on the technical aspects of why choose/not D. To put it simply, just have a go! Write a small prototype. - Did you enjoy it? - Did it reflect what you were thinking well? - Can others understand it? If you need help, feel free to jump on and post on D.learn. If you need more interactive help, come on IRC. We have a channel on FreeNode and even OFTC.
Aug 02 2015
prev sibling next sibling parent reply "yawniek" <dlang srtnwz.com> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:

 While it is easy to code in Python there are two things I do 
 not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code 
 which can be compiled and distributed as binaries (at least at 
 the beginning).
you can use the best of both worlds with pyd: https://github.com/ariovistus/pyd - write python Modules in D and/or - make your D code scriptable with python
Aug 02 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Monday, 3 August 2015 at 06:16:57 UTC, yawniek wrote:
 On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:

 While it is easy to code in Python there are two things I do 
 not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code 
 which can be compiled and distributed as binaries (at least at 
 the beginning).
you can use the best of both worlds with pyd: https://github.com/ariovistus/pyd - write python Modules in D and/or - make your D code scriptable with python
Also, note that you can write D in the ipython/jupyter notebook and have it interoperate with D libraries from code.dlang.org and with python. It's at an early stage, but so far I have found it to work well. https://github.com/DlangScience/PydMagic
Aug 03 2015
prev sibling next sibling parent reply "FreeSlave" <freeslave93 gmail.com> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, 
 and it is sort of difficult for me to pick up the right 
 language this project to be written. The project is going to 
 deal with the generation of the molecular structures and will 
 resemble to some extent some bio-informatic stuff. Personally I 
 code in two languages - Python, and a little bit in C (just 
 started to learn this language).

 [...]
Did you try PyPy implementation of python? It's claimed to be faster than CPython. If it's still not enough for you, then try D for sure. Write sample program that do calculations on real data, use gdc or ldc to get the optimized code and see if you're happy with results.
Aug 03 2015
parent "jmh530" <john.michael.hall gmail.com> writes:
On Monday, 3 August 2015 at 14:25:21 UTC, FreeSlave wrote:
 On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, 
 and it is sort of difficult for me to pick up the right 
 language this project to be written. The project is going to 
 deal with the generation of the molecular structures and will 
 resemble to some extent some bio-informatic stuff. Personally 
 I code in two languages - Python, and a little bit in C (just 
 started to learn this language).

 [...]
Did you try PyPy implementation of python? It's claimed to be faster than CPython. If it's still not enough for you, then try D for sure. Write sample program that do calculations on real data, use gdc or ldc to get the optimized code and see if you're happy with results.
Last time I checked there's lots of stuff that you can't use with pypy.
Aug 03 2015
prev sibling next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:
 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, 
 and it is sort of difficult for me to pick up the right 
 language this project to be written. The project is going to 
 deal with the generation of the molecular structures and will 
 resemble to some extent some bio-informatic stuff. Personally I 
 code in two languages - Python, and a little bit in C (just 
 started to learn this language).

 While it is easy to code in Python there are two things I do 
 not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code 
 which can be compiled and distributed as binaries (at least at 
 the beginning).

 When it comes to C, it is very difficult to code (I am a 
 chemist rather than computer scientist). The pointers, memory 
 allocation, absence of the truly dynamically allocated arrays, 
 etc, etc make the coding very long. C is too low level I 
 believe.

 I just wander how D would be suitable for my purpose? Please, 
 correct me if I am wrong, but in D the need of pointers is 
 minimal, there is a garbage collector, the arrays can be 
 dynamically allocated, the arrays can be sliced, ~=, etc which 
 makes it similar to python at some extent. I tried to write a 
 little code in D and it was very much intuitive and similar to 
 what I did both in Python and C.

 Any hints/thoughts/advises?

 With kind regards,
 Yury
I agree with bachmeier. You cannot go wrong. You mentioned nested loops. D allows you to concatenate (or "pipe") loops. So instead of foreach { foreach { foreach { } } } you have something like int[] numbers = [-2, 1, 6, -3, 10]; foreach (ref n; numbers .map!(a => a * 5) // multiply each value by 5 .filter!(a => a > 0)) // filter values that are 0 or less { // Do something } or just write auto result = numbers.map!(a => a * 5).filter!(a => a > 0); // ==> result = [5, 30, 50] You'd probably want to have a look at: http://dlang.org/phobos/std_algorithm.html and ranges (a very important concept in D): http://ddili.org/ders/d.en/ranges.html http://wiki.dlang.org/Component_programming_with_ranges Excessive use of nested loops is not necessary in D nor is it very common. This makes the code easier to maintain and less buggy in the end.
Aug 04 2015
next sibling parent reply maarten van damme via Digitalmars-d <digitalmars-d puremagic.com> writes:
I'm not a programmer myself and used D for a project in computational
electromagnetics. While I had to implement numerical integration and a bit
of linear algebra which was annoying (would be really useful in phobos), it
was a joy to work with and the resulting program was incredibly fast.
Most others used matlab and the difference in speed was more than a factor
100. Not only that, prototyping went quicker in D.

I've also written a simulation of the dual slit experiment which I'll drop
somewhere on github once the code is presentable.

So, if you don't mind having to implement a few algorithms that are already
available in numpy, D will be pleasant and fast.

2015-08-04 11:48 GMT+02:00 Chris via Digitalmars-d <
digitalmars-d puremagic.com>:

 On Sunday, 2 August 2015 at 16:25:18 UTC, Yura wrote:

 Dear D coders/developers,

 I am just thinking on one project in computational chemistry, and it is
 sort of difficult for me to pick up the right language this project to be
 written. The project is going to deal with the generation of the molecular
 structures and will resemble to some extent some bio-informatic stuff.
 Personally I code in two languages - Python, and a little bit in C (just
 started to learn this language).

 While it is easy to code in Python there are two things I do not like:

 1) Python is slow for nested loops (much slower comparing to C)
 2) Python is not compiled. However, I want to work with a code which can
 be compiled and distributed as binaries (at least at the beginning).

 When it comes to C, it is very difficult to code (I am a chemist rather
 than computer scientist). The pointers, memory allocation, absence of the
 truly dynamically allocated arrays, etc, etc make the coding very long. C
 is too low level I believe.

 I just wander how D would be suitable for my purpose? Please, correct me
 if I am wrong, but in D the need of pointers is minimal, there is a garbage
 collector, the arrays can be dynamically allocated, the arrays can be
 sliced, ~=, etc which makes it similar to python at some extent. I tried to
 write a little code in D and it was very much intuitive and similar to what
 I did both in Python and C.

 Any hints/thoughts/advises?

 With kind regards,
 Yury
I agree with bachmeier. You cannot go wrong. You mentioned nested loops. D allows you to concatenate (or "pipe") loops. So instead of foreach { foreach { foreach { } } } you have something like int[] numbers = [-2, 1, 6, -3, 10]; foreach (ref n; numbers .map!(a => a * 5) // multiply each value by 5 .filter!(a => a > 0)) // filter values that are 0 or less { // Do something } or just write auto result = numbers.map!(a => a * 5).filter!(a => a > 0); // ==> result = [5, 30, 50] You'd probably want to have a look at: http://dlang.org/phobos/std_algorithm.html and ranges (a very important concept in D): http://ddili.org/ders/d.en/ranges.html http://wiki.dlang.org/Component_programming_with_ranges Excessive use of nested loops is not necessary in D nor is it very common. This makes the code easier to maintain and less buggy in the end.
Aug 04 2015
parent reply "Chris" <wendlec tcd.ie> writes:
On Tuesday, 4 August 2015 at 13:25:22 UTC, maarten van damme 
wrote:
 I'm not a programmer myself and used D for a project in 
 computational
 electromagnetics. While I had to implement numerical 
 integration and a bit
 of linear algebra which was annoying (would be really useful in 
 phobos), it
 was a joy to work with and the resulting program was incredibly 
 fast.
 Most others used matlab and the difference in speed was more 
 than a factor
 100. Not only that, prototyping went quicker in D.
Good that you point that out. Most people I know claim that it's easier to develop/prototype in with Matlab. Apart from execution speed and fast prototyping, Matlab is proprietary. This alone is a deal breaker.
 I've also written a simulation of the dual slit experiment 
 which I'll drop somewhere on github once the code is 
 presentable.

 So, if you don't mind having to implement a few algorithms that 
 are already available in numpy, D will be pleasant and fast.
Aug 04 2015
next sibling parent reply "bachmeier" <no spam.com> writes:
On Tuesday, 4 August 2015 at 13:42:15 UTC, Chris wrote:

 Good that you point that out. Most people I know claim that 
 it's easier to develop/prototype in with Matlab. Apart from 
 execution speed and fast prototyping, Matlab is proprietary. 
 This alone is a deal breaker.
I can only imagine it being faster to prototype in Matlab if there are additional libraries available. D's just a way better language - Matlab was designed as a replacement for FORTRAN 77 - and static typing means the compiler catches a lot of bugs that you don't want to think about while prototyping.
Aug 04 2015
next sibling parent "Chris" <wendlec tcd.ie> writes:
On Tuesday, 4 August 2015 at 13:58:02 UTC, bachmeier wrote:
 On Tuesday, 4 August 2015 at 13:42:15 UTC, Chris wrote:

 Good that you point that out. Most people I know claim that 
 it's easier to develop/prototype in with Matlab. Apart from 
 execution speed and fast prototyping, Matlab is proprietary. 
 This alone is a deal breaker.
I can only imagine it being faster to prototype in Matlab if there are additional libraries available. D's just a way better language - Matlab was designed as a replacement for FORTRAN 77 - and static typing means the compiler catches a lot of bugs that you don't want to think about while prototyping.
I think it's a myth that Matlab is better for prototyping. A lot of people don't want to learn D (or any other language) and say that prototyping in Matlab is faster (which it is, of course, if you don't know D at all). Have you ever seen any of those quickly written Matlab programs? Oh deary me!
Aug 04 2015
prev sibling parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Tuesday, 4 August 2015 at 13:58:02 UTC, bachmeier wrote:
 I can only imagine it being faster to prototype in Matlab if 
 there are additional libraries available. D's just a way better 
 language - Matlab was designed as a replacement for FORTRAN 77 
 - and static typing means the compiler catches a lot of bugs 
 that you don't want to think about while prototyping.
It depends on what you need to do. I typically use Matlab/R/Python for statistics, matrix math, and optimization. Because all the libraries are either part of the language or readily available, I can do what I need to easily...much easier than in D. Wrt static typing, I don't think the issue is about catching bugs in Matlab. I just haven't had much an issue with mixing up types in Matlab. I can see the arguments about static typing and ahead of time compilation for performance, but that's less important for prototyping. One good thing about dynamic typing is that it allows for easier manipulation of matrices.
Aug 04 2015
parent reply "bachmeier" <no spam.com> writes:
On Tuesday, 4 August 2015 at 19:40:30 UTC, jmh530 wrote:

 Wrt static typing, I don't think the issue is about catching 
 bugs in Matlab. I just haven't had much an issue with mixing up 
 types in Matlab. I can see the arguments about static typing 
 and ahead of time compilation for performance, but that's less 
 important for prototyping. One good thing about dynamic typing 
 is that it allows for easier manipulation of matrices.
For me, the big win in prototyping comes from the ability to easily make significant changes to my program. With D, I am able to give the compiler a bunch of information about the types of data I'm working with, and the compiler handles all the details for me, better than a good research assistant. If I change the function to take different input or return different output, I don't have to look through 500 lines of code to make sure I'm keeping everything consistent. A couple of things that might make D more pleasant for me are: - I do a lot of simulation-related things, where the inputs and outputs can change a lot as I figure out how I want to do things, and - I use R. R was invented down the hall from C, and AFAICT, the C and R guys were believers that silent casting and undefined behavior are the foundation of a good programming language.
Aug 04 2015
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 4 August 2015 at 20:37:18 UTC, bachmeier wrote:
 A couple of things that might make D more pleasant for me are:
 - I do a lot of simulation-related things, where the inputs and 
 outputs can change a lot as I figure out how I want to do 
 things, and
 - I use R. R was invented down the hall from C, and AFAICT, the 
 C and R guys were believers that silent casting and undefined 
 behavior are the foundation of a good programming language.
We should talk. I work in a research group with a lot of simulation work going on, your perspectives on D for simulations could be useful to me. If you're interested, drop me an email: john dot loughran dot colvin at gmail dot com
Aug 04 2015
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 4 August 2015 at 13:42:15 UTC, Chris wrote:
 On Tuesday, 4 August 2015 at 13:25:22 UTC, maarten van damme 
 wrote:
 I'm not a programmer myself and used D for a project in 
 computational
 electromagnetics. While I had to implement numerical 
 integration and a bit
 of linear algebra which was annoying (would be really useful 
 in phobos), it
 was a joy to work with and the resulting program was 
 incredibly fast.
 Most others used matlab and the difference in speed was more 
 than a factor
 100. Not only that, prototyping went quicker in D.
Good that you point that out. Most people I know claim that it's easier to develop/prototype in with Matlab. Apart from execution speed and fast prototyping, Matlab is proprietary. This alone is a deal breaker.
Matlab is a linear algebra DSL with a general purpose mathematical programming language caked on to it, held in place purely by money and consumer lock-in.
Aug 04 2015
prev sibling parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Tuesday, 4 August 2015 at 09:48:07 UTC, Chris wrote:
 I agree with bachmeier. You cannot go wrong. You mentioned 
 nested loops. D allows you to concatenate (or "pipe") loops. So 
 instead of

 foreach
 {
   foreach
   {
     foreach
     {
     }
   }
 }

 you have something like

 int[] numbers = [-2, 1, 6, -3, 10];
 foreach (ref n; numbers
   .map!(a => a * 5)  // multiply each value by 5
   .filter!(a => a > 0))  // filter values that are 0 or less
 {
   //  Do something
 }
I don't think I had seen an example like this before (though it is obvious in retrospect). Is there any advantage in terms of performance?
Aug 04 2015
next sibling parent "bachmeier" <no spam.com> writes:
On Tuesday, 4 August 2015 at 18:56:20 UTC, jmh530 wrote:

 I don't think I had seen an example like this before (though it 
 is obvious in retrospect). Is there any advantage in terms of 
 performance?
The big win is in terms of being able to write complicated, correct code easily. However, there was a recent thread on the topic: http://forum.dlang.org/post/mailman.4829.1434623275.7663.digitalmars-d puremagic.com Walter said, "I expect that at the moment, range+algorithms code will likely be somewhat slower than old fashioned loops. This is because code generators have been tuned for decades to do a great job with loops. There's no intrinsic reason why ranges must do worse, so I expect they'll achieve parity. Ranges can move ahead because they can reduce the algorithmic complexity, whereas user written loops tend to be suboptimal."
Aug 04 2015
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 4 August 2015 at 18:56:20 UTC, jmh530 wrote:
 On Tuesday, 4 August 2015 at 09:48:07 UTC, Chris wrote:
 I agree with bachmeier. You cannot go wrong. You mentioned 
 nested loops. D allows you to concatenate (or "pipe") loops. 
 So instead of

 foreach
 {
   foreach
   {
     foreach
     {
     }
   }
 }

 you have something like

 int[] numbers = [-2, 1, 6, -3, 10];
 foreach (ref n; numbers
   .map!(a => a * 5)  // multiply each value by 5
   .filter!(a => a > 0))  // filter values that are 0 or less
 {
   //  Do something
 }
I don't think I had seen an example like this before (though it is obvious in retrospect). Is there any advantage in terms of performance?
ldc and gdc can often achieve parity with explicit loops.
Aug 04 2015
prev sibling next sibling parent reply "Yura" <min_yura mail.ru> writes:
Dear all,

Thank you for your replies. I am now really convinced that D is a 
decent choice for my project (also I am really happy to see that 
the forum is really active and apparently many of you use D for 
your scientific projects). I am just looking forward to writing 
the code. I had a very quick look at lecture given at DConf 2015 
- good talk, and I believe D has a big promise in Science. 
Perhaps the only problem being is the mathematical library, like 
numpy.

Until now I usually wrote the prototype algorithms in Python and 
then translated the code onto C for speed. It would be just dream 
to use only one language. The dominant languages in science now 
for production codes are Fortran or C/C++, may be D could become 
another option?

With kind regards,
Yury
Aug 05 2015
next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:
 Dear all,

 Thank you for your replies. I am now really convinced that D is 
 a decent choice for my project (also I am really happy to see 
 that the forum is really active and apparently many of you use 
 D for your scientific projects). I am just looking forward to 
 writing the code. I had a very quick look at lecture given at 
 DConf 2015 - good talk, and I believe D has a big promise in 
 Science. Perhaps the only problem being is the mathematical 
 library, like numpy.

 Until now I usually wrote the prototype algorithms in Python 
 and then translated the code onto C for speed. It would be just 
 dream to use only one language. The dominant languages in 
 science now for production codes are Fortran or C/C++, may be D 
 could become another option?

 With kind regards,
 Yury
I think NumPy was written in C(++) and is imported as a Python module. So if you can get your hands on the original underlying C(++) library, you can call NumPy directly from D, can't you? In case you do this, let me know how you fared with it. NumPy is usually the deadbeat argument when people have to choose between Python or other languages.
Aug 05 2015
parent "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Wednesday, 5 August 2015 at 18:20:20 UTC, Chris wrote:
 On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:
 Dear all,

 Thank you for your replies. I am now really convinced that D 
 is a decent choice for my project (also I am really happy to 
 see that the forum is really active and apparently many of you 
 use D for your scientific projects). I am just looking forward 
 to writing the code. I had a very quick look at lecture given 
 at DConf 2015 - good talk, and I believe D has a big promise 
 in Science. Perhaps the only problem being is the mathematical 
 library, like numpy.

 Until now I usually wrote the prototype algorithms in Python 
 and then translated the code onto C for speed. It would be 
 just dream to use only one language. The dominant languages in 
 science now for production codes are Fortran or C/C++, may be 
 D could become another option?

 With kind regards,
 Yury
I think NumPy was written in C(++) and is imported as a Python module. So if you can get your hands on the original underlying C(++) library, you can call NumPy directly from D, can't you? In case you do this, let me know how you fared with it. NumPy is usually the deadbeat argument when people have to choose between Python or other languages.
Isn't the useful bit of NumPy more like a set of wrappers around other C/C++/Fortran libraries? And the whole point of NumPy is that you can call it easily from python, which doesn't make for a nice calling convention from D (although you can call from PyD if you want). Anyway, I agree that it's a big project, but not an infeasible one to implement similar functionality in D.
Aug 05 2015
prev sibling parent reply "bachmeier" <no spam.com> writes:
On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:

 The dominant languages in science now for production codes are 
 Fortran or C/C++, may be D could become another option?

 With kind regards,
 Yury
Yes. The question is whether we can put together a group of developers to build the infrastructure, which is a lot more than just code. That means, in particular, good documentation and using it for our own projects. Everyone these days talks about how Python is a powerhouse scientific programming language. A decade ago it was crap. I know, because I watched it for years wishing I could use it. There were some poorly documented, domain-specific, hacked-together libraries, but Python was not for the most part a suitable choice. There is no reason we can't do the same for D. The main question is whether we are sufficiently committed to that goal. Others may consider Python, Julia, and Matlab to be good enough alternatives (I don't, but not everyone necessarily agrees with me).
Aug 05 2015
next sibling parent reply "Laeeth Isharc" <laeethnospam nospamlaeeth.com> writes:
On Wednesday, 5 August 2015 at 18:49:21 UTC, bachmeier wrote:
 On Wednesday, 5 August 2015 at 17:47:49 UTC, Yura wrote:

 The dominant languages in science now for production codes are 
 Fortran or C/C++, may be D could become another option?

 With kind regards,
 Yury
Yes. The question is whether we can put together a group of developers to build the infrastructure, which is a lot more than just code. That means, in particular, good documentation and using it for our own projects. Everyone these days talks about how Python is a powerhouse scientific programming language. A decade ago it was crap. I know, because I watched it for years wishing I could use it. There were some poorly documented, domain-specific, hacked-together libraries, but Python was not for the most part a suitable choice. There is no reason we can't do the same for D. The main question is whether we are sufficiently committed to that goal. Others may consider Python, Julia, and Matlab to be good enough alternatives (I don't, but not everyone necessarily agrees with me).
Yes - I fully agree with everything you say here. John Colvin has done great work in putting together a scientific computing portal for D, and writing a wrapper so you can write D within an ipython notebook (and call it transparently from python code). Great for rapid iteration and exploration of data, and it means you don't need to write the whole stack from scratch in D. One place I started in a small way was implementing a limited subset of dataframe functionality. There's not much to it, but it's something very handy to be able to slurp in data from csv or hdf5 that might not fit in a more rigid format and do spreadsheet type stuff with it. So instead of an array of variants, you define a type for the column (every entry in the column is the same type, but different columns may be of different types). In addition each column has a name and you can add and remove columns easily. I've implemented a just about usable version of that, but it's not pretty, rigorous, or especially efficient. The next stage is creating indexing by columns a la pandas. Dataframes aren't intellectually very exciting, but they are very useful for iterative data exploration and quick prototyping since all of that starts with getting the data in from somewhere in a standard format. The problem I have is that I have an ambitious project and too few resources for now. So I can't at this stage put much time into making anything someone else could use. But maybe we could work together on parts of this, if that would be interesting. I am speaking to Vlad Lefenfeld about this a bit too. On the pure numerical stuff, speak to John Colvin. If you want you can email me laeeth laeeth dot com. Laeeth.
Aug 05 2015
parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc wrote:
 Dataframes aren't intellectually very exciting, but they are 
 very useful for iterative data exploration and quick 
 prototyping since all of that starts with getting the data in 
 from somewhere in a standard format.
May not be intellectually exciting, but look at how popular pandas is for python.
Aug 05 2015
parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 6 August 2015 at 02:38:15 UTC, jmh530 wrote:
 On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc 
 wrote:
 Dataframes aren't intellectually very exciting, but they are 
 very useful for iterative data exploration and quick 
 prototyping since all of that starts with getting the data in 
 from somewhere in a standard format.
May not be intellectually exciting, but look at how popular pandas is for python.
Yes - exactly my point. I figured I needed to acknowledged this however, since smart people often work on what's stimulating rather than what's most useful given the choice. Having dataframes will open up more possibilities. Via PyD and some extra glue, it shouldn't be hard to make D dataframes easily convertible to pandas dataframes and back. Thats perfect for the Ipython notebook because you then don't need to convert all your code at once, and you still have access to Python notebooks.
Aug 05 2015
parent "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Thursday, 6 August 2015 at 06:56:11 UTC, Laeeth Isharc wrote:
 On Thursday, 6 August 2015 at 02:38:15 UTC, jmh530 wrote:
 On Wednesday, 5 August 2015 at 23:37:37 UTC, Laeeth Isharc 
 wrote:
 Dataframes aren't intellectually very exciting, but they are 
 very useful for iterative data exploration and quick 
 prototyping since all of that starts with getting the data in 
 from somewhere in a standard format.
May not be intellectually exciting, but look at how popular pandas is for python.
Yes - exactly my point. I figured I needed to acknowledged this however, since smart people often work on what's stimulating rather than what's most useful given the choice. Having dataframes will open up more possibilities. Via PyD and some extra glue, it shouldn't be hard to make D dataframes easily convertible to pandas dataframes and back. Thats perfect for the Ipython notebook because you then don't need to convert all your code at once, and you still have access to Python notebooks.
I mean access to Python libraries.
Aug 05 2015
prev sibling parent reply "Gerald Jansen" <gjansen ownmail.net> writes:
On Wednesday, 5 August 2015 at 18:49:21 UTC, bachmeier wrote:

 Yes. The question is whether we can put together a group of 
 developers to build the infrastructure, which is a lot more 
 than just code. That means, in particular, good documentation 
 and using it for our own projects.
Right on! I would be willing to help with documentation if there were a concerted effort in this direction. There have been a number of failed individual efforts over the years. So how can a group effort be promoted? Is the Dscience github project an adequate platform? How can other people get involved? Is a dedicated discussion group needed? Can we develop a plan of some sort rather than just a scatter of individual efforts?
 Everyone these days talks about how Python is a powerhouse 
 scientific programming language. A decade ago it was crap. I 
 know, because I watched it for years wishing I could use it. 
 There were some poorly documented, domain-specific, 
 hacked-together libraries, but Python was not for the most part 
 a suitable choice.
I started with Python in the years of the Numeric+numarray->NumPy transition. It was messy. Personally I think a unified library like NumPy, to underpin other more specialized libraries, is of paramount importance to any success of D in science. Ideally there would be a NumPy/ndarray usage-compatible module in D. That would make D much more attractive to potential Python converts and lower the entry barrier considerably.
Aug 06 2015
next sibling parent "ixid" <nuaccount gmail.com> writes:
On Thursday, 6 August 2015 at 08:11:49 UTC, Gerald Jansen wrote:
 Is the Dscience github project an adequate platform? How can 
 other people get involved? Is a dedicated discussion group 
 needed? Can we develop a plan of some sort rather than just a 
 scatter of individual efforts?
If a group do go ahead with this it would be good to add a discussion group on this forum for it to maximize visibility and involvement.
Aug 06 2015
prev sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 6 August 2015 at 08:11:49 UTC, Gerald Jansen wrote:
 On Wednesday, 5 August 2015 at 18:49:21 UTC, bachmeier wrote:

 Yes. The question is whether we can put together a group of 
 developers to build the infrastructure, which is a lot more 
 than just code. That means, in particular, good documentation 
 and using it for our own projects.
Right on! I would be willing to help with documentation if there were a concerted effort in this direction. There have been a number of failed individual efforts over the years. So how can a group effort be promoted? Is the Dscience github project an adequate platform? How can other people get involved? Is a dedicated discussion group needed? Can we develop a plan of some sort rather than just a scatter of individual efforts?
Yes, come join https://github.com/DlangScience. Ilya and I have a plan of sorts, but it needs formally writing down. For now we have been using a private Gitter room for discussion, which has been OK for now but likely won't scale. Anyone serious about getting involved, drop a message here: https://gitter.im/DlangScience/public and we can start to build a picture of what expertise we have and what we're missing. A proper public forum could be great, but I don't personally have time to set up something like that at the moment.
Aug 06 2015
prev sibling parent reply "Yura" <min_yura mail.ru> writes:
Good afternoon, gentlemen,

just want to describe my very limited experience. I have 
re-written about half of my Python code into D. I got it faster 
by 6 times. This is a good news.

However, I was amazed by performance of D vs Python for following 
simple nested loops (see below). D was faster by 2 order of 
magnitude!

Bearing in mind that Python is really used in computational 
chemistry/bioinformatics, I am sure D can be a good option in 
this field. In the modern strategy for the computational software 
python is used as a glue language and the number crunching parts 
are usually written in Fortran or C/C++. Apparently, with D one 
language can be used to write the entire code. Please, also look 
at this article:

http://www.worldcomp-proceedings.com/proc/p2012/PDP3426.pdf

Also, I wander about the results of this internship:

http://forum.dlang.org/post/laha9j$pc$1 digitalmars.com

With kind regards,
Yury


Python:

#!/usr/bin/python
import sys, string, os, glob, random
from math import *

a = 0

l = 1000

for i in range(l):
         for j in range(l):
                 for m in range(l):
                         a = a +i*i*0.7+j*j*0.8+m*m*0.9

print a

D:

import std.stdio;
// command line argument
import std.getopt;
import std.string;
import std.array;
import std.conv;
import std.math;

// main program starts here
void main(string[] args) {


int l = 1000;
double a = 0;
for (auto i=0;i<l;i++){
         for (auto j=0;j<l;j++) {
                 for (auto m=0;m<l;m++) {
                         a = a + i*i*0.7+j*j*0.8+m*m*0.9;
                         }

         }
}
writeln(a);
}
Aug 16 2015
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 17/08/2015 1:11 a.m., Yura wrote:
 Good afternoon, gentlemen,

 just want to describe my very limited experience. I have re-written
 about half of my Python code into D. I got it faster by 6 times. This is
 a good news.

 However, I was amazed by performance of D vs Python for following simple
 nested loops (see below). D was faster by 2 order of magnitude!

 Bearing in mind that Python is really used in computational
 chemistry/bioinformatics, I am sure D can be a good option in this
 field. In the modern strategy for the computational software python is
 used as a glue language and the number crunching parts are usually
 written in Fortran or C/C++. Apparently, with D one language can be used
 to write the entire code. Please, also look at this article:

 http://www.worldcomp-proceedings.com/proc/p2012/PDP3426.pdf

 Also, I wander about the results of this internship:

 http://forum.dlang.org/post/laha9j$pc$1 digitalmars.com

 With kind regards,
 Yury


 Python:

 #!/usr/bin/python
 import sys, string, os, glob, random
 from math import *

 a = 0

 l = 1000

 for i in range(l):
          for j in range(l):
                  for m in range(l):
                          a = a +i*i*0.7+j*j*0.8+m*m*0.9

 print a

 D:

 import std.stdio;
 // command line argument
 import std.getopt;
 import std.string;
 import std.array;
 import std.conv;
 import std.math;

 // main program starts here
 void main(string[] args) {


 int l = 1000;
 double a = 0;
 for (auto i=0;i<l;i++){
          for (auto j=0;j<l;j++) {
                  for (auto m=0;m<l;m++) {
                          a = a + i*i*0.7+j*j*0.8+m*m*0.9;
                          }

          }
 }
 writeln(a);
 }
Any chance for when you get the time/content, to create a research paper using your use case? It would be amazing publicity and even more so to get it published! Otherwise, we could always do with another user story :)
Aug 16 2015
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Sunday, 16 August 2015 at 13:11:12 UTC, Yura wrote:
 Good afternoon, gentlemen,

 just want to describe my very limited experience. I have 
 re-written about half of my Python code into D. I got it faster 
 by 6 times. This is a good news.

 However, I was amazed by performance of D vs Python for 
 following simple nested loops (see below). D was faster by 2 
 order of magnitude!

 Bearing in mind that Python is really used in computational 
 chemistry/bioinformatics, I am sure D can be a good option in 
 this field. In the modern strategy for the computational 
 software python is used as a glue language and the number 
 crunching parts are usually written in Fortran or C/C++. 
 Apparently, with D one language can be used to write the entire 
 code. Please, also look at this article:

 http://www.worldcomp-proceedings.com/proc/p2012/PDP3426.pdf

 Also, I wander about the results of this internship:

 http://forum.dlang.org/post/laha9j$pc$1 digitalmars.com

 With kind regards,
 Yury


 Python:

 #!/usr/bin/python
 import sys, string, os, glob, random
 from math import *

 a = 0

 l = 1000

 for i in range(l):
         for j in range(l):
                 for m in range(l):
                         a = a +i*i*0.7+j*j*0.8+m*m*0.9

 print a

 D:

 import std.stdio;
 // command line argument
 import std.getopt;
 import std.string;
 import std.array;
 import std.conv;
 import std.math;

 // main program starts here
 void main(string[] args) {


 int l = 1000;
 double a = 0;
 for (auto i=0;i<l;i++){
         for (auto j=0;j<l;j++) {
                 for (auto m=0;m<l;m++) {
                         a = a + i*i*0.7+j*j*0.8+m*m*0.9;
                         }

         }
 }
 writeln(a);
 }
Initially I thought the Python version is so slow because it uses `range` instead of `xrange`, but I tried them both and they both take about the same, so I guess the Python JIT(or even interpreter!) can optimize these allocations away. BTW - if you want to iterate over a range of numbers in D, you can use a foreach loop: foreach (i; 0 .. l) { foreach (j; 0 .. l) { foreach (m; 0 .. l) { a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9; } } } Or, to make it look more like the Python version, you can iterate over a range-returning function: import std.range : iota; foreach (i; iota(l)) { foreach (j; iota(l)) { foreach (m; iota(l)) { a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9; } } } There are also functions for building ranges from other ranges: import std.algorithm : cartesianProduct; import std.range : iota; foreach (i, j, m; cartesianProduct(iota(l), iota(l), iota(l))) { a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9; } Keep in mind though that using these functions, while making the code more readable(to those with some experience in D, at least), is bad for performance - for my first version I got about 5 seconds when building with DMD in debug mode, while for the last version I get 13 seconds when building with LDC in release mode.
Aug 16 2015
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 16 August 2015 at 13:59:33 UTC, Idan Arye wrote:
 Initially I thought the Python version is so slow because it 
 uses `range` instead of `xrange`, but I tried them both and 
 they both take about the same, so I guess the Python JIT(or 
 even interpreter!) can optimize these allocations away.

 BTW - if you want to iterate over a range of numbers in D, you 
 can use a foreach loop:

     foreach (i; 0 .. l) {
         foreach (j; 0 .. l) {
             foreach (m; 0 .. l) {
                 a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9;
             }

         }
     }

 Or, to make it look more like the Python version, you can 
 iterate over a range-returning function:

     import std.range : iota;
     foreach (i; iota(l)) {
         foreach (j; iota(l)) {
             foreach (m; iota(l)) {
                 a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9;
             }

         }
     }

 There are also functions for building ranges from other ranges:

     import std.algorithm : cartesianProduct;
     import std.range : iota;
     foreach (i, j, m; cartesianProduct(iota(l), iota(l), 
 iota(l))) {
         a = a + i * i * 0.7 + j * j * 0.8 + m * m * 0.9;
     }

 Keep in mind though that using these functions, while making 
 the code more readable(to those with some experience in D, at 
 least), is bad for performance - for my first version I got 
 about 5 seconds when building with DMD in debug mode, while for 
 the last version I get 13 seconds when building with LDC in 
 release mode.
There is a new implementation of cartesianProduct that makes the performance difference disappear for me with ldc and dmd. It's not in ldc's phobos yet so I had to copy it manually, but hopefully it will be in the next release.
Aug 17 2015
prev sibling parent "jmh530" <john.michael.hall gmail.com> writes:
On Sunday, 16 August 2015 at 13:11:12 UTC, Yura wrote:
 Python:

 #!/usr/bin/python
 import sys, string, os, glob, random
 from math import *

 a = 0

 l = 1000

 for i in range(l):
         for j in range(l):
                 for m in range(l):
                         a = a +i*i*0.7+j*j*0.8+m*m*0.9

 print a
While starting over with D might make a better framework for going forward, it might be less work than you think to speed-up your existing Python code base. Loops in Python are notoriously slow. The code you're using seems like a classic example of something that could be sped up. You could write the slow parts in C with Cython. Alternately, you could play with Numba's jit.
Aug 17 2015