www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Machine learning framework in D

reply Tim <tim.oliver tutanota.com> writes:
Hi all,

I'm needing to do a bit of ML soon for a project I'm involved in 
and I can't find any ML libs for D. Most of the libs available 
are for python (sci-kit learn, TensorFlow, etc.). I originally 
looked at using TensorFlow's C API but it doesn't include the 
actual ML stuff, just the tensor things. Could anybody help me 
out here?

Any recommendations? I was thinking of using RPC to talk to 
TensorFlow in python, could there be any bad sides to this? I 
haven't managed to get RPC to talk to python form D in the past

Thanks in advance
May 24
next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 25 May 2021 at 00:20:26 UTC, Tim wrote:
 Hi all,

 I'm needing to do a bit of ML soon for a project I'm involved 
 in and I can't find any ML libs for D.
 …
 Any recommendations? I was thinking of using RPC to talk to 
 TensorFlow in python, could there be any bad sides to this? I 
 haven't managed to get RPC to talk to python form D in the past
I haven’t tried it, but you might want to take a look at the Netflix Vector Flow library, which is written in D. https://github.com/Netflix/vectorflow —Jon
May 24
parent Tim <tim.oliver tutanota.com> writes:
On Tuesday, 25 May 2021 at 00:44:15 UTC, Jon Degenhardt wrote:
 On Tuesday, 25 May 2021 at 00:20:26 UTC, Tim wrote:
 Hi all,

 I'm needing to do a bit of ML soon for a project I'm involved 
 in and I can't find any ML libs for D.
 …
 Any recommendations? I was thinking of using RPC to talk to 
 TensorFlow in python, could there be any bad sides to this? I 
 haven't managed to get RPC to talk to python form D in the past
I haven’t tried it, but you might want to take a look at the Netflix Vector Flow library, which is written in D. https://github.com/Netflix/vectorflow —Jon
That looks really interesting, thanks! Not sure yet whether a NN would be what I need, but I'll definitely look into this more
May 24
prev sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 25 May 2021 at 00:20:26 UTC, Tim wrote:
 Hi all,

 I'm needing to do a bit of ML soon for a project I'm involved 
 in and I can't find any ML libs for D. Most of the libs 
 available are for python (sci-kit learn, TensorFlow, etc.). I 
 originally looked at using TensorFlow's C API but it doesn't 
 include the actual ML stuff, just the tensor things. Could 
 anybody help me out here?

 [snip]
Grain is a deep learning framework for D. https://github.com/ShigekiKarita/grain
May 24
parent reply mw <mingwu gmail.com> writes:
On Tuesday, 25 May 2021 at 01:04:19 UTC, jmh530 wrote:
 On Tuesday, 25 May 2021 at 00:20:26 UTC, Tim wrote:
 Hi all,

 I'm needing to do a bit of ML soon for a project I'm involved 
 in and I can't find any ML libs for D. Most of the libs 
 available are for python (sci-kit learn, TensorFlow, etc.). I 
 originally looked at using TensorFlow's C API but it doesn't 
 include the actual ML stuff, just the tensor things. Could 
 anybody help me out here?

 [snip]
Grain is a deep learning framework for D. https://github.com/ShigekiKarita/grain
Also, there is grain2[1], and tfd: tensorflow for D[2]. But seriously, all these ML libraries in Dlang do not have many needed features you can easily find in Python ML libraries. If we step back: when those big companies like Google (TensorFlow etc.), Facebook (PyTorch etc.), Amazon (MxNet etc.), M$ (CNTK etc.) have already invested billions of dollars in either building these libraries directly or sponsoring them, which have been used by millions of users worldwide, why we want to build & use yet another one just in D? Few D users also means you have high probability to hit bugs in some very basic stuff, e.g. bug in grain's Adam optimizer was only fixed 24 days ago [3]. D is already late for the ML game, maybe the best thing to do is concentrate the limited resource of the D communality on wrapping other mature libraries, e.g. tfd. Given the current situation, on the practical side I'm using D to pre-processing data, and feed them into Python's ML libraries. To communicate between D & Python: -- grpc-dlang cannot talk to Python [4], as you have noted, and this issue has been around for half year now. (As a workaround, I added a Python pyd client talk to D server [8]) -- apache-thrift has build failure [5], which I just send a PR [6] to fix it today. (again, I'm not sure how many real users these 2 libraries have, they are broken at some very basic stuff, not sure how other people can use them) -- pyd: this is what I'm using right now. There are some rough edges [7], but at least I can get the basic stuff working finally, i.e. D & Python can talk. Anyway, it's not easy, good luck! [1]: https://github.com/ShigekiKarita/grain2 [2]: https://github.com/ShigekiKarita/tfd [3]: https://github.com/ShigekiKarita/grain/pull/15 [4]: https://github.com/huntlabs/grpc-dlang/issues/15 [5]: https://github.com/D-Programming-Deimos/openssl/issues/63 [6]: https://github.com/apache/thrift/pull/2397 [7]: https://github.com/ariovistus/pyd/issues [8]: https://github.com/huntlabs/grpc-dlang/pull/19
May 25
parent mw <mingwu gmail.com> writes:
On Tuesday, 25 May 2021 at 23:37:24 UTC, mw wrote:
 Grain is a deep learning framework for D.

 https://github.com/ShigekiKarita/grain
Also, there is grain2[1], and tfd: tensorflow for D[2]. But seriously, all these ML libraries in Dlang do not have many needed features you can easily find in Python ML libraries. If we step back: when those big companies like Google (TensorFlow etc.), Facebook (PyTorch etc.), Amazon (MxNet etc.), M$ (CNTK etc.) have already invested billions of dollars in either building these libraries directly or sponsoring them, which have been used by millions of users worldwide, why we want to build & use yet another one just in D? Few D users also means you have high probability to hit bugs in some very basic stuff, e.g. bug in grain's Adam optimizer was only fixed 24 days ago [3]. D is already late for the ML game, maybe the best thing to do is concentrate the limited resource of the D communality on wrapping other mature libraries, e.g. tfd. Given the current situation, on the practical side I'm using D to pre-processing data, and feed them into Python's ML libraries. To communicate between D & Python: -- grpc-dlang cannot talk to Python [4], as you have noted, and this issue has been around for half year now. (As a workaround, I added a Python pyd client talk to D server [8]) -- apache-thrift has build failure [5], which I just send a PR [6] to fix it today. (again, I'm not sure how many real users these 2 libraries have, they are broken at some very basic stuff, not sure how other people can use them) -- pyd: this is what I'm using right now. There are some rough edges [7], but at least I can get the basic stuff working finally, i.e. D & Python can talk. Anyway, it's not easy, good luck!
links: 1: https://github.com/ShigekiKarita/grain2 2: https://github.com/ShigekiKarita/tfd 3: https://github.com/ShigekiKarita/grain/pull/15 4: https://github.com/huntlabs/grpc-dlang/issues/15 5: https://github.com/D-Programming-Deimos/openssl/issues/63 6: https://github.com/apache/thrift/pull/2397 7: https://github.com/ariovistus/pyd/issues 8: https://github.com/huntlabs/grpc-dlang/pull/19
May 25