www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - gRPC in D good idea for GSOC 2018?

reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
I know a project where D could benefit from gRPC in D, which is not 
among the supported languages:

   https://grpc.io/docs/

Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

Ali
Jan 15
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
Also, the deadline is in 1 week.
Jan 16
parent Seb <seb wilzba.ch> writes:
On Tuesday, 16 January 2018 at 17:58:40 UTC, jmh530 wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
Also, the deadline is in 1 week.
Yes it's a good idea for GSoC18 and we still accept ideas. Reply here if you have more ideas ;-)
Jan 18
prev sibling next sibling parent reply yawniek <yawniek srtnwz.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
Problem here is that D lacks a proper http/2 stack. unfortunately due to time constraints my libh2o binding project got stuck. Imo the real problem with D is still that phobos does not provide the important building blocks for today modern infrastrucutre: - fast and standardized eventloop compatible with Ds concurrency primitives - network protocol libs/ bindings ( http, http/2, tls, dns ) until there is effort to do this D will continue to be an ecosystem of hacks. I really like vibe.d but it comes with a lot of problems and it has the focus of providing a usable web framework and not implementing protocols in a fast and reusable way. so as long as "management" does not decide that internet protocols is something important in D nothing useful will come out. Just look how beautiful Golangs protocol implementations are and the whole ecosystem that focused effort spawned.
Jan 19
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 01/19/2018 02:14 AM, yawniek wrote:

 Just look how
 beautiful Golangs protocol implementations are and the whole ecosystem
 that focused effort spawned.
I agree and do have first-hand experience with that but I'm not experienced enough in this field to come up with a modern solution. If it were left to me, I would start looking at Go and Python solutions but I fear I wouldn't see design decisions that could have been better. Could you please summarize what needs to be done and in what order. I would be happy to take active part in this effort. Thank you, Ali
Jan 19
next sibling parent aberba <karabutaworld gmail.com> writes:
On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:
 On 01/19/2018 02:14 AM, yawniek wrote:

 Just look how
 beautiful Golangs protocol implementations are and the whole
ecosystem
 that focused effort spawned.
I have said this here before.
 I agree and do have first-hand experience with that but I'm not 
 experienced enough in this field to come up with a modern 
 solution. If it were left to me, I would start looking at Go 
 and Python solutions but I fear I wouldn't see design decisions 
 that could have been better.

 Could you please summarize what needs to be done and in what 
 order. I would be happy to take active part in this effort.

 Thank you,
 Ali
Jan 20
prev sibling parent yawniek <yawniek srtnwz.com> writes:
On Friday, 19 January 2018 at 18:34:31 UTC, Ali Çehreli wrote:
 On 01/19/2018 02:14 AM, yawniek wrote:
 Could you please summarize what needs to be done and in what 
 order. I would be happy to take active part in this effort.
well, my personal opinion this should be designed/decided on a green field and not again make the mistake to try to include too much that is just laying around. i think vibe.d's eventcore may (or may not) be too opinionated and providing too much: 1a. define an overal strategy how/where protocols should be implemented and whether to focus on speed or "elegant" implementations. 1b. define how high performance servers should ideally be implemented and how that works together with D's concurrency models. plus provide the necessary things such as streams/ranges. ideally endorse/include/create one primary eventloop implementation or interface. possibly rusts tokyo.rs could be an inspiration? https://tokio.rs/docs/getting-started/tls/ 2a. create reference implementations for a http parser or e.g. port something like picohttpparser https://github.com/h2o/picohttpparser but this should adhere to the approach defined in 1. 3. implement crypto functions. e.g. port picotls ( https://github.com/h2o/picotls ) but also provide/maintain proper openssl bindings. 4. implement http/2 grpc also, think about what happens when the world starts switching to quic, the ietf wg makes good progress from what i saw. a totally different approach/decision would be to say, that D is a glue language and one should use C libraries (libuv, openssl, picohttpparser). but then i think tooling for integrating those should be improved/standardized (e.g. package manager, binding automation).
Jan 20
prev sibling parent Joakim <dlang joakim.fea.st> writes:
On Friday, 19 January 2018 at 10:14:15 UTC, yawniek wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
Problem here is that D lacks a proper http/2 stack. unfortunately due to time constraints my libh2o binding project got stuck.
Have you seen Etienne's libhttp2? http://code.dlang.org/packages/libhttp2
Jan 20
prev sibling next sibling parent reply Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
An http/2 and gRPC solutions is probably necessary with tools like linkerd, envoy, and istio if D wants to be competitive in service mesh and distributed applications. http/2 and/or gRPC are both excellent ideas for GSoC 2018.
Jan 21
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
An http/2 and gRPC solutions is probably necessary with tools like linkerd, envoy, and istio if D wants to be competitive in service mesh and distributed applications. http/2 and/or gRPC are both excellent ideas for GSoC 2018.
I would consider them awful in a sense that there is no foundation to build them on. At best it will be a self-serving artifact poorly fitting with anything else. There is not even a standard way on handling IO as of yet. Basically do we want fiber-aware IO or blocking IO or explicit async with future/promise? Do we even have protobuf package?
Jan 21
next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
 Do we even have protobuf package?
https://github.com/msoucy/dproto it could receive some attention, there are pending issues for RPC I've been using msgpackrpc since no gRPC was available. But would be nice to have gRPC. NOTE: capnproto is a very interesting newer alternative to protobuf; https://github.com/capnproto/capnproto-dlang shows: Missing RPC part of Cap'n Proto. helping out capnproto project (esp around RPC) could be another idea. we definitely need a good way to do RPC in D, otherwise hard to integrate D with other services.
 I would consider them awful in a sense that there is no foundation to build
them on. At best it will be a self-serving artifact poorly fitting with
anything else.
But it would enable using D in places that were not previously possible (integrating with services); we could imaging providing a (semi) stable interface for grpc in D code and change implementation to use better foundations later On Sun, Jan 21, 2018 at 9:54 PM, Dmitry Olshansky via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is not among
 the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
An http/2 and gRPC solutions is probably necessary with tools like linkerd, envoy, and istio if D wants to be competitive in service mesh and distributed applications. http/2 and/or gRPC are both excellent ideas for GSoC 2018.
I would consider them awful in a sense that there is no foundation to build them on. At best it will be a self-serving artifact poorly fitting with anything else. There is not even a standard way on handling IO as of yet. Basically do we want fiber-aware IO or blocking IO or explicit async with future/promise? Do we even have protobuf package?
Jan 21
prev sibling parent Andrew Benton <Andrew.Benton675 gmail.com> writes:
On Monday, 22 January 2018 at 05:54:31 UTC, Dmitry Olshansky 
wrote:
 On Monday, 22 January 2018 at 04:40:53 UTC, Andrew Benton wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which 
 is not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
An http/2 and gRPC solutions is probably necessary with tools like linkerd, envoy, and istio if D wants to be competitive in service mesh and distributed applications. http/2 and/or gRPC are both excellent ideas for GSoC 2018.
I would consider them awful in a sense that there is no foundation to build them on. At best it will be a self-serving artifact poorly fitting with anything else.
I think it would likely be built on top of vibe or collie instead of being standalone, but I see your point.
 There is not even a standard way on handling IO as of yet.
 Basically do we want fiber-aware IO or blocking IO or explicit 
 async with future/promise?
You're right. It's definitely a problem. Given ease of use and the learning curve, I'd be in favor of fiber-aware IO or async/await types.
 Do we even have protobuf package?
We have at least two. dproto is a package that supports proto2. I've used it before and it's really nice. There's also the eponymous protobuf package, which integrates with protoc and can also do proto3. I haven't used it yet.
Jan 22
prev sibling parent reply Adrian Matoga <dlang.spam matoga.info> writes:
On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is 
 not among the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
I can share a fresh experience from mentoring a student in a similar (also RPC) project last summer. We built native D-Bus bindings in D based on libasync. The student had had no previous experience with D or RPC, and within ~2.5 months of focused work she implemented the following: 1. (de)serialization of all D-Bus data types, including the use of compile-time reflection to recursively marshall structs, arrays, and variants. Except Variant, for which we decided to make our own D-Bus-specific tagged union type, all other D-Bus types are mapped to built-in D types. 2. A class to connect to the bus via libasync sockets, read the incoming messages and dispatch them to the registered handlers, and send messages to the bus. 3. Proxy (client) and server class templates that generate all the code necessary to make the remote calls look almost like local ones (the return value/out arguments are passed to a delegate that handles the output instead of being returned synchronously). So, more or less an equivalent of vibe.d's REST interface generator, only with fewer customization points. There were still some opportunities for refactorings and optimizations, so I wouldn't consider it production ready. Also, some planned features weren't implemented, such as a more convenient way for handling signals or allowing transports other than unix sockets on libasync. On the other hand, what we have is almost 100% covered with unit tests. This not only made adding successive layers quite pleasant, as little (if any) debugging of previously written stuff was ever necessary, but also helps to keep the stuff working as we modify it. Based on my experience, I'd say that implementing gRPC may be of a right size for a GSoC project, as long as you split it into smaller components/layers, prioritize them, and focus on having at least the basic stuff usable and tested, instead of expecting it to cover all usage cases and be heavily optimized.
Jan 22
parent Timothee Cour <thelastmammoth gmail.com> writes:
for grpc, we should add to dproto (which is pretty good protobuf
library for D but lacks grpc) instead of starting from scratch, see
https://github.com/msoucy/dproto/issues/113 [your advice/opinions on
integrating with grpc?]


On Mon, Jan 22, 2018 at 12:24 PM, Adrian Matoga via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 15 January 2018 at 19:28:08 UTC, Ali Çehreli wrote:
 I know a project where D could benefit from gRPC in D, which is not among
 the supported languages:

   https://grpc.io/docs/

 Do you think gRPC support is worth adding to GSOC 2018 ideas?

   https://wiki.dlang.org/GSOC_2018_Ideas

 Ali
I can share a fresh experience from mentoring a student in a similar (also RPC) project last summer. We built native D-Bus bindings in D based on libasync. The student had had no previous experience with D or RPC, and within ~2.5 months of focused work she implemented the following: 1. (de)serialization of all D-Bus data types, including the use of compile-time reflection to recursively marshall structs, arrays, and variants. Except Variant, for which we decided to make our own D-Bus-specific tagged union type, all other D-Bus types are mapped to built-in D types. 2. A class to connect to the bus via libasync sockets, read the incoming messages and dispatch them to the registered handlers, and send messages to the bus. 3. Proxy (client) and server class templates that generate all the code necessary to make the remote calls look almost like local ones (the return value/out arguments are passed to a delegate that handles the output instead of being returned synchronously). So, more or less an equivalent of vibe.d's REST interface generator, only with fewer customization points. There were still some opportunities for refactorings and optimizations, so I wouldn't consider it production ready. Also, some planned features weren't implemented, such as a more convenient way for handling signals or allowing transports other than unix sockets on libasync. On the other hand, what we have is almost 100% covered with unit tests. This not only made adding successive layers quite pleasant, as little (if any) debugging of previously written stuff was ever necessary, but also helps to keep the stuff working as we modify it. Based on my experience, I'd say that implementing gRPC may be of a right size for a GSoC project, as long as you split it into smaller components/layers, prioritize them, and focus on having at least the basic stuff usable and tested, instead of expecting it to cover all usage cases and be heavily optimized.
Jan 26