www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D for microservices

reply Joakim <dlang joakim.fea.st> writes:
I just read the following two week-old comment on the ldc issue 
tracker, when someone tried to run D on Alpine linux:

"For now everything works(?) but I think the process could be 
improved.. Would be really cool to have LDC easily building 
alpine containers + static D binaries for microservice and 
tooling development. I'm pretty tired of reading Go code :)"
https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

It strikes me that microservices are a great way for new 
programming languages like D to get tried and gain some uptake, 
but that D might not be that easy to deploy to that scenario yet.

So this is a question for those deploying microservices, as I'm 
not in that field, what can the D devs do to make it as easy as 
possible to get D microservices up and running, make some Docker 
and Alpine containers with ldc/dub/vibe.d preinstalled publicly 
available?  What else, what kinds of libraries do you normally 
use?

This is a niche that D and all newer languages should target.  
How do we do it?
Oct 21
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 [...]
Rather a container with dub/ldc configured to compile with musl libc I think.
 What else, what kinds of libraries do you normally use?
Given current selection I bet most are hand-rolled except maybe vibe.d and common serialization libs.
Oct 22
prev sibling next sibling parent qznc <qznc web.de> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 This is a niche that D and all newer languages should target.  
 How do we do it?
Optimize the TechEmpower benchmark? Vibe.d looks quite weak there. https://www.techempower.com/benchmarks/
Oct 22
prev sibling next sibling parent Mengu <mengukagan gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 [...]
rock solid, easy, common-dev-proof, huge std lib. like that of golang.
Oct 22
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-22 04:48, Joakim wrote:
 I just read the following two week-old comment on the ldc issue tracker, 
 when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be improved.. 
 Would be really cool to have LDC easily building alpine containers + 
 static D binaries for microservice and tooling development. I'm pretty 
 tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new programming 
 languages like D to get tried and gain some uptake, but that D might not 
 be that easy to deploy to that scenario yet.
 
 So this is a question for those deploying microservices, as I'm not in 
 that field, what can the D devs do to make it as easy as possible to get 
 D microservices up and running, make some Docker and Alpine containers 
 with ldc/dub/vibe.d preinstalled publicly available?  What else, what 
 kinds of libraries do you normally use?
 
 This is a niche that D and all newer languages should target. How do we 
 do it?
* Support full static linking using DMD, which requires the TLS implementation to be modified * Support musl as the standard C library, I've discussed that before [1] * Database drivers for the common databases (PostgreSQL, MySQL, SQLite) compatible with vibe.d * Database driver abstraction on top of the above drivers, perhaps some lightweight ORM library * RabbitMQ library compatible with vibe.d * Serialization to/from JSON and YAML * Official Docker images with DMD and LDC wouldn't hurt * Pre-compiled DMD that works on Alpine. Fully statically linked DMD would help here That's what I can think of for now. [1] http://forum.dlang.org/post/nhem1l$1ee3$1 digitalmars.com -- /Jacob Carlborg
Oct 23
next sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Monday, 23 October 2017 at 12:08:52 UTC, Jacob Carlborg wrote:
 On 2017-10-22 04:48, Joakim wrote:
 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.
 
 So this is a question for those deploying microservices, as 
 I'm not in that field, what can the D devs do to make it as 
 easy as possible to get D microservices up and running, make 
 some Docker and Alpine containers with ldc/dub/vibe.d 
 preinstalled publicly available?  What else, what kinds of 
 libraries do you normally use?
 
 This is a niche that D and all newer languages should target. 
 How do we do it?
* Support full static linking using DMD, which requires the TLS implementation to be modified * Support musl as the standard C library, I've discussed that before [1] * Database drivers for the common databases (PostgreSQL, MySQL, SQLite) compatible with vibe.d * Database driver abstraction on top of the above drivers, perhaps some lightweight ORM library * RabbitMQ library compatible with vibe.d * Serialization to/from JSON and YAML * Official Docker images with DMD and LDC wouldn't hurt * Pre-compiled DMD that works on Alpine. Fully statically linked DMD would help here That's what I can think of for now. [1] http://forum.dlang.org/post/nhem1l$1ee3$1 digitalmars.com
Can you elaborate on how the TLS implementation needs to be changed? If someone wanted to work on rabbit bindings/wrapper, I might be open to sponsoring that. I'm not in love with Rabbit (one node uses more than 40% of memory so the node goes down, taking the cluster with it. really?), but we use it currently. I made a start on bindings for librabbitmq here: https://github.com/kaleidicassociates/rabbitmq-d Laeeth.
Oct 23
parent Jacob Carlborg <doob me.com> writes:
On 2017-10-23 14:17, Laeeth Isharc wrote:

 Can you elaborate on how the TLS implementation needs to be changed?
# echo 'void main() {}' > main.d && dmd -c main.d && gcc main.o -o main -m64 -static -L/root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64 -Xlinker -Bstatic -lphobos2 -lpthread -lm -lrt -ldl /root/.dvm/compilers/dmd-2.076.1/linux/bin/../lib64/libphobos2.a(sections_el _shared_782_420.o): In function `_D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv': src/rt/sections_elf_shared.d:(.text._D2rt19sections_elf_shared11getTLSRangeFNbNimmZAv[_D2rt19sections_elf_shared11getTLSRan eFNbNimmZAv]+0x38): undefined reference to `__tls_get_addr' collect2: error: ld returned 1 exit status I need to manually invoke GCC to link because DMD will pass "-Xlinker --export-dynamic -Xlinker -Bdynamic" and I need to add "-static".
 If someone wanted to work on rabbit bindings/wrapper, I might be open to 
 sponsoring that.  I'm not in love with Rabbit (one node uses more than 
 40% of memory so the node goes down, taking the cluster with it.  
 really?), but we use it currently.
 
 I made a start on bindings for librabbitmq here:
 https://github.com/kaleidicassociates/rabbitmq-d
Is that compatible with vibe.d? -- /Jacob Carlborg
Oct 23
prev sibling next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 23 October 2017 at 12:08:52 UTC, Jacob Carlborg wrote:
 On 2017-10-22 04:48, Joakim wrote:
 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:
 
 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550
 
 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.
 
 So this is a question for those deploying microservices, as 
 I'm not in that field, what can the D devs do to make it as 
 easy as possible to get D microservices up and running, make 
 some Docker and Alpine containers with ldc/dub/vibe.d 
 preinstalled publicly available?  What else, what kinds of 
 libraries do you normally use?
 
 This is a niche that D and all newer languages should target. 
 How do we do it?
* Support full static linking using DMD, which requires the TLS implementation to be modified * Support musl as the standard C library, I've discussed that before [1]
I can probably whip this together in a couple days, as I added the Bionic support before, but it's not something I need, so it'd just be a donation for you and others. That's why I'm trying to figure out who exactly wants this and if others want to chip in.
 * Database drivers for the common databases (PostgreSQL, MySQL, 
 SQLite) compatible with vibe.d
 * Database driver abstraction on top of the above drivers, 
 perhaps some lightweight ORM library
 * RabbitMQ library compatible with vibe.d
 * Serialization to/from JSON and YAML
Some of this stuff is on dub, but nothing that interests me or that I'd chip in with. Good to see that RabbitMQ supports binary protocols like AMQP and MQTT though, would be nuts if it were all just text like STOMP or especially JSON/YAML.
 * Official Docker images with DMD and LDC wouldn't hurt
 * Pre-compiled DMD that works on Alpine. Fully statically 
 linked DMD would help here
I'm sure someone could put these together if the above stuff worked. The question is who's interested in volunteering to help put this all together?
Oct 23
parent Jacob Carlborg <doob me.com> writes:
On 2017-10-23 17:35, Joakim wrote:

 I'm sure someone could put these together if the above stuff worked.  
 The question is who's interested in volunteering to help put this all 
 together?
Yeah, exactly. My plate if already full and all this is lower down on the priority list. -- /Jacob Carlborg
Oct 23
prev sibling parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 05:08, Jacob Carlborg wrote:
 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library
I've been looking pretty extensively at these two items recently. If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT. I had heard noises about that a few months ago. Anything happening on that front? An event loop is a key piece of a project (Async/Await) that I want to work on, and having it in DRuntime would make that project fantastically simpler. IMHO, the bulk of the time required is in getting an event loop into DRT, the rest is a LOT of relatively straightforward compiler lowering. IMHO, DRT is in significant need of an event loop system. This would allow us to simplify a large number of problems (Async/Await, GUI's, IO, etc). As near as I can tell, the problem isn't so much doing the work, but getting the required sign-off's for inclusion into DRT. Another problem that I've been made aware of is that vibe-core may not be ideal in certain situations. As this would be landed in DRT itself this would obviously need to be addressed. What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage? -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 23
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/10/2017 11:02 PM, Adam Wilson wrote:
 On 10/23/17 05:08, Jacob Carlborg wrote:
 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library
I've been looking pretty extensively at these two items recently. If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT. I had heard noises about that a few months ago. Anything happening on that front? An event loop is a key piece of a project (Async/Await) that I want to work on, and having it in DRuntime would make that project fantastically simpler. IMHO, the bulk of the time required is in getting an event loop into DRT, the rest is a LOT of relatively straightforward compiler lowering. IMHO, DRT is in significant need of an event loop system. This would allow us to simplify a large number of problems (Async/Await, GUI's, IO, etc). As near as I can tell, the problem isn't so much doing the work, but getting the required sign-off's for inclusion into DRT. Another problem that I've been made aware of is that vibe-core may not be ideal in certain situations. As this would be landed in DRT itself this would obviously need to be addressed. What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage?
*whispers* heyyy, heard about SPEW[0]? [0] https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d
Oct 23
parent Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 18:51, rikki cattermole wrote:
 On 23/10/2017 11:02 PM, Adam Wilson wrote:
 On 10/23/17 05:08, Jacob Carlborg wrote:
 * Database drivers for the common databases (PostgreSQL, MySQL, SQLite)
 compatible with vibe.d
 * Database driver abstraction on top of the above drivers, perhaps some
 lightweight ORM library
I've been looking pretty extensively at these two items recently. If the database drivers are compatible with Vibe.d AND we wish to provide a common abstraction layer for them (presumably via Phobos) then order for the abstraction layer to aware of the whether the driver is making a blocking or non-blocking call we must include Vibe.D in the abstraction layer. Ergo, we must include at least the vibe-core package in Phobos, or more preferably, DRT. I had heard noises about that a few months ago. Anything happening on that front? An event loop is a key piece of a project (Async/Await) that I want to work on, and having it in DRuntime would make that project fantastically simpler. IMHO, the bulk of the time required is in getting an event loop into DRT, the rest is a LOT of relatively straightforward compiler lowering. IMHO, DRT is in significant need of an event loop system. This would allow us to simplify a large number of problems (Async/Await, GUI's, IO, etc). As near as I can tell, the problem isn't so much doing the work, but getting the required sign-off's for inclusion into DRT. Another problem that I've been made aware of is that vibe-core may not be ideal in certain situations. As this would be landed in DRT itself this would obviously need to be addressed. What would the appetite be for working together to come up with a reasonably generic event loop for DRT that vibe and other systems could then leverage?
*whispers* heyyy, heard about SPEW[0]? [0] https://github.com/Devisualization/spew/blob/master/src/base/cf/spew/event_loop/defs.d
You've mentioned it a few times. ;-) -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 25
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-24 00:02, Adam Wilson wrote:

 I've been looking pretty extensively at these two items recently.
 
 If the database drivers are compatible with Vibe.d AND we wish to 
 provide a common abstraction layer for them (presumably via Phobos) then 
 order for the abstraction layer to aware of the whether the driver is 
 making a blocking or non-blocking call we must include Vibe.D in the 
 abstraction layer. Ergo, we must include at least the vibe-core package 
 in Phobos, or more preferably, DRT.
It can be an optional dependency. Looking at ddb [1] it's not that much code that will be different if vibe.d is used or not. It's basically only reading and writing to the socket that is different [2]. Dub provides predefined version for different dependencies, i.e. Have_vibe_d_core.
 I had heard noises about that a few months ago. Anything happening on 
 that front?
No, not as far as I know. Sönke seems really busy recently.
 What would the appetite be for working together to come up with a 
 reasonably generic event loop for DRT that vibe and other systems could 
 then leverage?
I would be really nice to database support directly in the standard library but it's not critical for me. It takes a lot of work with and massive overhead to get something into Phobos. It's also going to be really slow get in updates. I'm not sure if it's worth it. [1] https://github.com/pszturmaj/ddb [2] https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246 -- /Jacob Carlborg
Oct 23
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/23/17 23:29, Jacob Carlborg wrote:
 On 2017-10-24 00:02, Adam Wilson wrote:

 I've been looking pretty extensively at these two items recently.

 If the database drivers are compatible with Vibe.d AND we wish to
 provide a common abstraction layer for them (presumably via Phobos)
 then order for the abstraction layer to aware of the whether the
 driver is making a blocking or non-blocking call we must include
 Vibe.D in the abstraction layer. Ergo, we must include at least the
 vibe-core package in Phobos, or more preferably, DRT.
It can be an optional dependency. Looking at ddb [1] it's not that much code that will be different if vibe.d is used or not. It's basically only reading and writing to the socket that is different [2]. Dub provides predefined version for different dependencies, i.e. Have_vibe_d_core.
This of course makes the assumption that we clean-room our own protocol implementations which I am entirely against. Better to use what already exists.
 I had heard noises about that a few months ago. Anything happening on
 that front?
No, not as far as I know. Sönke seems really busy recently.
That is what I was afraid of.
 What would the appetite be for working together to come up with a
 reasonably generic event loop for DRT that vibe and other systems
 could then leverage?
I would be really nice to database support directly in the standard library but it's not critical for me. It takes a lot of work with and massive overhead to get something into Phobos. It's also going to be really slow get in updates. I'm not sure if it's worth it. [1] https://github.com/pszturmaj/ddb [2] https://github.com/pszturmaj/ddb/blob/master/source/ddb/postgres.d#L189-L246
I actually don't think the slow updates are huge problem as these DB interface libraries are pretty slow to change themselves. For example, ADO.NET didn't change significantly from it's 1.0 release until the arrival of Async/Await in .NET 4.5, which was 10 years later. The biggest addition prior to Async was TVP support and that was tiny comparatively and came in 2005. The libpq5 interface has barely changed in years. I can't speak to MySQL but IIRC it hasn't changed much either, at least not in a way that would effect the abstraction layer. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 25
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-26 00:53, Adam Wilson wrote:

 This of course makes the assumption that we clean-room our own protocol 
 implementations which I am entirely against. Better to use what already 
 exists.
I'm entirely against anything that is not compatible with vibe.d ;)
 I actually don't think the slow updates are huge problem as these DB 
 interface libraries are pretty slow to change themselves. For example, 
 ADO.NET didn't change significantly from it's 1.0 release until the 
 arrival of Async/Await in .NET 4.5, which was 10 years later. The 
 biggest addition prior to Async was TVP support and that was tiny 
 comparatively and came in 2005.
 
 The libpq5 interface has barely changed in years. I can't speak to MySQL 
 but IIRC it hasn't changed much either, at least not in a way that would 
 effect the abstraction layer.
I'm more concerned that I don't think we'll manage to implement a complete API and 100% bug free at the first try. -- /Jacob Carlborg
Oct 25
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/25/17 23:57, Jacob Carlborg wrote:
 On 2017-10-26 00:53, Adam Wilson wrote:

 This of course makes the assumption that we clean-room our own
 protocol implementations which I am entirely against. Better to use
 what already exists.
I'm entirely against anything that is not compatible with vibe.d ;)
My apologies, something rather the other direction. Instead of forcing compat with vibe.d, going to vibe.d and say: "here is our standard event-loop, it has everything you need, you'll need to use it for all the other goodness to work". I know others can make good arguments about why the vibe event-loop is insufficient, and I'll let them make them. (Something about not supporting GUI loops, paging Mr. Cattermole). If that is really the case I don't see how being entirely vibe.d compatible and meeting the universal standard requirements of Phobos is possible. There would need to be a requirements gathering phase so that the community as a whole can bring their use-cases before we dove into code.
 I actually don't think the slow updates are huge problem as these DB
 interface libraries are pretty slow to change themselves. For example,
 ADO.NET didn't change significantly from it's 1.0 release until the
 arrival of Async/Await in .NET 4.5, which was 10 years later. The
 biggest addition prior to Async was TVP support and that was tiny
 comparatively and came in 2005.

 The libpq5 interface has barely changed in years. I can't speak to
 MySQL but IIRC it hasn't changed much either, at least not in a way
 that would effect the abstraction layer.
I'm more concerned that I don't think we'll manage to implement a complete API and 100% bug free at the first try.
Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 26
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 26/10/2017 11:25 AM, Adam Wilson wrote:
 On 10/25/17 23:57, Jacob Carlborg wrote:
 On 2017-10-26 00:53, Adam Wilson wrote:

 This of course makes the assumption that we clean-room our own
 protocol implementations which I am entirely against. Better to use
 what already exists.
I'm entirely against anything that is not compatible with vibe.d ;)
My apologies, something rather the other direction. Instead of forcing compat with vibe.d, going to vibe.d and say: "here is our standard event-loop, it has everything you need, you'll need to use it for all the other goodness to work". I know others can make good arguments about why the vibe event-loop is insufficient, and I'll let them make them. (Something about not supporting GUI loops, paging Mr. Cattermole). If that is really the case I don't see how being entirely vibe.d compatible and meeting the universal standard requirements of Phobos is possible. There would need to be a requirements gathering phase so that the community as a whole can bring their use-cases before we dove into code.
The problem isn't the event loop design. Its a fairly solved problem. The way vibe.d's works is very specific to their use case (which isn't wrong if you only consider them). You can't 'hook' into it. Which makes it very undesirable for Phobos. Since it won't cover most use cases. Even if the source they are using is compatible with an external GUI event loop (which it should be for Windows from what I've read). So I wouldn't be starting with vibe.d's event loop model. Quite to the contrary, kill it. Build something that will last throughout the ages for everyone and put this problem to rest.
Oct 26
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:
 On 10/25/17 23:57, Jacob Carlborg wrote:
 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.
Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.
The bigger problem is API bugs. The review process is rigorous enough to weed out a lot of stuff, but the end result is typically an API that we _think_ looks good but usually hasn't seen much real world use (much as its design may be based on real world experience). And if it turns out that the API has problems, it can be very difficult to fix that in a way that doesn't break code. Sometimes, we're able to reasonably do something about it and sometimes not. In theory, std.experimental would mitigate that, and that's where anything that was accepted would go first, but the process for getting new modules into Phobos is very much geared towards getting an API right up front rather than getting something that's close and iterating towards where it ultimately should be. What would probably be better in general would be to be writing stuff that ends up on code.dlang.org first and gets some real world use there and then look at getting it into Phobos later rather than aiming directly for Phobos. Not only would that likely help lead towards better code being in Phobos, but we'd still get useful stuff even if it didn't make it into Phobos. - Jonathan M Davis
Oct 26
parent reply Adam Wilson <flyboynw gmail.com> writes:
On 10/26/17 17:51, Jonathan M Davis wrote:
 On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d wrote:
 On 10/25/17 23:57, Jacob Carlborg wrote:
 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.
Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.
The bigger problem is API bugs. The review process is rigorous enough to weed out a lot of stuff, but the end result is typically an API that we _think_ looks good but usually hasn't seen much real world use (much as its design may be based on real world experience). And if it turns out that the API has problems, it can be very difficult to fix that in a way that doesn't break code. Sometimes, we're able to reasonably do something about it and sometimes not. In theory, std.experimental would mitigate that, and that's where anything that was accepted would go first, but the process for getting new modules into Phobos is very much geared towards getting an API right up front rather than getting something that's close and iterating towards where it ultimately should be. What would probably be better in general would be to be writing stuff that ends up on code.dlang.org first and gets some real world use there and then look at getting it into Phobos later rather than aiming directly for Phobos. Not only would that likely help lead towards better code being in Phobos, but we'd still get useful stuff even if it didn't make it into Phobos. - Jonathan M Davis
I understand the concern, however, I can see potential mitigations. For one, steal an API concept from somewhere else. I've had reasonable success so far stealing ADO.NET and the refactoring it into something more idiomatic. Using that as a starting point gave me a pretty good understanding of what was needed and it gave me a prototype API that has been battle-tested. Has anything from code.dlang.org been pulled into Phobos yet? I'm not aware of anything. code.dlang.org is where Phobos projects go to quietly die in obscurity. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 26
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, October 26, 2017 19:19:57 Adam Wilson via Digitalmars-d wrote:
 On 10/26/17 17:51, Jonathan M Davis wrote:
 On Thursday, October 26, 2017 03:25:24 Adam Wilson via Digitalmars-d 
wrote:
 On 10/25/17 23:57, Jacob Carlborg wrote:
 I'm more concerned that I don't think we'll manage to implement a
 complete API and 100% bug free at the first try.
Depends on how one defines first try. Phobos as a pretty solid process for making sure these things are reasonably bug free. And near as I can tell, Phobos is pretty good about accepting bug fixes.
The bigger problem is API bugs. The review process is rigorous enough to weed out a lot of stuff, but the end result is typically an API that we _think_ looks good but usually hasn't seen much real world use (much as its design may be based on real world experience). And if it turns out that the API has problems, it can be very difficult to fix that in a way that doesn't break code. Sometimes, we're able to reasonably do something about it and sometimes not. In theory, std.experimental would mitigate that, and that's where anything that was accepted would go first, but the process for getting new modules into Phobos is very much geared towards getting an API right up front rather than getting something that's close and iterating towards where it ultimately should be. What would probably be better in general would be to be writing stuff that ends up on code.dlang.org first and gets some real world use there and then look at getting it into Phobos later rather than aiming directly for Phobos. Not only would that likely help lead towards better code being in Phobos, but we'd still get useful stuff even if it didn't make it into Phobos. - Jonathan M Davis
I understand the concern, however, I can see potential mitigations. For one, steal an API concept from somewhere else. I've had reasonable success so far stealing ADO.NET and the refactoring it into something more idiomatic. Using that as a starting point gave me a pretty good understanding of what was needed and it gave me a prototype API that has been battle-tested. Has anything from code.dlang.org been pulled into Phobos yet? I'm not aware of anything. code.dlang.org is where Phobos projects go to quietly die in obscurity.
Nothing has been pulled into Phobos from anywhere in a while. The last thing was probably checkedint, and before that it was std.allocator - both of which came from Andrei. ndslice was in std.experimental for a little while but Ilya removed it, because he decided that it wasn't working to have it in Phobos, because he couldn't continue to improve on it there. Beyond that, I think that the last thing was the logger, which has just languished in std.experimental. It's my understanding that it needs some more work, but basically, once we added std.experimental, nothing has made it into Phobos proper. And over the last few years, there haven't been many attempts to get anything into Phobos, so not much has even made it into std.experimental. There was the GSoC project for a new XML parser, but that project seems to have died after the student got too busy with school after GSoC, and Sonke's std.json replacement didn't finish making its way through the review process, and I think that Sonke has given up on getting it in (if I understand correctly, there was just too much disagreement over what the std.json replacement should look like). Overall, people have just stopped trying to get major stuff into Phobos. New functions get added to existing modules, but pretty much no one seems to want to go through the Phobos review process to get anything into Phobos. code.dlang.org is where folks put anything that they're doing that they want to make available, and IIRC, the only item from there that's even attempted to make it into Phobos was Sonke's JSON parser. Much as some folks continue to talk about getting some piece of functionality into Phobos, no one is trying if it's anything major. So, it's not like Phobos projects are going to code.dlang.org to die. In general, they simply aren't even being attempted, and any serious projects that do exist are on code.dlang.org. Some do sit there unfinished (e.g. std_experimental_xml), but largely, the authors just don't want to go to the effort of getting the code into Phobos. And honestly, in general, at this point, I don't think that I'd want to bother either. It's quite a lot of work to get something through the Phobos review process, and once it's in Phobos, you lose control over it. If I work on something major, I can just put it up on code.dlang.org, and anyone who wants to use it can, and I have full control over what I do with the code base without having to get approval from Andrei or anyone else as to what I do with it. So, unless we're talking about something that practically needs to be in the standard library, I doubt that I'd bother trying even if I wrote the code and was maintaining it. And most stuff really doesn't need to be in the standard library, much as it might be nice. But even if the goal is to get some of this stuff into Phobos, I still think that we're better off putting it up on code.dlang.org first and getting it battle-tested instead of pushing to get it put directly into Phobos. - Jonathan M Davis
Oct 26
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-10-26 12:25, Adam Wilson wrote:

 My apologies, something rather the other direction. Instead of forcing 
 compat with vibe.d, going to vibe.d and say: "here is our standard 
 event-loop, it has everything you need, you'll need to use it for all 
 the other goodness to work". I know others can make good arguments about 
 why the vibe event-loop is insufficient, and I'll let them make them. 
My concern is not about the event loop, it's about asynchronous IO. vibe.d needs to use asynchronous IO and I assume that's regardless what kind of event loop implementation is used. Does all the existing database drivers that you want to use support asynchronous IO? -- /Jacob Carlborg
Oct 27
parent Adam Wilson <flyboynw gmail.com> writes:
On 10/27/17 00:18, Jacob Carlborg wrote:
 On 2017-10-26 12:25, Adam Wilson wrote:

 My apologies, something rather the other direction. Instead of forcing
 compat with vibe.d, going to vibe.d and say: "here is our standard
 event-loop, it has everything you need, you'll need to use it for all
 the other goodness to work". I know others can make good arguments
 about why the vibe event-loop is insufficient, and I'll let them make
 them.
My concern is not about the event loop, it's about asynchronous IO. vibe.d needs to use asynchronous IO and I assume that's regardless what kind of event loop implementation is used. Does all the existing database drivers that you want to use support asynchronous IO?
PgSQL/MySQL/MSSQL all do, I think that covers about 90% of usage. IIRC Oracle does as well. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Oct 27
prev sibling next sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some uptake, 
 but that D might not be that easy to deploy to that scenario 
 yet.

 So this is a question for those deploying microservices, as I'm 
 not in that field, what can the D devs do to make it as easy as 
 possible to get D microservices up and running, make some 
 Docker and Alpine containers with ldc/dub/vibe.d preinstalled 
 publicly available?  What else, what kinds of libraries do you 
 normally use?

 This is a niche that D and all newer languages should target.  
 How do we do it?
We're going a bit in that direction, although it's a different thing in our kind of industry from a web company - we have fewer services and many fewer requests, but latency matters more for example. I was thinking we might use Go for some services that integrate and monitor things, but we could also use D. https://jobs.github.com/positions/8e98eac8-b504-11e7-9da8-0737a3dcef18 From the comment: "Would be really cool to have LDC easily building alpine containers + static D binaries" How can we generate a static binary ? I asked about this before, and the response was that it's a bad idea because of security vulns and so on. True if you are running on a conventional Linux host. But on the other hand, it's not that great if when the system phobos is upgraded all the D binaries break. You can write a script using rdmd or dub scripting feature, but that doesn't work for more complex programs. And on the other hand, for deployment, it's much easier to copy over one binary. (In a sense it's funny that the question was asked in the context of containers, because containers are kind of an alternative to having a single binary). When you're properly set-up of course it doesn't matter, but when you're moving towards that, a single binary is much easier. I guess I can figure out the answer to this question easily enough, but I think giving people the option might help with adoption of D for micro services. For example it's really just not that fun to make an AWS Lambda using D - being able to easily build a static binary would make the process much more pleasant. I wrote this up a while back: http://awslambda-d.readthedocs.io/en/latest/ "Since dmd links phobos dynamically on linux, and phobos/druntime aren't installed on the AWS lambda server, we will need to upload these to the servers and tell the application where to find them. (I should really have appended to LD_LIBRARY_PATH as I did with PATH). Now one can follow the regular instructions for AWS Lambda: create a .zip file with the D binary, the JS file, and the following libraries (update version numbers as appropriate): libcrypto.so.1.0.0 libphobos2.so.0.67 libevent-2.0.so.5 libssl.so.1.0.0 " Alpine is nice - would be good to have this as a standard target in time. Laeeth.
Oct 23
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-10-23 14:13, Laeeth Isharc wrote:

 How can we generate a static binary ?
It's already supported by LDC, using the -static flag. This Linux binary [1] is statically linked. [1] https://github.com/jacob-carlborg/remarkify/releases -- /Jacob Carlborg
Oct 23
prev sibling next sibling parent Cym13 <cpicard openmailbox.org> writes:
On Monday, 23 October 2017 at 12:13:12 UTC, Laeeth Isharc wrote:
 ...
 And on the other hand, for deployment, it's much easier to copy 
 over one binary.  (In a sense it's funny that the question was 
 asked in the context of containers, because containers are kind 
 of an alternative to having a single binary).  When you're 
 properly set-up of course it doesn't matter, but when you're 
 moving towards that, a single binary is much easier.
 ...
A bit OT but this part made me think of [1] of which you may appreciate the insight. Right now I kind of feel like with D we get the downside in size of fat binaries without the upside in portability (although solutions exist for the dedicated and patient ones). This might proove an important drawback in the future... we'll see. [1]: http://www.smashcompany.com/technology/why-would-anyone-choose-docker-over-fat-binaries
Oct 25
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, October 23, 2017 12:13:12 Laeeth Isharc via Digitalmars-d wrote:
 How can we generate a static binary ?  I asked about this before,
 and the response was that it's a bad idea because of security
 vulns and so on.  True if you are running on a conventional Linux
 host.  But on the other hand, it's not that great if when the
 system phobos is upgraded all the D binaries break.  You can
 write a script using rdmd or dub scripting feature, but that
 doesn't work for more complex programs.
I believe that it works just fine if you do the linking step yourself, but dmd doesn't properly handle it with -L-static: https://issues.dlang.org/show_bug.cgi?id=6952 I have no idea how easy it would be to the linking manually with dub. Alternatively, I believe that you can statically link against any 3rd party libraries that you're using by linking to the .a file rather than by the library name. That won't work for glibc, but much as I'd love to statically link that, it's my understanding that doing so is generally a bad idea. I don't recall the reasons though - and with the increased stability in linux's ABI, I would have thought that it would be far less of an issue than it would have been in the past. Certainly, the concept of being able to statically link a program and then just run it on most any recent linux distro without recompiling it sounds very appealing. Yes, you'll have to recompile it for any security updates to any of your dependencies, whereas that can be avoided with dynamic linking, but depending on what you're doing, that often isn't really an issue, and it can be managed if it is (heck, you're doing basically the same thing when maintaining something like a docker image). - Jonathan M Davis
Oct 26
prev sibling parent reply aberba <karabutaworld gmail.com> writes:
On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 I just read the following two week-old comment on the ldc issue 
 tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some uptake, 
 but that D might not be that easy to deploy to that scenario 
 yet.
Its the future. Netflix, Google, Facebook, etc. all have open source tools around microservices. Its currently ruled by JavaScript > Go > Java. JavaScript being the leader. They have these in common: 1. Easy to deploy their code in docker containers including alpine Linux. 2. They have APIs for major cloud services. Both official and third-party. 3. Good support for networking. HTTP, Websockets, IPC*, etc. Mostly HTTP. That's it. The major advantages besides the hype. Lets see about D based on these requirements. 1. Kind of. There are dmd-ldc-dub docker images on docker hub. One by sociomantic. None using Alpine Linux as base though. Most people prefer alpine cus its lightweight (not a requirement). 2. Nope. Official APIs comes when there's mass adoption for that language and has good ROI. No complete and production ready cloud services API that I know of. Seems D users in that area roll out something on their own for private use with features they need. But most cloud PaaS provide support for custom runtimes which D has one for Heroku. Major candidate is Google AppEngine. 3. Phobos doesn't have a D HTTP API. Its sad but we have "requests" at code.dlang.org which works. We have vibe.d for http servers (aka nodejs of D) and seems popular based on threads about it and downloads.
 So this is a question for those deploying microservices, as I'm 
 not in that field, what can the D devs do to make it as easy as 
 possible to get D microservices up and running, make some 
 Docker and Alpine containers with ldc/dub/vibe.d preinstalled 
 publicly available?  What else, what kinds of libraries do you 
 normally use?
There several database APIs at cods.dlang.org. I don't know why some complain about no db APIs.
 This is a niche that D and all newer languages should target.  
 How do we do it?
Good question.
Oct 28
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On Saturday, 28 October 2017 at 14:55:25 UTC, aberba wrote:
 On Sunday, 22 October 2017 at 02:48:57 UTC, Joakim wrote:
 I just read the following two week-old comment on the ldc 
 issue tracker, when someone tried to run D on Alpine linux:

 "For now everything works(?) but I think the process could be 
 improved.. Would be really cool to have LDC easily building 
 alpine containers + static D binaries for microservice and 
 tooling development. I'm pretty tired of reading Go code :)"
 https://github.com/ldc-developers/ldc/issues/2341#issuecomment-334626550

 It strikes me that microservices are a great way for new 
 programming languages like D to get tried and gain some 
 uptake, but that D might not be that easy to deploy to that 
 scenario yet.
Its the future.
Highly doubt that. It really depend on the scale of your operations.
 Netflix, Google, Facebook, etc. all have open source tools 
 around microservices. Its currently ruled by JavaScript > Go > 
 Java. JavaScript being the leader.

 They have these in common:
 1. Easy to deploy their code in docker containers including 
 alpine Linux.
Interestingly while Docker may seem all the rage in startups I find its use limited to test environments in the real world. Also Java fat jars were super easy to deploy ages before docker. They are also a great deal smaller.
 2. They have APIs for major cloud services. Both official and 
 third-party.
 3. Good support for networking. HTTP, Websockets, IPC*, etc. 
 Mostly HTTP.
Honestly APIs these days can be written in anything that is able to put together a few HTTP responses. Unless you are doing serious work like: - DBs - Search engines - ML pipelines - Real-time event processing systems .... Any semimodern language/technology with a several hosts can manage to saturate 1Gbit link. Some take a certain amount of tuning others work out of the box. If you go for 40gbit/s it may be important to choose the right technology otherwise it’s all a matter of taste.
Oct 28