www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Vote for std.process

reply "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
It is that time, If you would like to see the proposed 
std.process include into Phobos please vote yes. If one condition 
must be met specify under what condition, otherwise vote no.

std.process by Lars Kyllingstad and Steven Schveighoffer is a 
suggested improvement to the existing std.process and is a major 
change to the API. The original API remains but these will be 
going through deprecation.

In summary of the discussion there was concern of the use of 
Error and Exception. Lars is very interested in getting the 
standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
at the safest level to allow for change without breaking code.

Please place any further comments in the official review thread 
leaving only your vote and a short comment (there should be no 
need to reply to anyone).

Docs: 
http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html

Source: 
https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d

Friday April 19 PST will be the last day of voting.
Apr 11 2013
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes.
Yes!
Apr 11 2013
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Fri, 12 Apr 2013 06:46:51 +0200
"Jesse Phillips" <Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one condition 
 must be met specify under what condition, otherwise vote no.
 
Yes
Apr 11 2013
parent reply Manu <turkeyman gmail.com> writes:
Sorry to derail the topic, but I'd just like to raise a major gripe I've
had with introducing anything into phobos recently.

I see this pattern where something is designed, discussed, and then voted
into phobos. At this time the design looks good on paper, but there is very
little practical experience using the library.
The problem then is, once accepted, people start using it, and at some
point some issues are found, or ideas for improvement are made based on
user experience, but the module can no longer be touched due to the general
phobia of making breaking changes...

Can I suggest that ALL new modules should be added to exp. rather than
std.? Here they will stay for at least 1 year, and while they live in exp,
it is understood that they are still in an experimental introductory phase.
Users who choose to use modules in exp are accepting that the API may
receive changes, and consequently, accepting the responsibility to update
their code (and not complain about it breaking their program), should the
library be amended in its introductory phase.
At some time later when the module has been used in a decent amount of
software, and the API has stabilised, it can then be moved to std. This
move is obviously a breaking change its self, but anyone who has agrees to
import exp modules has already accepted the responsibility to update their
code as such.

Thoughts?


On 12 April 2013 16:12, Nick Sabalausky <SeeWebsiteToContactMe semitwist.com
 wrote:
 On Fri, 12 Apr 2013 06:46:51 +0200
 "Jesse Phillips" <Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one condition
 must be met specify under what condition, otherwise vote no.
Yes
Apr 11 2013
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
 Sorry to derail the topic, but I'd just like to raise a major 
 gripe I've
 had with introducing anything into phobos recently.

 I see this pattern where something is designed, discussed, and 
 then voted
 into phobos. At this time the design looks good on paper, but 
 there is very
 little practical experience using the library.
 The problem then is, once accepted, people start using it, and 
 at some
 point some issues are found, or ideas for improvement are made 
 based on
 user experience, but the module can no longer be touched due to 
 the general
 phobia of making breaking changes...

 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.? Here they will stay for at least 1 year, and while they 
 live in exp,
 it is understood that they are still in an experimental 
 introductory phase.
 Users who choose to use modules in exp are accepting that the 
 API may
 receive changes, and consequently, accepting the responsibility 
 to update
 their code (and not complain about it breaking their program), 
 should the
 library be amended in its introductory phase.
 At some time later when the module has been used in a decent 
 amount of
 software, and the API has stabilised, it can then be moved to 
 std. This
 move is obviously a breaking change its self, but anyone who 
 has agrees to
 import exp modules has already accepted the responsibility to 
 update their
 code as such.

 Thoughts?


 On 12 April 2013 16:12, Nick Sabalausky 
 <SeeWebsiteToContactMe semitwist.com
 wrote:
 On Fri, 12 Apr 2013 06:46:51 +0200
 "Jesse Phillips" <Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one 
 condition
 must be met specify under what condition, otherwise vote no.
Yes
Fully agree
Apr 11 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 08:24, Manu wrote:

 Thoughts?
I think it's a good idea. -- /Jacob Carlborg
Apr 11 2013
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.?
Java tried this. As a result, they are now stuck with "javax" because they realized breaking backwards compatibility wasn't worth the benefit of "graduating" packages from "javax" to "java". If the goal is to include new or newly-overhauled modules in official releases, I think simply putting a big red "EXPERIMENTAL" warning at the top of the documentation would be simpler. Then, the warning can be removed when there is a D release which has no breaking changes in said module since its last released version. However, another direction would be to make it easier to test such fresh modules. For example, it would be nice to have "nightly" builds of D, both for the "master" version, and a "mob" version, which includes some work-in-progress developments.
Apr 12 2013
parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 07:00:25 UTC, Vladimir Panteleev 
wrote:
 On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
 Can I suggest that ALL new modules should be added to exp. 
 rather than
 std.?
Java tried this. As a result, they are now stuck with "javax" because they realized breaking backwards compatibility wasn't worth the benefit of "graduating" packages from "javax" to "java".
Actually they also have another issue. Java supported since the begging something similar to a System package for dirty tricks, officially it was a Sun only API. http://www.docjar.com/docs/api/sun/misc/Unsafe.html So actually all Java implementations provide a compatible API, because many do use it. Younger languages seem not to have any issue breaking backwards compatibility for experimental code. -- Paulo
Apr 12 2013
prev sibling next sibling parent reply "angel" <andrey.gelman gmail.com> writes:
 ...
 Thoughts?
Right suggestion. As it was already discussed once, D community could follow Linux in this regard: each new module enters 'staging' folder. At some point in time it goes into 'mainline'. This could also streamline the process of deprecating the old library.
Apr 12 2013
parent "James Wirth" <jwirth1 suddenlink.net> writes:
All progress requires risk. My vote is:
YES
Apr 12 2013
prev sibling parent reply "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
On Friday, 12 April 2013 at 06:25:10 UTC, Manu wrote:
 I see this pattern where something is designed, discussed, and 
 then voted
 into phobos. At this time the design looks good on paper, but 
 there is very
 little practical experience using the library.
 The problem then is, once accepted, people start using it, and 
 at some
 point some issues are found, or ideas for improvement are made 
 based on
 user experience, but the module can no longer be touched due to 
 the general
 phobia of making breaking changes...
I think this needs to happen prior to the formal review/voting. I would say it should be a precursor to starting the official review, however this would raise the bar too high for things like Jacob's Serialization library; he has a working library, but it isn't ready for Phobos and it would be silly to require the translation prior to approving it for Phobos. How we choose to add to the exp module would need some consideration.
Apr 12 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 16:04, Jesse Phillips wrote:

 I think this needs to happen prior to the formal review/voting. I would
 say it should be a precursor to starting the official review, however
 this would raise the bar too high for things like Jacob's Serialization
 library; he has a working library, but it isn't ready for Phobos and it
 would be silly to require the translation prior to approving it for Phobos.
What's left is basically just to put it under the std namespace and convert tabs to spaces. -- /Jacob Carlborg
Apr 13 2013
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Yes.
Apr 11 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/12/13, Jesse Phillips <Jessekphillips+d gmail.com> wrote:
 It is that time, If you would like to see the proposed
 std.process include into Phobos please vote yes. If one condition
 must be met specify under what condition, otherwise vote no.
Yes.
Apr 11 2013
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed std.process include
 into Phobos please vote yes. If one condition must be met specify under
 what condition, otherwise vote no.
I didn't think I had much of an opinion on std.process, but I just gave it a quick once over and noticed, relating to our recent discussions about flagrant GC usage throughout phobos, that this is riddled with the exact sorts of issues I've been talking about: string[string] is used in the main API to receive environment variables; perhaps kinda convenient, but it's impossible to supply environment variables with loads of allocations. toStringz is used liberally; alternatively, alloca() could allocate the c-string's on the stack and zero terminate them there, passing a pointer to the stack string to the OS functions. String concatenation is rampant! Look at this code to parse the env variables (which are already an AA): foreach (var, val; childEnv) envz[pos++] = (var~'='~val~'\0').ptr; And many many more of the same... This is a lib that spawns a process, I can't imagine why this library should need to allocate a single byte, but it allocates wildly just to perform a trivial pass-through to the OS. This module is no exception either, basically all of phobos is like this, and I'd like to see careful consideration made to these details in the future. I've said before, I sadly have to avoid phobos like the plague. Some modules (like this one) that provide fundamental functionality - not just helper functions - can't be avoided. Requirements for those should be extra strict in my opinion.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
 And many many more of the same...
So how many of these will produce a measurable performance increase when optimized, considering the relative cost of process creation?
 This is a lib that spawns a process, I can't imagine why this 
 library should need to allocate a single byte
So that the library fits in 3500 lines instead of many more, and has an API that can be invoked using one line, instead of ten.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:13, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:

 And many many more of the same...
So how many of these will produce a measurable performance increase when optimized, considering the relative cost of process creation?
I agree that spawning processes is a low-frequency operation, but it's a principle I'm trying to illustrate here. This is a lib that spawns a process, I can't imagine why this library
 should need to allocate a single byte
So that the library fits in 3500 lines instead of many more, and has an API that can be invoked using one line, instead of ten.
You can argue whatever you like. I've said my part, and I wherever it lands is not my call. I'm just illustrating the point that phobos, like STL, and to a lesser extent the CRT, are to continue to be avoided like the plague in some fields unless these details are considered in future. Most of the required changes would be self-contained, and not affect the API at all.
Apr 12 2013
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:
 I agree that spawning processes is a low-frequency operation, 
 but it's a
 principle I'm trying to illustrate here.
My point was that it is not that it's low-frequency, it's that the OS process creation operation is so expensive, that a few memory allocations will not make much of a difference in comparison. It's the same as optimizing memory allocations in a program which is intrinsically disk- or network-bound.
 You can argue whatever you like. I've said my part, and I 
 wherever it lands is not my call.
Well, that's just not very constructive. Your complaint in valid in general, but I was pointing out that it is not much so when specifically aimed at std.process.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:30, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:

 I agree that spawning processes is a low-frequency operation, but it's a
 principle I'm trying to illustrate here.
My point was that it is not that it's low-frequency, it's that the OS process creation operation is so expensive, that a few memory allocations will not make much of a difference in comparison. It's the same as optimizing memory allocations in a program which is intrinsically disk- or network-bound.
Which OS are we talking about? What OS runs on an a Nintendo Wii? There's only 24mb of system memory in that machine, can we afford to allocate it frivolously? Will I avoid phobos as a policy? Yes. You can argue whatever you like. I've said my part, and I wherever it
 lands is not my call.
Well, that's just not very constructive. Your complaint in valid in general, but I was pointing out that it is not much so when specifically aimed at std.process.
I don't necessarily disagree with your point (although I don't agree either). Perhaps in the context of std.process it's not so important... but it's still an opportunity to set a precedent. Honestly, any new module could have appeared for approval at this particular moment and I would have made the same criticisms. Not necessarily picking on std.process in particular, I'm making a point about phobos, and what is considered acceptable.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
 Which OS are we talking about?
 What OS runs on an a Nintendo Wii? There's only 24mb of system 
 memory in
 that machine, can we afford to allocate it frivolously?
I wasn't aware that writing games for the Wii involves creating processes in the thousands.
 I don't necessarily disagree with your point (although I don't 
 agree
 either). Perhaps in the context of std.process it's not so 
 important... but
 it's still an opportunity to set a precedent.
 Honestly, any new module could have appeared for approval at 
 this
 particular moment and I would have made the same criticisms. Not
 necessarily picking on std.process in particular, I'm making a 
 point about
 phobos, and what is considered acceptable.
Don't forget that shifting focus when adjusting goals is always a question of tradeoffs. Sacrificing tersity, maintainability, usability, safety etc. for the sake of performance would be a poor move in the case of std.process.
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:18, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

 Which OS are we talking about?
 What OS runs on an a Nintendo Wii? There's only 24mb of system memory in
 that machine, can we afford to allocate it frivolously?
I wasn't aware that writing games for the Wii involves creating processes in the thousands.
Again, I'm just suggesting possibilities, and trying to illustrate that it's a STANDARD library, you can never predict where users will want to use it. I don't necessarily disagree with your point (although I don't agree
 either). Perhaps in the context of std.process it's not so important...
 but
 it's still an opportunity to set a precedent.
 Honestly, any new module could have appeared for approval at this
 particular moment and I would have made the same criticisms. Not
 necessarily picking on std.process in particular, I'm making a point about
 phobos, and what is considered acceptable.
Don't forget that shifting focus when adjusting goals is always a question of tradeoffs. Sacrificing tersity, maintainability, usability, safety etc. for the sake of performance would be a poor move in the case of std.process.
Can you perhaps quantify how any of those things would be reduced by addressing at least the details I highlight? I'm basically advocating making a habit of using the stack where possible. It's not exactly hard, or cryptic.
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:
 Again, I'm just suggesting possibilities, and trying to 
 illustrate that
 it's a STANDARD library, you can never predict where users will 
 want to use
 it.
So you are talking about an imaginary system where memory allocation is expensive but process creation is cheap. It's impossible to design and optimize software for some intangible goals. None of the systems that D targets, or aims to target, meet that criteria.
 Can you perhaps quantify how any of those things would be 
 reduced by
 addressing at least the details I highlight?
 I'm basically advocating making a habit of using the stack 
 where possible.
 It's not exactly hard, or cryptic.
Yes. However, I suggest the following instead: Please rewrite some part of std.process with performance in mind, and post it here for review. This way, we can analyze the benefits and drawbacks based on a concrete example, instead of vapor and hot air.
Apr 12 2013
parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 09:38:53 +0100, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:

 On Friday, 12 April 2013 at 08:26:48 UTC, Manu wrote:
 Again, I'm just suggesting possibilities, and trying to illustrate that
 it's a STANDARD library, you can never predict where users will want to  
 use
 it.
So you are talking about an imaginary system where memory allocation is expensive but process creation is cheap. It's impossible to design and optimize software for some intangible goals. None of the systems that D targets, or aims to target, meet that criteria.
I don't understand why you're so "against" these comments. There is no real "argument" here, you both want std.process to be as good as it can be and it will clearly be better if it performs faster and allocates less. Yes, premature optimisation is generally frowned upon in "user" code, but library code should ideally be optimised. Should this percieved lack of optimisation prevent it's inclusion now, no because optimisation wont change the API (it may extend it however).
 Can you perhaps quantify how any of those things would be reduced by
 addressing at least the details I highlight?
 I'm basically advocating making a habit of using the stack where  
 possible.
 It's not exactly hard, or cryptic.
Yes. However, I suggest the following instead: Please rewrite some part of std.process with performance in mind, and post it here for review. This way, we can analyze the benefits and drawbacks based on a concrete example, instead of vapor and hot air.
Fair point, after all someone has to do the work and it seems logical that the person with the most understand of the issue should so it. Whether he has the time however is the Q/problem. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:
 I don't understand why you're so "against" these comments.  
 There is no real "argument" here, you both want std.process to 
 be as good as it can be and it will clearly be better if it 
 performs faster and allocates less.
This is a fallacy. Performance is not free. Somebody must do the work. D contributors do not have an infinity of time and motivation. Optimizing code often implies making it more complicated. Straight-forward code is self-documenting. More complicated code is harder to read and review. More complicated code more easily hides bugs - including security bugs, such as buffer overflows. Maintaining optimized code requires understanding not only the high-level logic, but also the low-level optimization details. The benefits of optimizing std.process are likely to be so small, as to be difficult to measure. Would you still say that the above costs are worth the nearly-intangible gain?
Apr 12 2013
parent reply "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
<vladimir thecybershadow.net> wrote:

 On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:
 I don't understand why you're so "against" these comments.  There is no  
 real "argument" here, you both want std.process to be as good as it can  
 be and it will clearly be better if it performs faster and allocates  
 less.
This is a fallacy.
Nope, you're attacking a Strawman here. At most my statement is incomplete because it doesn't address the costs (which I left out for brevity).
 Performance is not free.
I never claimed it was (thus the Strawman). I simply claimed std.process would be "better" if it performed faster and allocated less - which is true.
 Somebody must do the work.
 D contributors do not have an infinity of time and motivation.
I know this as well as anyone, my own free time is in short supply.
 Optimizing code often implies making it more complicated.

 Straight-forward code is self-documenting.

 More complicated code is harder to read and review.

 More complicated code more easily hides bugs - including security bugs,  
 such as buffer overflows.

 Maintaining optimized code requires understanding not only the  
 high-level logic, but also the low-level optimization details.

 The benefits of optimizing std.process are likely to be so small, as to  
 be difficult to measure.
All true. However, complexity can and should be packaged in such a way as to localise, and this localised complex code should be tested to death and maintained by someone who understands it. It should be bracketed by sufficient comments and warnings about how/why it does what it does. The resulting packaged complexity, with it's associated cost can be re-used many times over for all the benefit it gives. Stack allocating the environment variables need not be a localised improvement but could be a standard library function which can be reused, for example.
 Would you still say that the above costs are worth the nearly-intangible  
 gain?
"nearly-intangible" is wrong. Library code is code which is used by (hopefully) millions of people, writing millions of applications, running for millions of hours, on millions of systems, creating thousands of processes, etc.. In short, a little effort now pays massive dividends over it's lifetime. So, yes, IMO the costs shown above are worth the resulting gains. D is a systems programming language, there is hope that it will penetrate a wide range of systems and environments - sure in many cases a little bit of memory use or performance loss is unimportant, but for many it will be the decisive factor which makes D unusable there. D is constantly being compared to other languages on the basis of performance, so it's clearly an important aspect of D's success. Library code needs to work first time and work well or people will roll their own wasting time, energy and in many cases getting some aspect of it just plain wrong. Again, I'm not suggesting (nor was Manu I think) that this in any way blocks the inclusion of std.process. But I do share his desire for performance to sometimes be a little higher up the priority list. R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
 All true.  However, complexity can and should be packaged in 
 such a way as to localise, and this localised complex code 
 should be tested to death and maintained by someone who 
 understands it.  It should be bracketed by sufficient comments 
 and warnings about how/why it does what it does.  The resulting 
 packaged complexity, with it's associated cost can be re-used 
 many times over for all the benefit it gives.

 Stack allocating the environment variables need not be a 
 localised improvement but could be a standard library function 
 which can be reused, for example.
Performant abstractions. Like the much-awaited allocator design? Either way, they can only diminuate, not remove the costs.
 Would you still say that the above costs are worth the 
 nearly-intangible gain?
"nearly-intangible" is wrong. Library code is code which is used by (hopefully) millions of people, writing millions of applications, running for millions of hours, on millions of systems, creating thousands of processes, etc.. In short, a little effort now pays massive dividends over it's lifetime. So, yes, IMO the costs shown above are worth the resulting gains. D is constantly being compared to other languages on the basis of performance, so it's clearly an important aspect of D's success. Library code needs to work first time and work well or people will roll their own wasting time, energy and in many cases getting some aspect of it just plain wrong.
But once again, you speak in vague terms. Consider the following hypothetical decisions and outcomes: 1. std.process is left at is. One user is angry / turned away because it performs 0.1% slower than it can be. 2. std.process is rewritten to minimize allocations. Code complexity goes up, new improvements are challenging to add; bugs pop up and go unfixed for a while because fewer programmers are qualified or willing to commit the effort of making correct fixes. More people are angry / turned away from D because its standard library is buggy. Of course, the above is an exaggerated illustration. But would optimizing all code left and right really make more D users happier? There's also the question of priorities. Would you rather than effort is spent on optimizing std.process (and dealing with all the fallout from any such optimizations), or working on something that is acutely missing and hurting D?
 D is a systems programming language, there is hope that it will 
 penetrate a wide range of systems and environments - sure in 
 many cases a little bit of memory use or performance loss is 
 unimportant, but for many it will be the decisive factor which 
 makes D unusable there.
This is surely an exaggeration. D does not attempt to please everyone out there who is choosing a programming language for their next project. There is no such language, nor can one exist. One has to accept that D has a number of goals, none of which are absolute, but merely point towards a certain, but not overly specific, point in the multidimensional matrix of trade-offs. D never was about achieving maximum performance in all possible cases.
Apr 12 2013
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Fri, 12 Apr 2013 13:00:29 +0200
"Vladimir Panteleev" <vladimir thecybershadow.net> wrote:
 
 There's also the question of priorities. Would you rather than 
 effort is spent on optimizing std.process (and dealing with all 
 the fallout from any such optimizations), or working on something 
 that is acutely missing and hurting D?
 
While I'm not necessarily disagreeing with the other points you raise, Phobos's excess allocations *are* verifiability hurting D. A couple of the biggest potential user bases for D are videogames and embedded. I feel confident in saying that no other language has as much potential in these areas as D has. But these groups have *already*, vocally, been facing the problem of avoiding/rewriting potentially-large parts of Phobos, or sticking with C/C++. Now, you could argue that many of these people are simply being overly fearful of a merely imagined problem, but even if that's true, the problem is still real in it's effects: Hindering D adoption and causing people to (perhaps needlessly) avoid/rewrite parts of Phobos. Now, I'm not suggesting that we do *or* don't start a big effort to minimize allocations throughout Phobos, I'm simply objecting to the implication that unnecessary allocations in Phobos aren't hurting D in any way.
Apr 12 2013
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 16:42:18 UTC, Nick Sabalausky wrote:
 While I'm not necessarily disagreeing with the other points you 
 raise,
 Phobos's excess allocations *are* verifiability hurting D. A 
 couple of
 the biggest potential user bases for D are videogames and 
 embedded. I
 feel confident in saying that no other language has as much 
 potential
 in these areas as D has. But these groups have *already*, 
 vocally, been
 facing the problem of avoiding/rewriting potentially-large 
 parts of
 Phobos, or sticking with C/C++.

 Now, you could argue that many of these people are simply being 
 overly
 fearful of a merely imagined problem, but even if that's true, 
 the
 problem is still real in it's effects: Hindering D adoption and 
 causing
 people to (perhaps needlessly) avoid/rewrite parts of Phobos.

 Now, I'm not suggesting that we do *or* don't start a big 
 effort to
 minimize allocations throughout Phobos, I'm simply objecting to 
 the
 implication that unnecessary allocations in Phobos aren't 
 hurting D in
 any way.
No, that's not what I was trying to say. I agree with your post completely. My point was specifically about std.process in particular, since I was misunderstanding why Manu was using std.process as an example.
Apr 12 2013
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 06:14:35 -0400, Regan Heath <regan netmail.co.nz>  
wrote:

 On Fri, 12 Apr 2013 10:17:54 +0100, Vladimir Panteleev  
 <vladimir thecybershadow.net> wrote:
 Optimizing code often implies making it more complicated.

 Straight-forward code is self-documenting.

 More complicated code is harder to read and review.

 More complicated code more easily hides bugs - including security bugs,  
 such as buffer overflows.

 Maintaining optimized code requires understanding not only the  
 high-level logic, but also the low-level optimization details.

 The benefits of optimizing std.process are likely to be so small, as to  
 be difficult to measure.
All true. However, complexity can and should be packaged in such a way as to localise, and this localised complex code should be tested to death and maintained by someone who understands it. It should be bracketed by sufficient comments and warnings about how/why it does what it does. The resulting packaged complexity, with it's associated cost can be re-used many times over for all the benefit it gives.
I would say I agree with both of you. In this particular instance, maintainability trumps performance, since creating processes is fundamentally not cheap. But if someone is willing to step up an give us a good alternative, there is no reason not to use it. It should be readable and well documented. I think in order to achieve good performance, we should be focused on custom allocators. That really is the roadblock to all of this. -Steve
Apr 12 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 12 April 2013 at 08:57:19 UTC, Regan Heath wrote:
 ...
It is not something to discuss in this topic. Yes, phobos has allocations issues. Yes, it is worth discussing. Does std.process do something special in that sense? No, possible problems are more related to lack of general configurable allocator. Please do not fill voting thread with noise if you have no _blocking_ objections.
Apr 12 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Apr-2013 12:26, Manu пишет:
 On 12 April 2013 18:18, Vladimir Panteleev <vladimir thecybershadow.net
 <mailto:vladimir thecybershadow.net>> wrote:

     On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:

         Which OS are we talking about?
         What OS runs on an a Nintendo Wii? There's only 24mb of system
         memory in
         that machine, can we afford to allocate it frivolously?


     I wasn't aware that writing games for the Wii involves creating
     processes in the thousands.


 Again, I'm just suggesting possibilities, and trying to illustrate that
 it's a STANDARD library, you can never predict where users will want to
 use it.
Very true. -- Dmitry Olshansky
Apr 12 2013
prev sibling next sibling parent reply "Manipulator" <volcz kth.se> writes:
Please correct me if I'm wrong.
It's not the allocation itself that is the problem it's that it 
will trigger the GC to stop the world at some point.
Apr 12 2013
parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:21, Manipulator <volcz kth.se> wrote:

 Please correct me if I'm wrong.
 It's not the allocation itself that is the problem it's that it will
 trigger the GC to stop the world at some point.
It's just another source of garbage. Which means this function of off limits beneath a routine that's disabled collection, for instance. There are lots of reasons to avoid making unnecessary garbage.
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
 Honestly, any new module could have appeared for approval at 
 this
 particular moment and I would have made the same criticisms.
Observation: this move wasn't something I would encourage. Jesse, the review manager for std.process, has specifically asked:
 Please place any further comments in the official review thread 
 leaving only your vote and a short comment (there should be no 
 need to reply to anyone).
There was ample time to discuss std.process during the informal and formal review. However, the situation with your replies is even worse: they don't even concern std.process in particular. Please consider starting a new thread next time, and stating your criticism aside to whatever poor module comes up the review queue ;)
Apr 12 2013
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 07:53:04 UTC, Manu wrote:
 On 12 April 2013 17:30, Vladimir Panteleev 
 <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 07:22:42 UTC, Manu wrote:

 I agree that spawning processes is a low-frequency operation, 
 but it's a
 principle I'm trying to illustrate here.
My point was that it is not that it's low-frequency, it's that the OS process creation operation is so expensive, that a few memory allocations will not make much of a difference in comparison. It's the same as optimizing memory allocations in a program which is intrinsically disk- or network-bound.
Which OS are we talking about? What OS runs on an a Nintendo Wii? There's only 24mb of system memory in that machine, can we afford to allocate it frivolously? Will I avoid phobos as a policy? Yes.
Do you create process at all? I always had the impression those environments used real time OS like libraries and multitasking is achieved by co-routines or threads, not real processes. Just a dummy comment from an industry outsider. -- Paulo
Apr 12 2013
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 03:04:08 -0400, Manu <turkeyman gmail.com> wrote:

 On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com>  
 wrote:

 It is that time, If you would like to see the proposed std.process  
 include
 into Phobos please vote yes. If one condition must be met specify under
 what condition, otherwise vote no.
I didn't think I had much of an opinion on std.process, but I just gave it a quick once over and noticed, relating to our recent discussions about flagrant GC usage throughout phobos, that this is riddled with the exact sorts of issues I've been talking about:
It's spawning a new process. What is the issue with allocating a bit of memory? The spawning of the process performance is going to dwarf that of any memory allocations.
 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.
I think you meant "without"? What would be your suggestion? string[string] is the built-in map type. How do you pass an environment map without having some allocations?
 toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer  
 to
 the stack string to the OS functions.
This would be a lot of effort for pretty much zero gain for the majority of cases.
 String concatenation is rampant! Look at this code to parse the env
 variables (which are already an AA):

     foreach (var, val; childEnv)
         envz[pos++] = (var~'='~val~'\0').ptr;
This could be improved. It could also be optimized into a single allocation automatically by the compiler (it might already be). The API would not be affected by this improvement, though.
 And many many more of the same...

 This is a lib that spawns a process, I can't imagine why this library
 should need to allocate a single byte, but it allocates wildly just to
 perform a trivial pass-through to the OS.
It is not a trivial pass-through. Different OSes require different parameter types. Look at the difference between the windows and posix implementations. We are writing a cross-platform library, so whatever we pick will have to be converted on at least one OS.
 This module is no exception either, basically all of phobos is like this,
 and I'd like to see careful consideration made to these details in the
 future.
I too like to avoid memory allocations. But the level to which you require is too stringent. Not even Tango, which made low heap-allocations a priority, avoided allocations in it's process library.
 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not just
 helper functions - can't be avoided. Requirements for those should be  
 extra
 strict in my opinion.
We cannot cater to all developers. To put all developers through a lot of pain in order to attract the fringe ones is not a practical goal. I think the convenience of std.process is what it is there for. For the highest performance, you are able to call the OS functions directly. -Steve
Apr 12 2013
parent reply Manu <turkeyman gmail.com> writes:
On 12 April 2013 17:35, Steven Schveighoffer <schveiguy yahoo.com> wrote:

 On Fri, 12 Apr 2013 03:04:08 -0400, Manu <turkeyman gmail.com> wrote:
string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.
I think you meant "without"?
Yes. What would be your suggestion? string[string] is the built-in map type.
  How do you pass an environment map without having some allocations?
I'd use string[]. toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer
 to
 the stack string to the OS functions.
This would be a lot of effort for pretty much zero gain for the majority of cases.
A trivial ... mixin template (?) could wrap it up, I don't see it particularly less convenient than calling toStringz(). Perhaps there are other tools missing from phobos if this is somehow hard... String concatenation is rampant! Look at this code to parse the env
 variables (which are already an AA):

     foreach (var, val; childEnv)
         envz[pos++] = (var~'='~val~'\0').ptr;
This could be improved. It could also be optimized into a single allocation automatically by the compiler (it might already be). The API would not be affected by this improvement, though.
I've never seem the compiler apply that optimisation, although I often wish it would. I saw an appender appear a few pages below, that would be an improvement here too I guess. And many many more of the same...
 This is a lib that spawns a process, I can't imagine why this library
 should need to allocate a single byte, but it allocates wildly just to
 perform a trivial pass-through to the OS.
It is not a trivial pass-through. Different OSes require different parameter types. Look at the difference between the windows and posix implementations. We are writing a cross-platform library, so whatever we pick will have to be converted on at least one OS.
Writing cross-platform code is my life. They are trivial conversions. It could be done on the stack easily. Granted, ideally the compiler could theoretically perform some of those improvements, but it doesn't, and probably depends on other changes to be able to do so anyway. For instance, the compiler couldn't confidently lower the allocations to the stack unless 'scope' arguments worked, so it knew it wouldn't escape the OS call it was being handed to. This module is no exception either, basically all of phobos is like this,
 and I'd like to see careful consideration made to these details in the
 future.
I too like to avoid memory allocations. But the level to which you require is too stringent. Not even Tango, which made low heap-allocations a priority, avoided allocations in it's process library.
But it would be trivial to use the stack in these cases. It doesn't matter if it adds a few lines under the hood. I'm really just trying to make the point that it should ALWAYS be a key consideration, and people should make it habit to think about/criticise these things when accepting code into phobos. This is the standard library, it will be used in more D code than any other library ever. Why skimp here? I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not just
 helper functions - can't be avoided. Requirements for those should be
 extra
 strict in my opinion.
We cannot cater to all developers.
In this case you certainly can, it would be fairly trivial to do. To put all developers through a lot of pain in order to attract the fringe
 ones is not a practical goal.
What pain? Are you saying a stack variable is somehow hard? If that's the case, then there's clearly some other fundamental tools missing from phobos. I think the convenience of std.process is what it is there for. For the
 highest performance, you are able to call the OS functions directly.
No, it's to save (ideally) _all_ programmers the effort of rewriting this module themselves. This is exactly the sort of thing I waste loads of my life doing, rewriting wheels because the author was a PC programmer, and simply didn't give a shit. What a waste of life >_< I re-iterate, maybe my points aren't of critical importance in relation to std.process specifically, I'm trying to raise some general issues with phobos submissions (that shouldn't exclude std.process either).
Apr 12 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:
 ...
While I generally agree with your point, std.process is hardly the place to start from. It is not performance-critical functionality and it is not basic functionality that may be used in other critical modules. It is blaming consequences instead of reasons, not really practical.
Apr 12 2013
parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:21, Dicebot <m.strashun gmail.com> wrote:

 On Friday, 12 April 2013 at 08:14:25 UTC, Manu wrote:

 ...
While I generally agree with your point, std.process is hardly the place to start from. It is not performance-critical functionality and it is not basic functionality that may be used in other critical modules. It is blaming consequences instead of reasons, not really practical.
And I generally agree with your point. Like I said, I would have made this criticisms of whatever came alone ;)
Apr 12 2013
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com> wrote:

 On 12 April 2013 17:35, Steven Schveighoffer <schveiguy yahoo.com> wrote:
 What would be your suggestion?  string[string] is the built-in map type.
  How do you pass an environment map without having some allocations?
I'd use string[].
You mean with format "a=b"? I suppose that's possible, though horrible to work with before passing in. Plus what happens if you have ["a=b", "a=c"] ? AA's prevent these kinds of mistakes/contradictions. I think someone suggested using a templated map type, that is probably a good idea. Then you can provide whatever type you wish, with opIndex capability.
 toStringz is used liberally; alternatively, alloca() could allocate the
  c-string's on the stack and zero terminate them there, passing a  
 pointer to
  the stack string to the OS functions.
  This would be a lot of effort for pretty much zero gain for the  
 majority of cases.
A trivial ... mixin template (?) could wrap it up, I don't see it particularly less convenient than calling toStringz(). Perhaps there are other tools missing from phobos if this is somehow hard...
alloca is hard to use, specifically because it can't be wrapped into a function. I don't know how easy it would be to do with a mixin, I haven't tried it. Note that toStringz likely is going to avoid reallocation or copying if possible -- it's very likely that a string already has a 0 after it, and also very likely that if it doesn't, it has room to append a 0. I still feel that if it can't be done easily, it over-complicates code for very little gain. As others have said too, this is an implementation detail. It can be done as a later pull request if you feel up to making it work! -Steve
Apr 12 2013
parent reply "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer 
wrote:
 On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com>
 I'd use string[].
You mean with format "a=b"? I suppose that's possible, though horrible to work with before passing in. Plus what happens if you have ["a=b", "a=c"] ? AA's prevent these kinds of mistakes/contradictions.
I prefer Manu's idea with the API accepting string[], it's closer to the native format. Then you could simply provide a convenience conversion from a map... ex env!["foo" : "bar"] which would convert it to ["foo=bar"]. This also has the added benefit of being self documenting(considering the lack of named parameters). But most importantly So the user has a free choice of constructing the env parameter manually in the most efficient way or using the lazy convenience function.
Apr 12 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> wrote:

 On Friday, 12 April 2013 at 15:43:27 UTC, Steven Schveighoffer wrote:
 On Fri, 12 Apr 2013 04:14:15 -0400, Manu <turkeyman gmail.com>
 I'd use string[].
You mean with format "a=b"? I suppose that's possible, though horrible to work with before passing in. Plus what happens if you have ["a=b", "a=c"] ? AA's prevent these kinds of mistakes/contradictions.
I prefer Manu's idea with the API accepting string[], it's closer to the native format.
That's great for the library on POSIX systems, but for the user, it's not as straightforward. One must construct "x=y" strings instead of setting x to y. Unless you are using literals, this involves an allocation anyway. Also, this is not the native format on Windows which passes the environment in one large string delimited by nulls.
 Then you could simply provide a convenience conversion from a map...
 ex env!["foo" : "bar"] which would convert it to ["foo=bar"].

 This also has the added benefit of being self documenting(considering  
 the lack of named parameters).
So for the most convenient/common case, you want to add an allocation?
 But most importantly So the user has a free choice of constructing the  
 env parameter manually in the most efficient way or using the lazy  
 convenience function.
I think the best suggestion so far is to allow a templated version for which you can provide an opIndex method, and a possible toEnv method (versioned for both Posix and Windows of course). Then you have the power to avoid allocation as much as possible. At the moment with AA's being the interface, we have no possibility to avoid allocation, and I understand that point. But I don't want to eliminate that case or make it specifically more inefficient. -Steve
Apr 12 2013
parent reply "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
wrote:
 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> 
 wrote:


 So for the most convenient/common case, you want to add an 
 allocation?
with the original proposal there is one anyway... But with my suggested approach you could create many processes reusing the same env... only paying the conversion/allocation cost once(outside of the loop).
Apr 12 2013
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 16:32:37 -0400, Tove <tove fransson.se> wrote:

 On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer wrote:
 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> wrote:


 So for the most convenient/common case, you want to add an allocation?
with the original proposal there is one anyway...
I meant add an additional allocation to what is there. There needs to be one allocation to collect all the variables into one long string (on Windows), and with your suggestion, it has to go through an intermediate "key=value" format.
 But with my suggested approach you could create many processes reusing  
 the same env... only paying the conversion/allocation cost once(outside  
 of the loop).
This would be attractive. But I still want a indexable object. Having to generate a=b strings is awkward. It could be something that does the right thing on POSIX (generate "a=b" under the hood) or Windows (Probably can cache the generated environment string). -Steve
Apr 12 2013
parent "Tove" <tove fransson.se> writes:
On Friday, 12 April 2013 at 20:52:55 UTC, Steven Schveighoffer 
wrote:
 On Fri, 12 Apr 2013 16:32:37 -0400, Tove <tove fransson.se> 
 wrote:

 On Friday, 12 April 2013 at 20:24:05 UTC, Steven Schveighoffer 
 wrote:
 On Fri, 12 Apr 2013 15:26:12 -0400, Tove <tove fransson.se> 
 wrote:


 So for the most convenient/common case, you want to add an 
 allocation?
with the original proposal there is one anyway...
I meant add an additional allocation to what is there. There needs to be one allocation to collect all the variables into one long string (on Windows), and with your suggestion, it has to go through an intermediate "key=value" format.
 But with my suggested approach you could create many processes 
 reusing the same env... only paying the conversion/allocation 
 cost once(outside of the loop).
This would be attractive. But I still want a indexable object. Having to generate a=b strings is awkward. It could be something that does the right thing on POSIX (generate "a=b" under the hood) or Windows (Probably can cache the generated environment string). -Steve
Hmhm, I see your point. Could our custom indexable object have 'alias this' to an 'union env'? which is the in param to the process api?
Apr 12 2013
prev sibling parent Martin Nowak <code dawg.eu> writes:
On 04/12/2013 10:14 AM, Manu wrote:
         String concatenation is rampant! Look at this code to parse the env
         variables (which are already an AA):

              foreach (var, val; childEnv)
                  envz[pos++] = (var~'='~val~'\0').ptr;


     This could be improved.  It could also be optimized into a single
     allocation automatically by the compiler (it might already be).  The
     API would not be affected by this improvement, though.


 I've never seem the compiler apply that optimisation, although I often
 wish it would.
 I saw an appender appear a few pages below, that would be an improvement
 here too I guess.
It is only a single runtime call to _d_arraycatnT which precomputes the resulting length. https://github.com/D-Programming-Language/druntime/blob/e5415ed4f0638fcd1d0f8e06aec4197ad108449f/src/rt/lifetime.d#L2031
Apr 13 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu <turkeyman gmail.com>:

 
 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.
 
Most (GC) allocations could be fixed without breaking the API. I can see 2 places where the API forces _GC_ allocations: * string[string] * thrown Exceptions are allocated with the GC There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult... If you want to get rid of all (not only GC) allocations, there's another issue: As D strings are not zero terminated we'll always have allocations passing those to C code. Maybe we should have a cstring type in phobos which would just be a string which is guaranteed to be zero terminated. http://dpaste.dzfl.pl/e76aa995 (needs some inout though)
Apr 12 2013
next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Friday, 12 April 2013 at 08:34:15 UTC, Johannes Pfau wrote:
 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:

 
 I've said before, I sadly have to avoid phobos like the 
 plague. Some
 modules (like this one) that provide fundamental functionality 
 - not
 just helper functions - can't be avoided. Requirements for 
 those
 should be extra strict in my opinion.
 
Most (GC) allocations could be fixed without breaking the API. I can see 2 places where the API forces _GC_ allocations: * string[string] * thrown Exceptions are allocated with the GC There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult... If you want to get rid of all (not only GC) allocations, there's another issue: As D strings are not zero terminated we'll always have allocations passing those to C code. Maybe we should have a cstring type in phobos which would just be a string which is guaranteed to be zero terminated. http://dpaste.dzfl.pl/e76aa995 (needs some inout though)
This is the Object Pascal way, although you can reach to a point where you get language native strings and C strings mixed all over the place. Not to mention that C strings should not be allowed in safe code. -- Paulo
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:34, Johannes Pfau <nospam example.com> wrote:

 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:

 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.
Most (GC) allocations could be fixed without breaking the API. I can see 2 places where the API forces _GC_ allocations: * string[string] * thrown Exceptions are allocated with the GC There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult...
Exceptions are exceptional, I don't think that's an issue. the string[string] in the api is guaranteed garbage, and will never be changed. What's the advantage of using an AA in this case? If you want to get rid of all (not only GC) allocations, there's another
 issue: As D strings are not zero terminated we'll always have
 allocations passing those to C code.
They are being passed through to a C function that doesn't retain the pointer; they can be allocated on the stack. Maybe we should have a cstring
 type in phobos which would just be a string which is guaranteed to be
 zero terminated.
Maybe, although the point here is to wrap up and hide the C API, so that's not useful here. A helper in phobos for allocating a (c)string on the stack may be useful...
Apr 12 2013
prev sibling parent Manu <turkeyman gmail.com> writes:
On 12 April 2013 18:34, Johannes Pfau <nospam example.com> wrote:

 Am Fri, 12 Apr 2013 17:04:08 +1000
 schrieb Manu <turkeyman gmail.com>:

 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those
 should be extra strict in my opinion.
Most (GC) allocations could be fixed without breaking the API. I can see 2 places where the API forces _GC_ allocations: * string[string] * thrown Exceptions are allocated with the GC There is no simple solution for these. Maybe we'll have a hashtable in the standard library at some point which will allow to use custom allocators. Then std.process could use these. Exceptions require some thinking. You can of course allocate them using any allocator, but freeing them is difficult...
Exceptions are exceptional, I don't think that's an issue. the string[string] in the api is guaranteed garbage, and will never be changed. What's the advantage of using an AA in this case? If you want to get rid of all (not only GC) allocations, there's another
 issue: As D strings are not zero terminated we'll always have
 allocations passing those to C code.
They are being passed through to a C function that doesn't retain the pointer; they can be allocated on the stack. Maybe we should have a cstring
 type in phobos which would just be a string which is guaranteed to be
 zero terminated.
Maybe, although the point here is to wrap up and hide the C API, so that's not useful here. A helper in phobos for allocating a (c)string on the stack may be useful...
Apr 12 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 17:04:08 +1000
schrieb Manu <turkeyman gmail.com>:


 
 toStringz is used liberally; alternatively, alloca() could allocate
 the c-string's on the stack and zero terminate them there, passing a
 pointer to the stack string to the OS functions.
 
Are you sure this is possible in this specific case? The execve manpage says: execve() does not return on success, and the text, data, bss, and _stack_ of the calling process are overwritten by that of the program loaded. There is no documentation if the environment variables are copied internally in execve.
Apr 12 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-04-12 09:04, Manu wrote:

 I've said before, I sadly have to avoid phobos like the plague. Some
 modules (like this one) that provide fundamental functionality - not
 just helper functions - can't be avoided. Requirements for those should
 be extra strict in my opinion.
Have you tried Tango? For most functions that do some array or string processing you have the option of passing your own buffer. If the buffer is too small, the function will allocate, otherwise it won't. Docs: http://www.dsource.org/projects/tango/docs/current/ Tango D2: https://github.com/SiegeLord/Tango-D2 -- /Jacob Carlborg
Apr 12 2013
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-Apr-2013 11:04, Manu пишет:
 On 12 April 2013 14:46, Jesse Phillips <Jessekphillips+d gmail.com
 <mailto:Jessekphillips+d gmail.com>> wrote:

     It is that time, If you would like to see the proposed std.process
     include into Phobos please vote yes. If one condition must be met
     specify under what condition, otherwise vote no.


 I didn't think I had much of an opinion on std.process, but I just gave
 it a quick once over and noticed, relating to our recent discussions
 about flagrant GC usage throughout phobos, that this is riddled with the
 exact sorts of issues I've been talking about:

 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.

 toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer
 to the stack string to the OS functions.
Most of this is unsolvable until there is a STANDARD interface/policy/hooks for allocators in Phobos. All work to get around the cases like that at the moment is just a waste of time or specific hacks for particular bottlenecks. I did a fair amount of these in std.regex to make matching GC-free and almost heap-free - it preallocates on the first call. It's an example of a necessary hack and it's not pretty. More then that I'm sure I'd have to re-write the whole memory allocation code once we (finally!) have he standard way to do it. TL;DR - until using allocators is easy and straight-forward (and standard) there would be no end of GC-only hegeomny 'cause it's easier/simpler. Not much to discuss ;)
 String concatenation is rampant! Look at this code to parse the env
 variables (which are already an AA):

 foreach (var, val; childEnv)

 envz[pos++] = (var~'='~val~'\0').ptr;
Horrible... But let's focus on the API parts. The internals could be fixed with a pull request or two. Freaking out because of a GC happy code doesn't help those waiting for *any* sane process management module in std. -- Dmitry Olshansky
Apr 12 2013
prev sibling parent "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
This is good feedback and all, but it would be good to have had 
this placed in the official review thread.

But it is too late now and I would not want to break the 
discussion up.
Apr 12 2013
prev sibling next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
Yes.
Apr 12 2013
parent "timotheecour" <timothee.cour2 gmail.com> writes:
Yes.
Apr 12 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 12 Apr 2013 06:46:51 +0200
schrieb "Jesse Phillips" <Jessekphillips+d gmail.com>:

 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one condition 
 must be met specify under what condition, otherwise vote no.
 
 std.process by Lars Kyllingstad and Steven Schveighoffer is a 
 suggested improvement to the existing std.process and is a major 
 change to the API. The original API remains but these will be 
 going through deprecation.
 
 In summary of the discussion there was concern of the use of 
 Error and Exception. Lars is very interested in getting the 
 standard hierarchy http://wiki.dlang.org/DIP33 thus leaving these 
 at the safest level to allow for change without breaking code.
 
 Please place any further comments in the official review thread 
 leaving only your vote and a short comment (there should be no 
 need to reply to anyone).
 
 Docs: 
 http://www.kyllingen.net/code/std-process2/phobos-prerelease/std_process.html
 
 Source: 
 https://github.com/kyllingstad/phobos/blob/std-process2/std/process.d
 
 Friday April 19 PST will be the last day of voting.
Yes!
Apr 12 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Apr 12, 2013 at 06:46:51AM +0200, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed std.process
 include into Phobos please vote yes. If one condition must be met
 specify under what condition, otherwise vote no.
Yes! --T
Apr 12 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes.
Yes! :)
Apr 12 2013
prev sibling next sibling parent "Regan Heath" <regan netmail.co.nz> writes:
On Fri, 12 Apr 2013 05:46:51 +0100, Jesse Phillips  
<Jessekphillips+d gmail.com> wrote:

 It is that time, If you would like to see the proposed std.process  
 include into Phobos please vote yes. If one condition must be met  
 specify under what condition, otherwise vote no.
Yes. -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Apr 12 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.

 std.process by Lars Kyllingstad and Steven Schveighoffer is a 
 suggested improvement to the existing std.process and is a 
 major change to the API. The original API remains but these 
 will be going through deprecation.
Yes. A compiler warning like for deprecation should added for "experimental" IMO rather than use exp. vs std.
Apr 12 2013
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 4/12/13, Manu <turkeyman gmail.com> wrote:
 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.
I think the functions which take the environment as an associative array can be changed without breaking any user code. E.g.: Pid spawnProcess(Env)(in char[][] args, File stdin = std.stdio.stdin, File stdout = std.stdio.stdout, File stderr = std.stdio.stderr, /* was const string[string] env = null */ const Env env = Env.init, Config config = Config.none) if (isSomeAssociativeArray!Env) // introduce constraint { } Then the user can provide their custom stack-based type which implements opIn_r.
Apr 12 2013
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 11:24:37 -0400, Andrej Mitrovic  
<andrej.mitrovich gmail.com> wrote:

 On 4/12/13, Manu <turkeyman gmail.com> wrote:
 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.
I think the functions which take the environment as an associative array can be changed without breaking any user code. E.g.: Pid spawnProcess(Env)(in char[][] args, File stdin = std.stdio.stdin, File stdout = std.stdio.stdout, File stderr = std.stdio.stderr, /* was const string[string] env = null */ const Env env = Env.init, Config config = Config.none) if (isSomeAssociativeArray!Env) // introduce constraint { } Then the user can provide their custom stack-based type which implements opIn_r.
It's a good idea. I believe it can be done later without breaking any code. -Steve
Apr 12 2013
prev sibling next sibling parent "Joshua Niehus" <jm.niehus gmail.com> writes:
yes
Apr 12 2013
prev sibling next sibling parent Mike Wey <mike-wey example.com> writes:
Yes.

-- 
Mike Wey
Apr 12 2013
prev sibling next sibling parent "Matej Nanut" <matejnanut gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.
Yes.
Apr 13 2013
prev sibling next sibling parent Martin Nowak <code dawg.eu> writes:
YES
Apr 13 2013
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Friday, 12 April 2013 at 04:46:52 UTC, Jesse Phillips wrote:
 It is that time, If you would like to see the proposed 
 std.process include into Phobos please vote yes. If one 
 condition must be met specify under what condition, otherwise 
 vote no.
Yes
Apr 14 2013
prev sibling parent "Jesse Phillips" <Jessekphillips+d gmail.com> writes:
Yes
Apr 16 2013