www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Was: Re: Vote for std.process

reply "Regan Heath" <regan netmail.co.nz> writes:
I've moved this to another thread to allay complaints.

 "Vladimir Panteleev" <vladimir thecybershadow.net>
 On Friday, 12 April 2013 at 10:14:35 UTC, Regan Heath wrote:
 All true.  However, complexity can and should be packaged insuch a way  
 as to localise, and this localised complex codeshould be tested to  
 death and maintained by someone whounderstands it.  It should be  
 bracketed by sufficient commentsand warnings about how/why it does what  
 it does.  The resultingpackaged complexity, with it's associated cost  
 can be re-usedmany times over for all the benefit it gives.

 Stack allocating the environment variables need not be alocalised  
 improvement but could be a standard library functionwhich can be  
 reused, for example.

Performant abstractions. Like the much-awaited allocator design? Either way, they can only diminuate, not remove the costs.

non localised improvements have a fixed cost and ever increasing benefit - is the point I was making here. Agreed, you cannot remove the cost, and in fact well written reusable code often carries a slightly higher cost by it's very nature.
 Would you still say that the above costs are worth the 
 nearly-intangible gain?

"nearly-intangible" is wrong. Library code is code which isused by (hopefully) millions of people, writing millions ofapplications, running for millions of hours, on millions ofsystems, creating thousands of processes, etc.. In short, alittle effort now pays massive dividends over it's lifetime. So, yes, IMO the costs shown above are worth the resultinggains. D is constantly being compared to other languages on the basisof performance, so it's clearly an important aspect of D'ssuccess. Library code needs to work first time and work well or peoplewill roll their own wasting time, energy and in many casesgetting some aspect of it just plain wrong.

But once again, you speak in vague terms.

The initial point was a vague one, not a specific one. Manu wasn't attempting to block std.process, he had a general concern which I share.
 Consider the following hypothetical decisions and outcomes:

 1. std.process is left at is. One user is angry / turned awaybecause it  
 performs 0.1% slower than it can be.

It very much matters *who* that 1 user is. And, the count may be higher, and we might never "hear" from these people as they find other solutions. We're lucky that some people who try D and have issues tell us about them, they may be 5% of the total for all we know.
 2. std.process is rewritten to minimize allocations. Codecomplexity goes  
 up, new improvements are challenging to add; bugspop up and go unfixed  
 for a while because fewer programmers arequalified or willing to commit  
 the effort of making correctfixes. More people are angry / turned away  
 from D because itsstandard library is buggy.

 Of course, the above is an exaggerated illustration.

Indeed, you've downplayed point #1 and exaggerated point #2. In reality the suggested improvements would add only very minor complexity and prevent none of the current crop of contributors from working with/on std.process.
 But would optimizing all code left and right really make more D users 
 happier?

Yes, as well as the users of their applications. True, none of them will even realise they could have been less happy, so none of them will realise the effort that went into it, but all of them will be better off.
 There's also the question of priorities. Would you rather thaneffort is  
 spent on optimizing std.process (and dealing with allthe fallout from  
 any such optimizations), or working on somethingthat is acutely missing  
 and hurting D?

Add the missing items, without a doubt - which is why no-one is suggesting blocking std.process over this issue.
 D is a systems programming language, there is hope that it will 
 penetrate a wide range of systems and environments - sure inmany cases  
 a little bit of memory use or performance loss isunimportant, but for  
 many it will be the decisive factor whichmakes D unusable there.

This is surely an exaggeration.

Why? There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms. Manu is using D for games development on modern high-end gaming PCs and he is still concerned with memory and performance. So, there's 2 very different cases where memory and performance are still a concern, and .. if they become too much of a concern another solution will have to be sought.. and that's bad news for D.
 D does not attempt to please everyone out there who is choosing a 
 programming language for their next project. There is no suchlanguage,  
 nor can one exist. One has to accept that D has anumber of goals, none  
 of which are absolute, but merely pointtowards a certain, but not overly  
 specific, point in themultidimensional matrix of trade-offs. D never was  
 aboutachieving maximum performance in all possible cases.

All true, but performance is one of D's top draw cards: <quote>The D programming language. Modern convenience. Modeling power. Native **efficiency**.</quote> (**emphasis mine**) So, it behoves us to make sure the standard library keeps that in mind. R
Apr 12 2013
next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
 I've moved this to another thread to allay complaints.

Thanks! I completely agree that if code can be made more performant without a significant increase in complexity, then we should do so. While it is mostly (but not entirely) irrelevant in the context of std.process, it is a problem that should be tackled in Phobos as a whole. Several things could/should be done: It would be nice to have some sugar on top of alloca(), the use of which is usually considered bad practice. Someone (bearophile?) once suggested static arrays whose length is determined at runtime, which would be a great addition to the language: void foo(int n) { int[n] myArr; ... } Furthermore, we need to get the allocator design in place. In SciD, I use David Simcha's region allocator to allocate temporary workspace, and it works really well. The only times I use 'new' is when I need persistent memory (e.g. for a return value) and the user-supplied buffer is too small. Phobos would greatly benefit from doing this too. Finally, an example from the new std.process which got some heavy criticism in the other thread: envz[pos++] = (var~'='~val~'\0').ptr; I have been operating under the assumption that the compiler is smart enough to make the above a single allocation. If it isn't, I would consider it a compiler issue. That said, I am aware that std.process could be improved in some places. Lars
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad 
wrote:
 Finally, an example from the new std.process which got some 
 heavy criticism in the other thread:

     envz[pos++] = (var~'='~val~'\0').ptr;

 I have been operating under the assumption that the compiler is 
 smart enough to make the above a single allocation.  If it 
 isn't, I would consider it a compiler issue.

Multiple chained array concatenations are performed at once, using the _d_arraycatnT function in Druntime.
Apr 12 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Friday, 12 April 2013 at 12:43:57 UTC, Vladimir Panteleev 
wrote:
 On Friday, 12 April 2013 at 12:30:09 UTC, Lars T. Kyllingstad 
 wrote:
 Finally, an example from the new std.process which got some 
 heavy criticism in the other thread:

    envz[pos++] = (var~'='~val~'\0').ptr;

 I have been operating under the assumption that the compiler 
 is smart enough to make the above a single allocation.  If it 
 isn't, I would consider it a compiler issue.

Multiple chained array concatenations are performed at once, using the _d_arraycatnT function in Druntime.

Good to know. Lars
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:
 The initial point was a vague one, not a specific one.  Manu 
 wasn't attempting to block std.process, he had a general 
 concern which I share.

OK, but so far my interpretation and replies were mostly in the context of std.process - this module being an example where performance improvements would have a very small real-life benefit. I agree that (generally speaking) improving the performance of the code in std.algorithm/array/range would be worth the effort and complexity.
 It very much matters *who* that 1 user is.  And, the count may 
 be higher, and we might never "hear" from these people as they 
 find other solutions.  We're lucky that some people who try D 
 and have issues tell us about them, they may be 5% of the total 
 for all we know.

The same applies to the other side of the argument. A buggy standard library probably leaves a worse impression than a slow standard library...
 In reality the suggested improvements would add only very minor 
 complexity and prevent none of the current crop of contributors 
 from working with/on std.process.

Well, how do you qualify the amount of optimization that is appropriate? For example, the code in std.process would be even faster, if it was completely written in assembler. I hope we'll agree than in practice, this would be absurd. Now, what set of well-defined arguments would conclude that rewriting it in assembler is pointless, but optimizing memory allocations is not? All three versions of std.process would perform as well as far as the end-user can perceive.
 Yes, as well as the users of their applications.  True, none of 
 them will even realise they could have been less happy, so none 
 of them will realise the effort that went into it, but all of 
 them will be better off.

Absolutely - if you ignore the costs. 100%-correct faster code is always better than 100%-correct slower code, but the costs are the counter-argument.
 Add the missing items, without a doubt - which is why no-one is 
 suggesting blocking std.process over this issue.

Blocking is one thing, but asking for faster code where it doesn't really matter - when there are areas where D could be improved at much higher gain per effort - is another.
 D is a systems programming language, there is hope that it


There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms.

OK, but once again - how does that line up with the purpose of std.process? I can see how std.algorithm can be useful in low-spec embedded/gaming systems, but std.process?
 Manu is using D for games development on modern high-end gaming 
 PCs and he is still concerned with memory and performance.

In Manu's case, every bit of performance counts in the code that runs in tight loops, e.g. for every game frame. However, does that include std.process?
 All true, but performance is one of D's top draw cards:

 <quote>The D programming language. Modern convenience. Modeling 
 power. Native **efficiency**.</quote> (**emphasis mine**)

 So, it behoves us to make sure the standard library keeps that 
 in mind.

Again, I don't (generally) disagree for the general case, however I think it pays to mind the context and perspective. When the context is std.process and the perspective is the relative cost of process creation, it seems like quite a pointless argument.
Apr 12 2013
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.net> writes:
On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:
 string[string] is used in the main API to receive environment 
 variables;
 perhaps kinda convenient, but it's impossible to supply 
 environment
 variables with loads of allocations.

Environment variables are a mapping of strings to strings. The natural way to express such a mapping in D is with a string[string]. It shouldn't be necessary to allocate an AA literal, though.
 toStringz is used liberally; alternatively, alloca() could 
 allocate the
 c-string's on the stack and zero terminate them there, passing 
 a pointer to
 the stack string to the OS functions.

It is kind of hard to use alloca() in a safe manner in D, because DMD will happily inline functions that use it. The following program will overflow the stack if compiled with -inline: void doStuff() { auto p = alloca(100); } void main() { foreach (i; 0 .. 1_000_000) doStuff(); } This is of course fixable, but until that happens, I would consider alloca() a no-go for Phobos.
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--047d7b472748d80ec804da29d696
Content-Type: text/plain; charset=UTF-8

On 12 April 2013 22:30, Lars T. Kyllingstad <public kyllingen.net> wrote:

 On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:

 I've moved this to another thread to allay complaints.

Thanks! I completely agree that if code can be made more performant without a significant increase in complexity, then we should do so. While it is mostly (but not entirely) irrelevant in the context of std.process, it is a problem that should be tackled in Phobos as a whole. Several things could/should be done: It would be nice to have some sugar on top of alloca(), the use of which is usually considered bad practice. Someone (bearophile?) once suggested static arrays whose length is determined at runtime, which would be a great addition to the language: void foo(int n) { int[n] myArr; ... }

That's beautiful! Furthermore, we need to get the allocator design in place. In SciD, I use
 David Simcha's region allocator to allocate temporary workspace, and it
 works really well.  The only times I use 'new' is when I need persistent
 memory (e.g. for a return value) and the user-supplied buffer is too small.
  Phobos would greatly benefit from doing this too.

 Finally, an example from the new std.process which got some heavy
 criticism in the other thread:

     envz[pos++] = (var~'='~val~'\0').ptr;

 I have been operating under the assumption that the compiler is smart
 enough to make the above a single allocation.  If it isn't, I would
 consider it a compiler issue.

Does it? I've not seen the compiler do that, although I'd like to think it should be possible. 1 allocation is better than 3 I guess, however, I wonder if that code could be restructured to use the stack aswell. alloca() is really underrated! I can't imagine why people don't like it. Perhaps some more helpers built around it might encourage its use? It does feel a little bit 'raw', like malloc(). It implies some annoying casts. --047d7b472748d80ec804da29d696 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On 12 April 2013 22:30, Lars T. Kyllingstad <span dir=3D"l= tr">&lt;<a href=3D"mailto:public kyllingen.net" target=3D"_blank">public ky= llingen.net</a>&gt;</span> wrote:<br><div class=3D"gmail_extra"><div class= =3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im">On Friday, 12 April 2013 a= t 11:37:14 UTC, Regan Heath wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I&#39;ve moved this to another thread to allay complaints.<br> </blockquote> <br></div> Thanks!<br> <br> I completely agree that if code can be made more performant without a signi= ficant increase in complexity, then we should do so. =C2=A0While it is most= ly (but not entirely) irrelevant in the context of std.process, it is a pro= blem that should be tackled in Phobos as a whole. =C2=A0Several things coul= d/should be done:<br> <br> It would be nice to have some sugar on top of alloca(), the use of which is= usually considered bad practice. =C2=A0Someone (bearophile?) once suggeste= d static arrays whose length is determined at runtime, which would be a gre= at addition to the language:<br> <br> =C2=A0 =C2=A0 void foo(int n)<br> =C2=A0 =C2=A0 {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 int[n] myArr;<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ...<br> =C2=A0 =C2=A0 }<br></blockquote><div><br></div><div style>That&#39;s beauti= ful!</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:= 0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Furthermore, we need to get the allocator design in place. =C2=A0In SciD, I= use David Simcha&#39;s region allocator to allocate temporary workspace, a= nd it works really well. =C2=A0The only times I use &#39;new&#39; is when I= need persistent memory (e.g. for a return value) and the user-supplied buf= fer is too small. =C2=A0Phobos would greatly benefit from doing this too.<b= r> <br> Finally, an example from the new std.process which got some heavy criticism= in the other thread:<br> <br> =C2=A0 =C2=A0 envz[pos++] =3D (var~&#39;=3D&#39;~val~&#39;\0&#39;).ptr;<br> <br> I have been operating under the assumption that the compiler is smart enoug= h to make the above a single allocation. =C2=A0If it isn&#39;t, I would con= sider it a compiler issue.<br></blockquote><div><br></div><div style>Does i= t? I&#39;ve not seen the compiler do that, although I&#39;d like to think i= t should be possible. 1 allocation is better than 3 I guess, however, I won= der if that code could be restructured to use the stack aswell.</div> <div style>alloca() is really underrated! I can&#39;t imagine why people do= n&#39;t like it. Perhaps some more helpers built around it might encourage = its use? It does feel a little bit &#39;raw&#39;, like malloc(). It implies= some annoying casts.</div> </div></div></div> --047d7b472748d80ec804da29d696--
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--14dae93a17230d91a004da2a0877
Content-Type: text/plain; charset=UTF-8

On 12 April 2013 23:08, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 11:37:14 UTC, Regan Heath wrote:

 It very much matters *who* that 1 user is.  And, the count may be higher,
 and we might never "hear" from these people as they find other solutions.
  We're lucky that some people who try D and have issues tell us about them,
 they may be 5% of the total for all we know.

The same applies to the other side of the argument. A buggy standard library probably leaves a worse impression than a slow standard library...

If allocating a string on the stack makes it buggy, then there is something really wrong. It should be no less convenient if appropriate helpers are available. With consideration to the string[string] argument, surely instances like that can be reconsidered? How is string[] going to produce more bugs than string[string]? You're being paranoid, or sensationalising the effect of simple optimisation. In reality the suggested improvements would add only very minor complexity
 and prevent none of the current crop of contributors from working with/on
 std.process.

Well, how do you qualify the amount of optimization that is appropriate?

As much is convenient without causing you to start obscuring your code? That's my personal rule. But I make it a habit to consider efficiency when designing code, I never retrofit it. I tend to choose designs that are both simple and efficient at the start. For example, the code in std.process would be even faster, if it was
 completely written in assembler. I hope we'll agree than in practice, this
 would be absurd. Now, what set of well-defined arguments would conclude
 that rewriting it in assembler is pointless, but optimizing memory
 allocations is not? All three versions of std.process would perform as well
 as far as the end-user can perceive.

Actually, it would probably be slower if hand-written in assembler. And again, speed is not my concern here, it's inconsiderate the allocation policy. Yes, as well as the users of their applications. True, none of them will
 even realise they could have been less happy, so none of them will realise
 the effort that went into it, but all of them will be better off.

Absolutely - if you ignore the costs. 100%-correct faster code is always better than 100%-correct slower code, but the costs are the counter-argument.

Can you describe the 'costs'? Add the missing items, without a doubt - which is why no-one is suggesting
 blocking std.process over this issue.

Blocking is one thing, but asking for faster code where it doesn't really matter - when there are areas where D could be improved at much higher gain per effort - is another.

I'm asking for code that doesn't needlessly allocate, as a policy/habit in phobos. D is a systems programming language, there is hope that it


There exist platforms and environments where memory and performance are concerns, if the D standard library code is not "careful" in it's use of both then it will be less suitable than C (for example) and so D will not penetrate those platforms.

OK, but once again - how does that line up with the purpose of std.process? I can see how std.algorithm can be useful in low-spec embedded/gaming systems, but std.process? Manu is using D for games development on modern high-end gaming PCs and
 he is still concerned with memory and performance.

In Manu's case, every bit of performance counts in the code that runs in tight loops, e.g. for every game frame. However, does that include std.process?

I'm interested in eliminating allocations. It's just another function that can't be called in a no-gc area. If it used the stack for its temporaries, no problem. All true, but performance is one of D's top draw cards:
 <quote>The D programming language. Modern convenience. Modeling power.
 Native **efficiency**.</quote> (**emphasis mine**)

 So, it behoves us to make sure the standard library keeps that in mind.

Again, I don't (generally) disagree for the general case, however I think it pays to mind the context and perspective. When the context is std.process and the perspective is the relative cost of process creation, it seems like quite a pointless argument.

It was the first module that appeared for consideration since the recent discussions about irresponsible GC usage. The argument applies to everything considered for acceptance into phobos. I'd like to see it applied as a systematic consideration in the future, irrespective of the module being considered. Avoiding allocation for temporaries shouldn't be hard, if some tools are missing, then that is something that needs further discussion I guess. --14dae93a17230d91a004da2a0877 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On 12 April 2013 23:08, Vladimir Panteleev <span dir=3D"lt= r">&lt;<a href=3D"mailto:vladimir thecybershadow.net" target=3D"_blank">vla= dimir thecybershadow.net</a>&gt;</span> wrote:<br><div class=3D"gmail_extra= "><div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im">On Friday, 12 April 2013 a= t 11:37:14 UTC, Regan Heath wrote:<br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It very much matters= *who* that 1 user is. =C2=A0And, the count may be higher, and we might nev= er &quot;hear&quot; from these people as they find other solutions. =C2=A0W= e&#39;re lucky that some people who try D and have issues tell us about the= m, they may be 5% of the total for all we know.<br> </blockquote></div><div class=3D"im"> <br></div> The same applies to the other side of the argument. A buggy standard librar= y probably leaves a worse impression than a slow standard library...</block= quote><div><br></div><div style>If allocating a string on the stack makes i= t buggy, then there is something really wrong. It should be no less conveni= ent if appropriate helpers are available.</div> <div style>With consideration to the string[string] argument, surely instan= ces like that can be reconsidered? How is string[] going to produce more bu= gs than string[string]?</div><div style>You&#39;re being paranoid, or sensa= tionalising the effect of simple optimisation.</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"><div class=3D"im"><blockquote= class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli= d;padding-left:1ex"> In reality the suggested improvements would add only very minor complexity = and prevent none of the current crop of contributors from working with/on s= td.process.<br> </blockquote> <br></div> Well, how do you qualify the amount of optimization that is appropriate?<br=
</blockquote><div><br></div><div style>As much is convenient without causi=

<div style>But I make it a habit to consider efficiency when designing code= , I never retrofit it. I tend to choose designs that are both simple and ef= ficient at the start.</div><div><br></div><blockquote class=3D"gmail_quote"= style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> For example, the code in std.process would be even faster, if it was comple= tely written in assembler. I hope we&#39;ll agree than in practice, this wo= uld be absurd. Now, what set of well-defined arguments would conclude that = rewriting it in assembler is pointless, but optimizing memory allocations i= s not? All three versions of std.process would perform as well as far as th= e end-user can perceive.</blockquote> <div><br></div><div style>Actually, it would probably be slower if hand-wri= tten in assembler. And again, speed is not my concern here, it&#39;s incons= iderate the allocation policy.</div><div><br></div><blockquote class=3D"gma= il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef= t:1ex"> <div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Yes, as well as the users of their applications. =C2=A0True, none of them w= ill even realise they could have been less happy, so none of them will real= ise the effort that went into it, but all of them will be better off.<br> </blockquote> <br></div> Absolutely - if you ignore the costs. 100%-correct faster code is always be= tter than 100%-correct slower code, but the costs are the counter-argument.= </blockquote><div><br></div><div style>Can you describe the &#39;costs&#39;= ?</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"><div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Add the missing items, without a doubt - which is why no-one is suggesting = blocking std.process over this issue.<br> </blockquote> <br></div> Blocking is one thing, but asking for faster code where it doesn&#39;t real= ly matter - when there are areas where D could be improved at much higher g= ain per effort - is another.<br></blockquote><div><br></div><div style> I&#39;m asking for code that doesn&#39;t needlessly allocate, as a policy/h= abit in phobos.</div><div><br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im"><blockquote class=3D"gmail= _quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:= 1ex"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> D is a systems programming language, there is hope that it<br> </blockquote></blockquote></div> Why?<div class=3D"im"><br> <br> There exist platforms and environments where memory and performance are con= cerns, if the D standard library code is not &quot;careful&quot; in it&#39;= s use of both then it will be less suitable than C (for example) and so D w= ill not penetrate those platforms.<br> </div></blockquote> <br> OK, but once again - how does that line up with the purpose of std.process?= I can see how std.algorithm can be useful in low-spec embedded/gaming syst= ems, but std.process?<div class=3D"im"><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Manu is using D for games development on modern high-end gaming PCs and he = is still concerned with memory and performance.<br> </blockquote> <br></div> In Manu&#39;s case, every bit of performance counts in the code that runs i= n tight loops, e.g. for every game frame. However, does that include std.pr= ocess?</blockquote><div><br></div><div style>I&#39;m interested in eliminat= ing allocations. It&#39;s just another function that can&#39;t be called in= a no-gc area. If it used the stack for its temporaries, no problem.</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"><div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> All true, but performance is one of D&#39;s top draw cards:<br> <br> &lt;quote&gt;The D programming language. Modern convenience. Modeling power= . Native **efficiency**.&lt;/quote&gt; (**emphasis mine**)<br> <br> So, it behoves us to make sure the standard library keeps that in mind.<br> </blockquote> <br></div> Again, I don&#39;t (generally) disagree for the general case, however I thi= nk it pays to mind the context and perspective. When the context is std.pro= cess and the perspective is the relative cost of process creation, it seems= like quite a pointless argument.<br> </blockquote></div><br></div><div class=3D"gmail_extra" style>It was the fi= rst module that appeared for consideration since the recent discussions abo= ut irresponsible GC usage. The argument applies to everything considered fo= r acceptance into phobos.</div> <div class=3D"gmail_extra" style>I&#39;d like to see it applied as a system= atic consideration in the future, irrespective of the module being consider= ed. Avoiding allocation for temporaries shouldn&#39;t be hard, if some tool= s are missing, then that is something that needs further discussion I guess= .</div> </div> --14dae93a17230d91a004da2a0877--
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e01536914db836504da2a1304
Content-Type: text/plain; charset=UTF-8

On 12 April 2013 23:21, Lars T. Kyllingstad <public kyllingen.net> wrote:

 On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:

 string[string] is used in the main API to receive environment variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.

Environment variables are a mapping of strings to strings. The natural way to express such a mapping in D is with a string[string]. It shouldn't be necessary to allocate an AA literal, though.

That's a good point, do AA's support literals that don't allocate? You can't even produce an array literal without it needlessly allocating. toStringz is used liberally; alternatively, alloca() could allocate the
 c-string's on the stack and zero terminate them there, passing a pointer
 to
 the stack string to the OS functions.

It is kind of hard to use alloca() in a safe manner in D, because DMD will happily inline functions that use it. The following program will overflow the stack if compiled with -inline: void doStuff() { auto p = alloca(100); } void main() { foreach (i; 0 .. 1_000_000) doStuff(); } This is of course fixable, but until that happens, I would consider alloca() a no-go for Phobos.

Very good point. This is a problem. Hmmm... --089e01536914db836504da2a1304 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On 12 April 2013 23:21, Lars T. Kyllingstad <span dir=3D"l= tr">&lt;<a href=3D"mailto:public kyllingen.net" target=3D"_blank">public ky= llingen.net</a>&gt;</span> wrote:<br><div class=3D"gmail_extra"><div class= =3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">On Friday, 12 April 2013 at 07:04:23 UTC, Ma= nu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> string[string] is used in the main API to receive environment variables;<br=

variables with loads of allocations.<br> </blockquote> <br> Environment variables are a mapping of strings to strings. =C2=A0The natura= l way to express such a mapping in D is with a string[string]. =C2=A0It sho= uldn&#39;t be necessary to allocate an AA literal, though.<br></blockquote>= <div> <br></div><div style>That&#39;s a good point, do AA&#39;s support literals = that don&#39;t allocate? You can&#39;t even produce an array literal withou= t it needlessly allocating.</div><div><br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> toStringz is used liberally; alternatively, alloca() could allocate the<br> c-string&#39;s on the stack and zero terminate them there, passing a pointe= r to<br> the stack string to the OS functions.<br> </blockquote> <br> It is kind of hard to use alloca() in a safe manner in D, because DMD will = happily inline functions that use it. =C2=A0The following program will over= flow the stack if compiled with -inline:<br> <br> void doStuff()<br> {<br> =C2=A0 =C2=A0 auto p =3D alloca(100);<br> }<br> <br> void main()<br> {<br> =C2=A0 =C2=A0 foreach (i; 0 .. 1_000_000) doStuff();<br> }<br> <br> This is of course fixable, but until that happens, I would consider alloca(= ) a no-go for Phobos.<br> </blockquote></div><br></div><div class=3D"gmail_extra" style>Very good poi= nt. This is a problem.</div><div class=3D"gmail_extra" style>Hmmm...</div><= /div> --089e01536914db836504da2a1304--
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 13:39:38 UTC, Manu wrote:
 If allocating a string on the stack makes it buggy, then there 
 is something
 really wrong. It should be no less convenient if appropriate 
 helpers are
 available.

Please see my reply to your other post.
 With consideration to the string[string] argument, surely 
 instances like
 that can be reconsidered? How is string[] going to produce more 
 bugs than
 string[string]?

env ~= "FOO=BAR"; This will probably not do what you want if there was already a line starting with "FOO=" in env. An array of strings is a less direct representation of the environment than a string map. Certain common operations, such as finding the value of a variable, or setting / overwriting a variable, become more difficult.
 You're being paranoid, or sensationalising the effect of simple
 optimisation.

Strong words...
 And
 again, speed is not my concern here, it's inconsiderate the 
 allocation
 policy.

 I'm interested in eliminating allocations. It's just another 
 function that
 can't be called in a no-gc area. If it used the stack for its 
 temporaries,
 no problem.

Why allocations, specifically, if not for the performance costs of allocation and garbage collection?
 Can you describe the 'costs'?

See my previous posts in today's discussions.
 As much is convenient without causing you to start obscuring 
 your code?
 That's my personal rule.
 But I make it a habit to consider efficiency when designing 
 code, I never
 retrofit it. I tend to choose designs that are both simple and 
 efficient at
 the start.

OK, so if I understand you correctly: you would like Phobos to adopt a policy of avoiding heap allocations whenever possible, and this argument applies to std.process not because doing so would result in a tangible improvement of its performance or other metric, but for the purpose of being consistent across Phobos. Assuming that the language can provide or allow implementing suitably safe abstractions for doing so without complicating the code much, I think that's a goal worth looking forward, and we have been doing so for some time (hence the pending allocator design).
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--047d7b2e0887be314204da2b2087
Content-Type: text/plain; charset=UTF-8

On 13 April 2013 00:19, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 13:39:38 UTC, Manu wrote:

 If allocating a string on the stack makes it buggy, then there is
 something
 really wrong. It should be no less convenient if appropriate helpers are
 available.

Please see my reply to your other post. With consideration to the string[string] argument, surely instances like
 that can be reconsidered? How is string[] going to produce more bugs than
 string[string]?

env ~= "FOO=BAR"; This will probably not do what you want if there was already a line starting with "FOO=" in env. An array of strings is a less direct representation of the environment than a string map. Certain common operations, such as finding the value of a variable, or setting / overwriting a variable, become more difficult.

I didn't see any attempt to index the array by key in this case. That's what an AA is for, and it's not being used here, so it's not a job for an AA. I wouldn't use env ~= "FOO=BAR"; I would use env ~= EnvVar("FOO", "BAR"); Or whatever key/value pair structure you like. You're being paranoid, or sensationalising the effect of simple
 optimisation.

Strong words...

Well it seemed appropriate. I can't understand what's so wildly complex that it would make code utterly unmaintainable, and error prone. And
 again, speed is not my concern here, it's inconsiderate the allocation
 policy.

I'm interested in eliminating allocations. It's just another function that
 can't be called in a no-gc area. If it used the stack for its temporaries,
 no problem.

Why allocations, specifically, if not for the performance costs of allocation and garbage collection?

That's one aspect, but it's also about having control over the allocation patterns of your program in general. Lots of small allocations fragment the heap, and they also push the memory barrier. I couldn't disable the GC and call into phobos functions for very long, micro-allocations of temporaries not being freed would quickly eat all the system memory. Reducing allocations is always better where possible. As much is convenient without causing you to start obscuring your code?

 But I make it a habit to consider efficiency when designing code, I never
 retrofit it. I tend to choose designs that are both simple and efficient
 at
 the start.

OK, so if I understand you correctly: you would like Phobos to adopt a policy of avoiding heap allocations whenever possible, and this argument applies to std.process not because doing so would result in a tangible improvement of its performance or other metric, but for the purpose of being consistent across Phobos. Assuming that the language can provide or allow implementing suitably safe abstractions for doing so without complicating the code much, I think that's a goal worth looking forward, and we have been doing so for some time (hence the pending allocator design).

It starts as soon as the majority agree it's important enough to enforce. Although I think a tool like: char[len] stackString; would be a super-useful tool to make this considerably more painless. Some C compilers support this. --047d7b2e0887be314204da2b2087 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On 13 April 2013 00:19, Vladimir Panteleev <span dir=3D"lt= r">&lt;<a href=3D"mailto:vladimir thecybershadow.net" target=3D"_blank">vla= dimir thecybershadow.net</a>&gt;</span> wrote:<br><div class=3D"gmail_extra= "><div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"im">On Friday, 12 April 2013 at 13:39:38 UTC= , Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> If allocating a string on the stack makes it buggy, then there is something= <br> really wrong. It should be no less convenient if appropriate helpers are<br=

</blockquote> <br></div> Please see my reply to your other post.<div class=3D"im"><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> With consideration to the string[string] argument, surely instances like<br=

r> string[string]?<br> </blockquote> <br></div> env ~=3D &quot;FOO=3DBAR&quot;;<br> <br> This will probably not do what you want if there was already a line startin= g with &quot;FOO=3D&quot; in env.<br> <br> An array of strings is a less direct representation of the environment than= a string map. Certain common operations, such as finding the value of a va= riable, or setting / overwriting a variable, become more difficult.</blockq= uote> <div><br></div><div style>I didn&#39;t see any attempt to index the array b= y key in this case. That&#39;s what an AA is for, and it&#39;s not being us= ed here, so it&#39;s not a job for an AA.</div><div style><br></div><div st= yle> I wouldn&#39;t use env ~=3D &quot;FOO=3DBAR&quot;;</div><div style>I would = use env ~=3D EnvVar(&quot;FOO&quot;, &quot;BAR&quot;);</div><div style>Or w= hatever key/value pair structure you like.</div><div><br></div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1= px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:= 1ex"> <div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> You&#39;re being paranoid, or sensationalising the effect of simple<br> optimisation.<br> </blockquote> <br></div> Strong words...</blockquote><div><br></div><div style>Well it seemed approp= riate. I can&#39;t understand what&#39;s so wildly complex that it would ma= ke code utterly unmaintainable, and error prone.</div><div><br></div><block= quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-w= idth:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding= -left:1ex"> <div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> And<br> again, speed is not my concern here, it&#39;s inconsiderate the allocation<= br> policy.<br> </blockquote> <br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);b= order-left-style:solid;padding-left:1ex"> I&#39;m interested in eliminating allocations. It&#39;s just another functi= on that<br> can&#39;t be called in a no-gc area. If it used the stack for its temporari= es,<br> no problem.<br> </blockquote> <br></div> Why allocations, specifically, if not for the performance costs of allocati= on and garbage collection?</blockquote><div><br></div><div style>That&#39;s= one aspect, but it&#39;s also about having control over the allocation pat= terns of your program in general. Lots of small allocations fragment the he= ap, and they also push the memory barrier.</div> <div style>I couldn&#39;t disable the GC and call into phobos functions=C2= =A0for very long, micro-allocations of temporaries not being freed would qu= ickly eat all the system memory.</div><div style>Reducing allocations is al= ways better where possible.<br> </div><div style><br></div><blockquote class=3D"gmail_quote" style=3D"margi= n:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204= );border-left-style:solid;padding-left:1ex"><div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex">As much is convenient without causing you to start obscuri= ng your code?<br> </blockquote></div><div class=3D"im"><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(2= 04,204,204);border-left-style:solid;padding-left:1ex"> That&#39;s my personal rule.<br> But I make it a habit to consider efficiency when designing code, I never<b= r> retrofit it. I tend to choose designs that are both simple and efficient at= <br> the start.<br> </blockquote> <br></div> OK, so if I understand you correctly: you would like Phobos to adopt a poli= cy of avoiding heap allocations whenever possible, and this argument applie= s to std.process not because doing so would result in a tangible improvemen= t of its performance or other metric, but for the purpose of being consiste= nt across Phobos. Assuming that the language can provide or allow implement= ing suitably safe abstractions for doing so without complicating the code m= uch, I think that&#39;s a goal worth looking forward, and we have been doin= g so for some time (hence the pending allocator design).<br> </blockquote></div></div><br><div style>It starts as soon as the majority a= gree it&#39;s important enough to enforce.</div><div style>Although I think= a tool like: char[len] stackString; would be a super-useful tool to make t= his considerably more painless. Some C compilers support this.</div> </div> --047d7b2e0887be314204da2b2087--
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 14:58:10 UTC, Manu wrote:
 I didn't see any attempt to index the array by key in this 
 case. That's
 what an AA is for, and it's not being used here, so it's not a 
 job for an
 AA.

Sorry, not following. Are you suggesting to use a dynamic array for creating processes, but an associative array for examining the current process's environment?
 I wouldn't use env ~= "FOO=BAR";
 I would use env ~= EnvVar("FOO", "BAR");
 Or whatever key/value pair structure you like.

OK, but I don't see how it changes your argument. Also, you mentioned string[] earlier.
 Well it seemed appropriate. I can't understand what's so wildly 
 complex
 that it would make code utterly unmaintainable, and error prone.

I did not say it would be _utterly_ unmaintainable or error prone. Just more so.
 It starts as soon as the majority agree it's important enough 
 to enforce.
 Although I think a tool like: char[len] stackString; would be a
 super-useful tool to make this considerably more painless. Some 
 C compilers
 support this.

I believe this is the feature: http://en.wikipedia.org/wiki/Variable-length_array It is part of C99. ------------------------------------------------------------- Earlier today, I wrote:
 Please rewrite some part of std.process with performance in 
 mind, and post it here for review. This way, we can analyze the 
 benefits and drawbacks based on a concrete example, instead of 
 vapor and hot air.

I've tried doing this myself, for the bit of code you brought up (constructing environment variables). Here's what I ended up with: ------------------------------------------------------------- import std.array; private struct StaticAppender(size_t SIZE, T) { T[SIZE] buffer = void; Appender!(T[]) appender; alias appender this; } /// Returns a struct containing a fixed-size buffer and an /// appender. The appender will use the buffer (which has /// SIZE elements) until it runs out of space, at which /// point it will reallocate itself on the heap. auto staticAppender(size_t SIZE, T)() { StaticAppender!(SIZE, T) result; result.appender = appender(result.buffer[]); return result; } /// Allows allocating a T[] given a size. /// Contains a fized-size buffer of T elements with SIZE /// length. When asked to allocate an array with /// length <= than SIZE, the buffer is used instead of /// the heap. struct StaticArray(size_t SIZE, T) { T[SIZE] buffer = void; T[] get(size_t size) { if (size <= SIZE) { buffer[0..size] = T.init; return buffer[0..size]; } else return new T[size]; } } // -------------------------------------------------------- void exec(const(char)*[] envz) { /+ ... +/ } void oldWay(string[string] environment) { auto envz = new const(char)*[environment.length + 1]; int pos = 0; foreach (var, val; environment) envz[pos++] = (var~'='~val~'\0').ptr; exec(envz); } void newWay(string[string] environment) { auto buf = staticAppender!(4096, char)(); StaticArray!(64, size_t) envpBuf; size_t[] envp = envpBuf.get(environment.length + 1); size_t pos; foreach (var, val; environment) { envp[pos++] = buf.data.length; buf.put(var); buf.put('='); buf.put(val); buf.put('\0'); } // Convert offsets to pointers in-place auto envz = cast(const(char)*[])envp; foreach (n; 0..pos) envz[n] += cast(size_t)buf.data.ptr; exec(envz); } ------------------------------------------------------------- As you can see, the code is quite more verbose, even with the helper types. It's no longer obvious at a glance what the code is doing. Perhaps you can come up with better abstractions?
Apr 12 2013
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 12 Apr 2013 09:42:43 -0400, Manu <turkeyman gmail.com> wrote:

 On 12 April 2013 23:21, Lars T. Kyllingstad <public kyllingen.net> wrote:

 On Friday, 12 April 2013 at 07:04:23 UTC, Manu wrote:

 string[string] is used in the main API to receive environment  
 variables;
 perhaps kinda convenient, but it's impossible to supply environment
 variables with loads of allocations.

Environment variables are a mapping of strings to strings. The natural way to express such a mapping in D is with a string[string]. It shouldn't be necessary to allocate an AA literal, though.

That's a good point, do AA's support literals that don't allocate? You can't even produce an array literal without it needlessly allocating.

No, because it would have to be COW. However, a string[string] literal that uses strings only must allocate the structure of the AA, not the strings themselves. The compiler might be able to optimize this by figuring out how much memory to allocate in order to contain the entire AA structure, then create one block that has all the data in it. It still needs a new block though, otherwise, what happens when you change an AA that was once a literal? I think the best path forward is to replace the functions with a templated one that takes a indexable type as the env pointer. Then one can optimize as much as one desires. -Steve
Apr 12 2013
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e012948020feb2d04da2c0d4c
Content-Type: text/plain; charset=UTF-8

On 13 April 2013 01:29, Vladimir Panteleev <vladimir thecybershadow.net>wrote:

 On Friday, 12 April 2013 at 14:58:10 UTC, Manu wrote:

 I didn't see any attempt to index the array by key in this case. That's
 what an AA is for, and it's not being used here, so it's not a job for an
 AA.

Sorry, not following. Are you suggesting to use a dynamic array for creating processes, but an associative array for examining the current process's environment?

I didn't see anywhere where it was possible to example the current processed environment? I only saw the mechanism for feeding additional env vars to the system command. Linear array of key/value pair struct would be fine. I wouldn't use env ~= "FOO=BAR";
 I would use env ~= EnvVar("FOO", "BAR");
 Or whatever key/value pair structure you like.

OK, but I don't see how it changes your argument. Also, you mentioned string[] earlier.

Sorry. I didn't think it through at the time. It starts as soon as the majority agree it's important enough to enforce.

 super-useful tool to make this considerably more painless. Some C
 compilers
 support this.

I believe this is the feature: http://en.wikipedia.org/wiki/**Variable-length_array<http://en.wikipedia.org/wiki/Variable-length_array> It is part of C99. ------------------------------**------------------------------**- Earlier today, I wrote: Please rewrite some part of std.process with performance in mind, and
 post it here for review. This way, we can analyze the benefits and
 drawbacks based on a concrete example, instead of vapor and hot air.

I've tried doing this myself, for the bit of code you brought up (constructing environment variables). Here's what I ended up with: ------------------------------**------------------------------**- import std.array; private struct StaticAppender(size_t SIZE, T) { T[SIZE] buffer = void; Appender!(T[]) appender; alias appender this; } /// Returns a struct containing a fixed-size buffer and an /// appender. The appender will use the buffer (which has /// SIZE elements) until it runs out of space, at which /// point it will reallocate itself on the heap. auto staticAppender(size_t SIZE, T)() { StaticAppender!(SIZE, T) result; result.appender = appender(result.buffer[]); return result; } /// Allows allocating a T[] given a size. /// Contains a fized-size buffer of T elements with SIZE /// length. When asked to allocate an array with /// length <= than SIZE, the buffer is used instead of /// the heap. struct StaticArray(size_t SIZE, T) { T[SIZE] buffer = void; T[] get(size_t size) { if (size <= SIZE) { buffer[0..size] = T.init; return buffer[0..size]; } else return new T[size]; } } // ------------------------------**-------------------------- void exec(const(char)*[] envz) { /+ ... +/ } void oldWay(string[string] environment) { auto envz = new const(char)*[environment.**length + 1]; int pos = 0; foreach (var, val; environment) envz[pos++] = (var~'='~val~'\0').ptr; exec(envz); } void newWay(string[string] environment) { auto buf = staticAppender!(4096, char)(); StaticArray!(64, size_t) envpBuf; size_t[] envp = envpBuf.get(environment.length + 1); size_t pos; foreach (var, val; environment) { envp[pos++] = buf.data.length; buf.put(var); buf.put('='); buf.put(val); buf.put('\0'); } // Convert offsets to pointers in-place auto envz = cast(const(char)*[])envp; foreach (n; 0..pos) envz[n] += cast(size_t)buf.data.ptr; exec(envz); } ------------------------------**------------------------------**- As you can see, the code is quite more verbose, even with the helper types. It's no longer obvious at a glance what the code is doing. Perhaps you can come up with better abstractions?

Beautiful! Actually, I think if you look a couple of pages below, I think you'll see something rather like that already there in the windows code. Thought not sure why you use a size_t array which you just cast to a char* array? I think you could also fold that into one pass, rather than 2. And I'm not sure about this line: envz[n] += cast(size_t)buf.data.ptr; But those helpers make the problem rather painless. I wonder if there's opportunity for improvement by having appender support the ~ operator? Might be able to jig it to use natural concat syntax rather than put()... --089e012948020feb2d04da2c0d4c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On 13 April 2013 01:29, Vladimir Panteleev <span dir=3D"lt= r">&lt;<a href=3D"mailto:vladimir thecybershadow.net" target=3D"_blank">vla= dimir thecybershadow.net</a>&gt;</span> wrote:<br><div class=3D"gmail_extra= "><div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"im">On Friday, 12 April 2013 at 14:58:10 UTC= , Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> I didn&#39;t see any attempt to index the array by key in this case. That&#= 39;s<br> what an AA is for, and it&#39;s not being used here, so it&#39;s not a job = for an<br> AA.<br> </blockquote> <br></div> Sorry, not following.<br> <br> Are you suggesting to use a dynamic array for creating processes, but an as= sociative array for examining the current process&#39;s environment?</block= quote><div><br></div><div style>I didn&#39;t see anywhere where it was poss= ible to example the current processed environment? I only saw the mechanism= for feeding additional env vars to the system command.</div> <div style>Linear array of key/value pair struct would be fine.</div><div><= br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e= x;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-styl= e:solid;padding-left:1ex"> <div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-= left-style:solid;padding-left:1ex">I wouldn&#39;t use env ~=3D &quot;FOO=3D= BAR&quot;;<br> I would use env ~=3D EnvVar(&quot;FOO&quot;, &quot;BAR&quot;);<br> Or whatever key/value pair structure you like.<br> </blockquote> <br></div> OK, but I don&#39;t see how it changes your argument. Also, you mentioned s= tring[] earlier.</blockquote><div><br></div><div style>Sorry. I didn&#39;t = think it through at the time.</div><div style><br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo= rder-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex">It starts as soon as the majority agree it&#39;s important= enough to enforce.<br> </blockquote></div><div class=3D"im"><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(2= 04,204,204);border-left-style:solid;padding-left:1ex"> Although I think a tool like: char[len] stackString; would be a<br> super-useful tool to make this considerably more painless. Some C compilers= <br> support this.<br> </blockquote> <br></div> I believe this is the feature:<br> <br> <a href=3D"http://en.wikipedia.org/wiki/Variable-length_array" target=3D"_b= lank">http://en.wikipedia.org/wiki/<u></u>Variable-length_array</a><br> <br> It is part of C99.<br> <br> ------------------------------<u></u>------------------------------<u></u>-= <br> <br> Earlier today, I wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Please rewrite some part of std.process with performance in mind, and post = it here for review. This way, we can analyze the benefits and drawbacks bas= ed on a concrete example, instead of vapor and hot air.<br> </blockquote> <br> I&#39;ve tried doing this myself, for the bit of code you brought up (const= ructing environment variables). Here&#39;s what I ended up with:<br> <br> ------------------------------<u></u>------------------------------<u></u>-= <br> <br> import std.array;<br> <br> private struct StaticAppender(size_t SIZE, T)<br> {<br> =C2=A0 =C2=A0 T[SIZE] buffer =3D void;<br> =C2=A0 =C2=A0 Appender!(T[]) appender;<br> =C2=A0 =C2=A0 alias appender this;<br> }<br> <br> /// Returns a struct containing a fixed-size buffer and an<br> /// appender. The appender will use the buffer (which has<br> /// SIZE elements) until it runs out of space, at which<br> /// point it will reallocate itself on the heap.<br> auto staticAppender(size_t SIZE, T)()<br> {<br> =C2=A0 =C2=A0 StaticAppender!(SIZE, T) result;<br> =C2=A0 =C2=A0 result.appender =3D appender(result.buffer[]);<br> =C2=A0 =C2=A0 return result;<br> }<br> <br> /// Allows allocating a T[] given a size.<br> /// Contains a fized-size buffer of T elements with SIZE<br> /// length. When asked to allocate an array with<br> /// length &lt;=3D than SIZE, the buffer is used instead of<br> /// the heap.<br> struct StaticArray(size_t SIZE, T)<br> {<br> =C2=A0 =C2=A0 T[SIZE] buffer =3D void;<br> =C2=A0 =C2=A0 T[] get(size_t size)<br> =C2=A0 =C2=A0 {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (size &lt;=3D SIZE)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 buffer[0..size] =3D T.init;<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return buffer[0..size];<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 }<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 else<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return new T[size];<br> =C2=A0 =C2=A0 }<br> }<br> <br> // ------------------------------<u></u>--------------------------<br> <br> void exec(const(char)*[] envz) { /+ ... +/ }<br> <br> void oldWay(string[string] environment)<br> {<br> =C2=A0 =C2=A0 auto envz =3D new const(char)*[environment.<u></u>length + 1]= ;<br> =C2=A0 =C2=A0 int pos =3D 0;<br> =C2=A0 =C2=A0 foreach (var, val; environment)<div class=3D"im"><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 envz[pos++] =3D (var~&#39;=3D&#39;~val~&#39;\0&= #39;).ptr;<br> <br></div> =C2=A0 =C2=A0 exec(envz);<br> }<br> <br> void newWay(string[string] environment)<br> {<br> =C2=A0 =C2=A0 auto buf =3D staticAppender!(4096, char)();<br> =C2=A0 =C2=A0 StaticArray!(64, size_t) envpBuf;<br> =C2=A0 =C2=A0 size_t[] envp =3D envpBuf.get(environment.length + 1);<br> <br> =C2=A0 =C2=A0 size_t pos;<br> =C2=A0 =C2=A0 foreach (var, val; environment)<br> =C2=A0 =C2=A0 {<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 envp[pos++] =3D buf.data.length;<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 buf.put(var);<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 buf.put(&#39;=3D&#39;);<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 buf.put(val);<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 buf.put(&#39;\0&#39;);<br> =C2=A0 =C2=A0 }<br> <br> =C2=A0 =C2=A0 // Convert offsets to pointers in-place<br> =C2=A0 =C2=A0 auto envz =3D cast(const(char)*[])envp;<br> =C2=A0 =C2=A0 foreach (n; 0..pos)<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 envz[n] +=3D cast(size_t)buf.data.ptr;<br> <br> =C2=A0 =C2=A0 exec(envz);<br> }<br> <br> ------------------------------<u></u>------------------------------<u></u>-= <br> <br> As you can see, the code is quite more verbose, even with the helper types.= It&#39;s no longer obvious at a glance what the code is doing. Perhaps you= can come up with better abstractions?<br> </blockquote></div><br></div><div class=3D"gmail_extra" style>Beautiful! Ac= tually, I think if you look a couple of pages below, I think you&#39;ll see= something rather like that already there in the windows code.</div><div cl= ass=3D"gmail_extra" style> <br></div><div class=3D"gmail_extra" style>Thought not sure why you use a s= ize_t array which you just cast to a char* array?</div><div class=3D"gmail_= extra" style>I think you could also fold that into one pass, rather than 2.= </div> <div class=3D"gmail_extra" style>And I&#39;m not sure about this line: envz= [n] +=3D cast(size_t)buf.data.ptr;</div><div class=3D"gmail_extra" style><b= r></div><div class=3D"gmail_extra" style>But those helpers make the problem= rather painless.</div> <div class=3D"gmail_extra" style>I wonder if there&#39;s opportunity for im= provement by having appender support the ~ operator? Might be able to jig i= t to use natural concat syntax rather than put()...</div></div> --089e012948020feb2d04da2c0d4c--
Apr 12 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 12 April 2013 at 16:04:09 UTC, Manu wrote:
 I didn't see anywhere where it was possible to example the 
 current
 processed environment? I only saw the mechanism for feeding 
 additional env
 vars to the system command.
 Linear array of key/value pair struct would be fine.

There's the "environment" object. It generally acts like a string[string], and has a toAA() method that constructs a real string[string].
 Beautiful! Actually, I think if you look a couple of pages 
 below, I think
 you'll see something rather like that already there in the 
 windows code.

I see, it also uses appender, although appender's buffer will be on the heap, and there is no need for an envz array.
 Thought not sure why you use a size_t array which you just cast 
 to a char*
 array?
 I think you could also fold that into one pass, rather than 2.
 And I'm not sure about this line: envz[n] += 
 cast(size_t)buf.data.ptr;

The problem is that the pointer to the data may change once appender reallocates the buffer when it reaches the current buffer's capacity. For this reason, we can't store pointers to the strings we store, since they can "move" around until the point that we're done appending. I think this is the most common gotcha when writing / working with appenders, and it bit be once too. As you can see, the code is not completely obvious ;)
 But those helpers make the problem rather painless.
 I wonder if there's opportunity for improvement by having 
 appender support
 the ~ operator? Might be able to jig it to use natural concat 
 syntax rather
 than put()...

Allowing put() take multiple arguments would be an improvement as well - not just in usability, but performance as well, since it would only need to check for overflow once for all arguments. I have this in my own appender: https://github.com/CyberShadow/ae/blob/master/utils/appender.d Rob Jacques was working on a Phobos appender replacement which also had this, I believe: http://d.puremagic.com/issues/show_bug.cgi?id=5813 Too bad nothing came out of the latter.
Apr 12 2013
prev sibling next sibling parent "Mr. Anonymous" <mailnew4ster gmail.com> writes:
 void oldWay(string[string] environment)
 {
     auto envz = new const(char)*[environment.length + 1];
     int pos = 0;
     foreach (var, val; environment)
         envz[pos++] = (var~'='~val~'\0').ptr;

     exec(envz);
 }

 void newWay(string[string] environment)
 {
     auto buf = staticAppender!(4096, char)();
     StaticArray!(64, size_t) envpBuf;
     size_t[] envp = envpBuf.get(environment.length + 1);

     size_t pos;
     foreach (var, val; environment)
     {
         envp[pos++] = buf.data.length;
         buf.put(var);
         buf.put('=');
         buf.put(val);
         buf.put('\0');
     }

     // Convert offsets to pointers in-place
     auto envz = cast(const(char)*[])envp;
     foreach (n; 0..pos)
         envz[n] += cast(size_t)buf.data.ptr;

     exec(envz);
 }

I just thought of something! I don't comment here often, but I want to express my opinion: Currently D, being a system language, offers full control over the allocation method, be it the stack or the heap. The helpers above show how flexible it is in doing custom optimized stuff if one wants to. But there's an obvious drawback, quoting Vladimir: "the code is quite more verbose, even with the helper types. It's no longer obvious at a glance what the code is doing." But think for a moment - does the programmer usually needs to choose the allocation method? Why explicit is default? It could be the other way around: void oldWay(string[string] environment) { const(char)* envz[environment.length + 1]; // Compiler decides whether to use stack or heap. int pos = 0; foreach (var, val; environment) envz[pos++] = (var~'='~val~'\0').ptr; // Use appender. Use stack and switch to heap if necessary exec(envz); } How nifty would that be, don't you think? Another benefit could be that the compiler could adjust the stack allocation limit per architecture, and probably it could be defined as a command line parameter, e.g. when targeting a low end device. This principle could work on fields other than allocation, e.g. parameter passing by ref/value. The programmer needs to only specify whether he wants a copy or the actual value, and the compiler would decide the optimal way to pass it. e.g.: immutable int n; func(n); // by value immutable int arr[30]; func(arr); // by ref But these are probably suggestions for D3 or something... Too drastic :)
Apr 12 2013
prev sibling next sibling parent "Mr. Anonymous" <mailnew4ster gmail.com> writes:
After some Googling, I found out a similar technique already 
exists: auto_buffer in C++.
http://goo.gl/3RLK6

I think it would be perfect to make it the default allocation 
method in D.
Apr 13 2013
prev sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
 Someone (bearophile?) once suggested static arrays whose length is determined
at runtime, which would  > be a great addition to the language:
    void foo(int n)    {        int[n] myArr;    }

That would be great. Question about the type: it won't be int[n] as n is runtime not compile time. would it be int[] or some other type? pros of some other type: makes some optimizations that benefit from the fact it's stack allocated possible cons: we need to be able to reuse existing algos that don't make such distinction. maybe another type that would satisfy both type traits: isDynamicArray!T isAllocaArray!T On Sat, Apr 13, 2013 at 4:19 AM, Mr. Anonymous <mailnew4ster gmail.com> wrote:
 After some Googling, I found out a similar technique already exists:
 auto_buffer in C++.
 http://goo.gl/3RLK6

 I think it would be perfect to make it the default allocation method in D.

Apr 13 2013