www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Possible change to array runtime?

reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
Buried inside another large thread, Don and Dicebot complained about D2's  
prevention of array stomping as being the biggest breaking change for D2,  
not that it breaks code, but silently destroys performance.

A little background for people who aren't aware:

1. D1 took the position that if an array slice started at the beginning of  
a block, and you appended to that slice, it would extend in-block no  
matter what else was referencing that data.

In other words:

char[] arr = "abc"; // this was ok in D1, which did not have the immutable  
keyword
char[] arr2 = arr[0..1];
arr2 ~= "de";

assert(arr == "ade"); // stomped!

This was a "feature", not a bug, because the idea is, once you have built  
a large or appropriately sized buffer, you may want to forget what it had,  
and re-use it by doing:

arr.length = 0;

2. D2 added a mechanism to prevent stomping, especially in the case of  
immutable, by storing the "used" length of a block inside the block itself.

This was done by yours truly, and at the same time, I also improved  
performance of array appending by exploiting the fact that D2 has  
thread-local storage by default to avoid a global GC lock.

However, any code that is compiled from D1, the following line still works  
fine:

arr.length = 0;

But has a vastly different meaning. Instead of re-using the block, it  
*abandons* the block, and allocates a new one. This leaves the original  
block as garbage, especially if nothing is left pointing at the original  
block.

3. Don's company uses D1 as its language, I highly recommend watching  
Don's Dconf13 presentation (and look forward to his Dconf14 one!) to see  
how effective D code can create unbelievable speed, especially where array  
slices are concerned. But to the above line, in D2, they must add the  
following code to get the same behavior:

arr.assumeSafeAppend();

This is an "unsafe" operation which says, "I don't care that there is  
still data in there, that other slices may refer to, just throw it away."

And in fact, I believe the performance would improve over D1 if they did  
that. But at the moment, just "switching to D2" will cause unacceptable  
performance loss.

However, there may be a way to keep the original behavior in the right  
circumstances. It's a tad bit hacky, but makes a lot of sense. So here  
goes:

Proposal: iff you are setting the length of an array via the length  
property AND you are setting the length to 0 AND the array starts at the  
beginning of the block AND the array is mutable (not const or immutable),  
THEN assumeSafeAppend is called on that array AUTOMATICALLY.

rationale for this not affecting existing code:

If you mean to abandon an array, left to other references, are you more  
likely to use arr = null, or arr.length = 0? The former disavows all  
claims to the array, allows it to be garbage collected if no other  
references exist, and is shorter to type. arr.length = 0 retains the  
pointer to the array, which keeps it in memory, but it has no access to  
the existing data. The only valid operation on that array at that point is  
to append, which means you are going to reallocate anyway, there was no  
reason to keep a pointer at the original array.

I think in vast majority of cases, where one intends the behavior to  
reallocate, they would use arr = null. In the other cases, I would argue,  
they don't know what they are doing, they incorrectly think it's going to  
take over the array again. In that case, they likely don't care if the  
existing data gets stomped.

It does not disallow the original operation, you can do the same thing  
with arr = arr[0..0];

The reason we disallow it on const or immutable is simply to prevent  
implicit overwriting of immutable data. In cases where you have a buffer  
you are building, and especially in cases of D1 code porting to D2, const  
and immutable would not exist.

I think this would "fix" the problem Sociomantic has, and not break any  
existing code that wasn't already broken.

The single largest drawback that I can think of is that the behavior would  
be different in a very small number of cases. Are those cases valid? Is  
this enough to reject the proposal?

Destroy.

-Steve
Mar 13 2014
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 15:24:01 UTC, Steven Schveighoffer 
wrote:
 Destroy.

 -Steve
Please keep in mind that if the objects stored are RAII, then if/when we will have a finalizing GC, the stomped elements will have been leaked. Clobbering elements is more than just "I won't use these elements anymore", it's "I won't use them, and they are safe to be discarded of right now". In know that's a big "if", but it could happen. If we go the way of your proposal, we are definitively closing that door. -------- Semi-on topic, it would greatly help if assumeSafeAppend was nothrow. As of right now, there are a lot of places in phobos where it could be used, but isn't, because of this. In particular, assumeSafeAppend can be used to both shrink an array, *or* grow an array up-to capacity. Its greater use in phobos would help give it more visibility and exposure.
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:

 On Thursday, 13 March 2014 at 15:24:01 UTC, Steven Schveighoffer wrote:
 Destroy.

 -Steve
Please keep in mind that if the objects stored are RAII, then if/when we will have a finalizing GC, the stomped elements will have been leaked. Clobbering elements is more than just "I won't use these elements anymore", it's "I won't use them, and they are safe to be discarded of right now". In know that's a big "if", but it could happen. If we go the way of your proposal, we are definitively closing that door.
I'm not understanding this. Can you explain further/give example?
 Semi-on topic, it would greatly help if assumeSafeAppend was nothrow. As  
 of right now, there are a lot of places in phobos where it could be  
 used, but isn't, because of this. In particular, assumeSafeAppend can be  
 used to both shrink an array, *or* grow an array up-to capacity.

 Its greater use in phobos would help give it more visibility and  
 exposure.
https://github.com/D-Programming-Language/druntime/pull/147 reserve and capacity were made nothrow, not sure why assumeSafeAppend shouldn't also be. -Steve
Mar 13 2014
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer 
wrote:
 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:
 Please keep in mind that if the objects stored are RAII, then 
 if/when we will have a finalizing GC, the stomped elements 
 will have been leaked.

 Clobbering elements is more than just "I won't use these 
 elements anymore", it's "I won't use them, and they are safe 
 to be discarded of right now".

 In know that's a big "if", but it could happen. If we go the 
 way of your proposal, we are definitively closing that door.
I'm not understanding this. Can you explain further/give example?
Well, image "File": It's a struct that owns a "file handle", and when the struct is destroyed, it releases/closes the file. Basic RAII. Now, imagine we want to open several files. We'd use: File[] files; "As of today", this does not end well, as the GC does not finalize the array elements, and the file handles are leaked. We could hope, that one day, the GC *will* finalize the array elements. However, with your proposal, imagine this code: File[] files; files ~= File("foo"); //Opens file "foo" files.length = 0; files ~= File("bar"); //Clobbers "foo" With this setup, the File handling "foo" gets clobbered, ruining any chance of releasing it ever. The "only way" to make it work (AFAIK), would be for "length = 0" to first finalize the elements in the array. However, you do that, you may accidentally destroy elements that are still "live" and referenced by another array. The same example would work with RefCounted or whatever. I'm not too hot about this proposal. My main gripe is that while "length = 0" *may* mean "*I* want to discard this data", there is no guarantee you don't have someone else that has a handle on said data, and sure as hell doesn't want it clobbered. For what it's worth, I think the problem would go away all by itself if "assumeSafeAppend" had more exposition, and was actually used. I think a simpler solution would be to simply educate users of this function, and promote its use. Its simpler than adding a special case language change.
 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why 
 assumeSafeAppend shouldn't also be.

 -Steve
The irony is that reserve can't actually be tagged pure nor nothrow: Since it can cause relocation, it can call postblit, which in turn may actually be impure or throw. assumeSafeAppend, on the other hand, is *certifiably* pure and nothrow, since it never actually touches any data. I had opened a pull to fix this, but it was never merged, due to the back and forth "fiasco" of tagging said reserve. Since I'm not fluent in D-runtime, I just let the issue drop.
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:

 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:
 Please keep in mind that if the objects stored are RAII, then if/when  
 we will have a finalizing GC, the stomped elements will have been  
 leaked.

 Clobbering elements is more than just "I won't use these elements  
 anymore", it's "I won't use them, and they are safe to be discarded of  
 right now".

 In know that's a big "if", but it could happen. If we go the way of  
 your proposal, we are definitively closing that door.
I'm not understanding this. Can you explain further/give example?
Well, image "File": It's a struct that owns a "file handle", and when the struct is destroyed, it releases/closes the file. Basic RAII. Now, imagine we want to open several files. We'd use: File[] files; "As of today", this does not end well, as the GC does not finalize the array elements, and the file handles are leaked. We could hope, that one day, the GC *will* finalize the array elements. However, with your proposal, imagine this code: File[] files; files ~= File("foo"); //Opens file "foo" files.length = 0; files ~= File("bar"); //Clobbers "foo" With this setup, the File handling "foo" gets clobbered, ruining any chance of releasing it ever.
Line 3 should be files = null. There is no point to setting length to 0, and mostly this is a relic from D1 code. That was my basis of why it shouldn't cause problems.
 The "only way" to make it work (AFAIK), would be for "length = 0" to  
 first finalize the elements in the array. However, you do that, you may  
 accidentally destroy elements that are still "live" and referenced by  
 another array.
In fact, assumeSafeAppend *should* finalize the elements in the array, if it had that capability. When you call assumeSafeAppend, you are telling the runtime that you are done with the extra elements.
 I'm not too hot about this proposal. My main gripe is that while "length  
 = 0" *may* mean "*I* want to discard this data", there is no guarantee  
 you don't have someone else that has a handle on said data, and sure as  
 hell doesn't want it clobbered.
Again, the =null is a better solution. There is no point to keeping the same block in reference, but without any access to elements, unless you want to overwrite it. The problem is that we have this new mechanism that keeps those intact. If I could have imagined this outcome and was aware of this logic, I would have kept the length = 0 mechanics from D1 to begin with.
 For what it's worth, I think the problem would go away all by itself if  
 "assumeSafeAppend" had more exposition, and was actually used. I think a  
 simpler solution would be to simply educate users of this function, and  
 promote its use. Its simpler than adding a special case language change.
The issue I'm examining is that people are reluctant to move off of D1, because their D1 code behaves well when they do length = 0, and it behaves badly in D2, even though it compiles and works correctly. They do not have assumeSafeAppend in D1. I'm unsure why they have not used it, either out of ignorance of the function or they have decided it's not worth the effort, I have no idea.
 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why assumeSafeAppend  
 shouldn't also be.
The irony is that reserve can't actually be tagged pure nor nothrow: Since it can cause relocation, it can call postblit, which in turn may actually be impure or throw. assumeSafeAppend, on the other hand, is *certifiably* pure and nothrow, since it never actually touches any data.
It does touch data, but it should be pure and nothrow.
 I had opened a pull to fix this, but it was never merged, due to the  
 back and forth "fiasco" of tagging said reserve. Since I'm not fluent in  
 D-runtime, I just let the issue drop.
Which PR? -Steve
Mar 13 2014
next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer 
wrote:
 I had opened a pull to fix this, but it was never merged, due 
 to the back and forth "fiasco" of tagging said reserve. Since 
 I'm not fluent in D-runtime, I just let the issue drop.
Which PR? -Steve
This first one was: https://github.com/D-Programming-Language/druntime/pull/553 It "simply" marked it as nothrow, pure, but was turned down, as it called "throwing" code. The second on is: https://github.com/D-Programming-Language/druntime/pull/632 It shuffles code around to "prove" the call is actually nothrow, but IMO, is not worth it. I believe in the first pull more than the second, which I believe is more "proof". If you are willing to review, I can re-open the first.
Mar 13 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 14:54:12 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:

 On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer wrote:
 I had opened a pull to fix this, but it was never merged, due to the  
 back and forth "fiasco" of tagging said reserve. Since I'm not fluent  
 in D-runtime, I just let the issue drop.
Which PR? -Steve
This first one was: https://github.com/D-Programming-Language/druntime/pull/553 It "simply" marked it as nothrow, pure, but was turned down, as it called "throwing" code.
Please reopen/update this one, I will help convince Martin. -Steve
Mar 13 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer 
wrote:
 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:

 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven 
 Schveighoffer wrote:
 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra 
 <monarchdodra gmail.com> wrote:
 Please keep in mind that if the objects stored are RAII, 
 then if/when we will have a finalizing GC, the stomped 
 elements will have been leaked.

 Clobbering elements is more than just "I won't use these 
 elements anymore", it's "I won't use them, and they are safe 
 to be discarded of right now".

 In know that's a big "if", but it could happen. If we go the 
 way of your proposal, we are definitively closing that door.
I'm not understanding this. Can you explain further/give example?
Well, image "File": It's a struct that owns a "file handle", and when the struct is destroyed, it releases/closes the file. Basic RAII. Now, imagine we want to open several files. We'd use: File[] files; "As of today", this does not end well, as the GC does not finalize the array elements, and the file handles are leaked. We could hope, that one day, the GC *will* finalize the array elements. However, with your proposal, imagine this code: File[] files; files ~= File("foo"); //Opens file "foo" files.length = 0; files ~= File("bar"); //Clobbers "foo" With this setup, the File handling "foo" gets clobbered, ruining any chance of releasing it ever.
Line 3 should be files = null. There is no point to setting length to 0, and mostly this is a relic from D1 code. That was my basis of why it shouldn't cause problems.
well... "should"/"could". The problem (IMO) is taking perfectly valid code, and making a subtle and silent change to it, changing its behavior and potentially breaking it. It's really the most pernicious kind of change.
 The "only way" to make it work (AFAIK), would be for "length = 
 0" to first finalize the elements in the array. However, you 
 do that, you may accidentally destroy elements that are still 
 "live" and referenced by another array.
In fact, assumeSafeAppend *should* finalize the elements in the array, if it had that capability. When you call assumeSafeAppend, you are telling the runtime that you are done with the extra elements.
Good point. Very very good point. As a matter of fact, this could be implemented right now, couldn't it?
 I'm not too hot about this proposal. My main gripe is that 
 while "length = 0" *may* mean "*I* want to discard this data", 
 there is no guarantee you don't have someone else that has a 
 handle on said data, and sure as hell doesn't want it 
 clobbered.
Again, the =null is a better solution. There is no point to keeping the same block in reference, but without any access to elements, unless you want to overwrite it. The problem is that we have this new mechanism that keeps those intact. If I could have imagined this outcome and was aware of this logic, I would have kept the length = 0 mechanics from D1 to begin with.
I don't like the fact that this can only be implemented in the compiler. Because we *are* talking about "literal" 0, right? Run-time 0 wouldn't have this behavior? By implementing this, then no user defined array wrapper would be able to emulate this " = 0" special behavior. By that same token, I don't see why "0" would get such special treatment, when I'm certain you'd find just as many instance of "length = 1;", which means to say "keep the first element, and start clobering from there".
 For what it's worth, I think the problem would go away all by 
 itself if "assumeSafeAppend" had more exposition, and was 
 actually used. I think a simpler solution would be to simply 
 educate users of this function, and promote its use. Its 
 simpler than adding a special case language change.
The issue I'm examining is that people are reluctant to move off of D1, because their D1 code behaves well when they do length = 0, and it behaves badly in D2, even though it compiles and works correctly. They do not have assumeSafeAppend in D1. I'm unsure why they have not used it, either out of ignorance of the function or they have decided it's not worth the effort, I have no idea.
Well... "s/some people/sociomantic/". How hard would it be to add it to D1, to help the migration process? I know it's "closed", but there are always exceptions. Honestly, I'm 100% fine with braking "compile-time" changes. Behavioral changes on the other hand...
 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why 
 assumeSafeAppend shouldn't also be.
The irony is that reserve can't actually be tagged pure nor nothrow: Since it can cause relocation, it can call postblit, which in turn may actually be impure or throw. assumeSafeAppend, on the other hand, is *certifiably* pure and nothrow, since it never actually touches any data.
It does touch data, but it should be pure and nothrow.
Right. *BUT*, it is not safe. This means that any code that uses "length = 0;" becomes unsafe. At least I *think* it's unsafe, there is always a think line between "get my hands dirty under the hood" and "memory safety". I'm pretty sure that with creative use of assumeSafeAppend, you could do illegal memory access.
Mar 13 2014
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 15:06:47 -0400, monarch_dodra <monarchdodra gmail.com>  
wrote:

 On Thursday, 13 March 2014 at 18:09:55 UTC, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:

 On Thursday, 13 March 2014 at 16:17:17 UTC, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 11:53:15 -0400, monarch_dodra  
 <monarchdodra gmail.com> wrote:
 Please keep in mind that if the objects stored are RAII, then  
 if/when we will have a finalizing GC, the stomped elements will have  
 been leaked.

 Clobbering elements is more than just "I won't use these elements  
 anymore", it's "I won't use them, and they are safe to be discarded  
 of right now".

 In know that's a big "if", but it could happen. If we go the way of  
 your proposal, we are definitively closing that door.
I'm not understanding this. Can you explain further/give example?
Well, image "File": It's a struct that owns a "file handle", and when the struct is destroyed, it releases/closes the file. Basic RAII. Now, imagine we want to open several files. We'd use: File[] files; "As of today", this does not end well, as the GC does not finalize the array elements, and the file handles are leaked. We could hope, that one day, the GC *will* finalize the array elements. However, with your proposal, imagine this code: File[] files; files ~= File("foo"); //Opens file "foo" files.length = 0; files ~= File("bar"); //Clobbers "foo" With this setup, the File handling "foo" gets clobbered, ruining any chance of releasing it ever.
Line 3 should be files = null. There is no point to setting length to 0, and mostly this is a relic from D1 code. That was my basis of why it shouldn't cause problems.
well... "should"/"could". The problem (IMO) is taking perfectly valid code, and making a subtle and silent change to it, changing its behavior and potentially breaking it. It's really the most pernicious kind of change.
Yes, it's perfectly valid. We could go through a deprecation process, but I don't think it's worth it. I have also proposed a mechanism to detect these calls, which could be turned on if you think your code might be affected by this (see my response to Andrei). All in all, it's not a very good excuse for breaking anyone's code, especially in a way that potentially can corrupt data. It's the whole reason we got rid of stomping in the first place.
 The "only way" to make it work (AFAIK), would be for "length = 0" to  
 first finalize the elements in the array. However, you do that, you  
 may accidentally destroy elements that are still "live" and referenced  
 by another array.
In fact, assumeSafeAppend *should* finalize the elements in the array, if it had that capability. When you call assumeSafeAppend, you are telling the runtime that you are done with the extra elements.
Good point. Very very good point. As a matter of fact, this could be implemented right now, couldn't it?
Actually, I think it can. But, no other code in the array runtime finalizes array elements. I don't think that it would be good to make this inconsistent, even if it's the right thing to do.
 I'm not too hot about this proposal. My main gripe is that while  
 "length = 0" *may* mean "*I* want to discard this data", there is no  
 guarantee you don't have someone else that has a handle on said data,  
 and sure as hell doesn't want it clobbered.
Again, the =null is a better solution. There is no point to keeping the same block in reference, but without any access to elements, unless you want to overwrite it. The problem is that we have this new mechanism that keeps those intact. If I could have imagined this outcome and was aware of this logic, I would have kept the length = 0 mechanics from D1 to begin with.
I don't like the fact that this can only be implemented in the compiler. Because we *are* talking about "literal" 0, right? Run-time 0 wouldn't have this behavior?
Yes, runtime 0 would have this behavior, it would be in the runtime this would change.
 By that same token, I don't see why "0" would get such special  
 treatment, when I'm certain you'd find just as many instance of "length  
 = 1;", which means to say "keep the first element, and start clobering  
 from there".
Because it's nonsense to keep a reference to data that you have no access to. A reference to a 1-element array still has elements you can access, you can envision a reason to keep that without wanting to append to it. In other words, the only sane reason to shrink it to 0 but keep the reference not null is to append into the same block again. You should never see a length = 0 call without an assumeSafeAppend afterwards.
 The issue I'm examining is that people are reluctant to move off of D1,  
 because their D1 code behaves well when they do length = 0, and it  
 behaves badly in D2, even though it compiles and works correctly. They  
 do not have assumeSafeAppend in D1. I'm unsure why they have not used  
 it, either out of ignorance of the function or they have decided it's  
 not worth the effort, I have no idea.
Well... "s/some people/sociomantic/". How hard would it be to add it to D1, to help the migration process? I know it's "closed", but there are always exceptions.
This was actually asked of me, by the Tango team long ago. But they just wanted the cache and not the stomping fix. The stomping adds overhead, which makes things slower. But it's mitigated by the fact that with thread-local storage embedded into the type, we can avoid taking the GC lock. This results in a speedup. I doubt it would be worthwhile.
 Honestly, I'm 100% fine with braking "compile-time" changes. Behavioral  
 changes on the other hand...
Yes, it's not a good compromise. It rests solely on the expectation that any code which sets length to 0 should be calling assumeSafeAppend afterwards, or would expect the same functionality. However, it could possibly be the case that someone wrote that expecting the current behavior. I think realistically, as correct as the proposal is, it's not likely to be accepted.
 https://github.com/D-Programming-Language/druntime/pull/147

 reserve and capacity were made nothrow, not sure why assumeSafeAppend  
 shouldn't also be.
The irony is that reserve can't actually be tagged pure nor nothrow: Since it can cause relocation, it can call postblit, which in turn may actually be impure or throw. assumeSafeAppend, on the other hand, is *certifiably* pure and nothrow, since it never actually touches any data.
It does touch data, but it should be pure and nothrow.
Right. *BUT*, it is not safe. This means that any code that uses "length = 0;" becomes unsafe.
First, I'll qualify that proposal is for this to happen only in specific circumstances, as outlined before. But it's not actually un safe, unless you are talking about immutable data (and this proposal specifically does not do that).
 At least I *think* it's unsafe, there is always a think line between  
 "get my hands dirty under the hood" and "memory safety".

 I'm pretty sure that with creative use of assumeSafeAppend, you could do  
 illegal memory access.
Not anything that you couldn't already do with an array element change. -Steve
Mar 13 2014
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 13/03/14 19:09, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 13:44:01 -0400, monarch_dodra <monarchdodra gmail.com>
wrote:
 The "only way" to make it work (AFAIK), would be for "length = 0" to first
 finalize the elements in the array. However, you do that, you may accidentally
 destroy elements that are still "live" and referenced by another array.
In fact, assumeSafeAppend *should* finalize the elements in the array, if it had that capability. When you call assumeSafeAppend, you are telling the runtime that you are done with the extra elements.
 I'm not too hot about this proposal. My main gripe is that while "length = 0"
 *may* mean "*I* want to discard this data", there is no guarantee you don't
 have someone else that has a handle on said data, and sure as hell doesn't
 want it clobbered.
Again, the =null is a better solution. There is no point to keeping the same block in reference, but without any access to elements, unless you want to overwrite it.
Problem is, this still seems like safety-by-programmer-virtue. It's far too easy to write ".length = 0" casually and without any attention to consequences one way or the other. Isn't the whole point of the requirement to use assumeSafeAppend that it really forces the user to say, "Yes, I want to do away with the contents of the array and I take full responsibility for ensuring there's nothing else pointing to it that will break" ... ? I must say that personally I'd rather have the safety-by-default and the obligation to write assumeSafeAppend (or use Appender!T) where performance needs it, than risk code breaking because someone's function accidentally throws away stuff that I still had a slice of.
Mar 16 2014
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Sunday, March 16, 2014 13:14:15 Joseph Rushton Wakeling wrote:
 Problem is, this still seems like safety-by-programmer-virtue. It's far too
 easy to write ".length = 0" casually and without any attention to
 consequences one way or the other.
 
 Isn't the whole point of the requirement to use assumeSafeAppend that it
 really forces the user to say, "Yes, I want to do away with the contents of
 the array and I take full responsibility for ensuring there's nothing else
 pointing to it that will break" ... ?
 
 I must say that personally I'd rather have the safety-by-default and the
 obligation to write assumeSafeAppend (or use Appender!T) where performance
 needs it, than risk code breaking because someone's function accidentally
 throws away stuff that I still had a slice of.
I tend to agree with this. Setting an array's length to 0 with the expectation that you will then reuse that memory is inherently unsafe. What if there are other arrays still referring to the same data? They'll be stomped, which could do some very nasty things - especially if we're talking about structs rather than strings. assumeSafeAppend is explicitly unsafe and makes it clear what you're doing, whereas while setting an array's length to 0 may be generally nonsensical if you're not intending to reuse the memory again, having that essentially call assumeSafeAppend for you could result in very pernicious bugs when someone is foolish enough to set an array's length to 0 when they still have other slices to some or all of that array. I really think that the assumeSafeAppend needs to be explicit. - Jonathan M Davis
Mar 19 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/13/14, 8:24 AM, Steven Schveighoffer wrote:
 Proposal: iff you are setting the length of an array via the length
 property AND you are setting the length to 0 AND the array starts at the
 beginning of the block AND the array is mutable (not const or
 immutable), THEN assumeSafeAppend is called on that array AUTOMATICALLY.
I think this is a sensible proposal, and well argued. However it may break code (even code that arguably wasn't really nicely thought through), and in subtle ways. "We upgraded the compiler, and our 50KLOC app now produces invalid results." Very hard to get working on that, in particular since it's unclear whether this particular matter was the actual culprit. I think there's a much simpler approach that could help. Define and document a function clearForAppend that resets the length to 0 and puts the array in gear for appending. Maybe an optional parameter indicates a minimum desired capacity. Then people who want to migrate from D1 to D2 or generally consider using this idiom can grep their source code for "length *= *0" and make the change on their own pace and with appropriate testing. Andrei
Mar 13 2014
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 12:29:59 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 3/13/14, 8:24 AM, Steven Schveighoffer wrote:
 Proposal: iff you are setting the length of an array via the length
 property AND you are setting the length to 0 AND the array starts at the
 beginning of the block AND the array is mutable (not const or
 immutable), THEN assumeSafeAppend is called on that array AUTOMATICALLY.
I think this is a sensible proposal, and well argued. However it may break code (even code that arguably wasn't really nicely thought through), and in subtle ways. "We upgraded the compiler, and our 50KLOC app now produces invalid results." Very hard to get working on that, in particular since it's unclear whether this particular matter was the actual culprit.
A good point that the error that would occur, if you depended on setting length = 0 to not overwrite your mutable array, would be corruption, and that is not a good result. Even if your code was foolish to begin with. As a mechanism to check this, we could throw an exception when appending to an array that has been set to length = 0, but not had assumeSafeAppend called on it. This would obviously just be a bug-finding mechanism, and not for production.
 I think there's a much simpler approach that could help. Define and  
 document a function clearForAppend that resets the length to 0 and puts  
 the array in gear for appending. Maybe an optional parameter indicates a  
 minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider  
 using this idiom can grep their source code for "length *= *0" and make  
 the change on their own pace and with appropriate testing.
Arguably this is already possible within a project, such as Sociomantic's. They haven't done it yet. I don't know why. -Steve
Mar 13 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/13/14, 10:18 AM, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 12:29:59 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 I think there's a much simpler approach that could help. Define and
 document a function clearForAppend that resets the length to 0 and
 puts the array in gear for appending. Maybe an optional parameter
 indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider
 using this idiom can grep their source code for "length *= *0" and
 make the change on their own pace and with appropriate testing.
Arguably this is already possible within a project, such as Sociomantic's. They haven't done it yet. I don't know why.
Don? Andrei
Mar 13 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 13 March 2014 at 17:18:05 UTC, Steven Schveighoffer 
wrote:
 I think there's a much simpler approach that could help. 
 Define and document a function clearForAppend that resets the 
 length to 0 and puts the array in gear for appending. Maybe an 
 optional parameter indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally 
 consider using this idiom can grep their source code for 
 "length *= *0" and make the change on their own pace and with 
 appropriate testing.
Arguably this is already possible within a project, such as Sociomantic's. They haven't done it yet. I don't know why. -Steve
It has not been mentioned as example of blocker but example of most horrible breaking change that can happen if dmd user is not aware of it before upgrade. I think only practical thing that blocks D2 transition right now is that no one can afford to work on it full-time and amount of changes needed is too big to be done as a side work - array stomping is not only problematic area, I personally think const will cause much more problems. Really hope that with recent hires doing a focused porting effort will soon become a real option. D1 is so D1 :P
Mar 14 2014
next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Mar 2014 10:50:43 -0400, Dicebot <public dicebot.lv> wrote:

 On Thursday, 13 March 2014 at 17:18:05 UTC, Steven Schveighoffer wrote:
 I think there's a much simpler approach that could help. Define and  
 document a function clearForAppend that resets the length to 0 and  
 puts the array in gear for appending. Maybe an optional parameter  
 indicates a minimum desired capacity.

 Then people who want to migrate from D1 to D2 or generally consider  
 using this idiom can grep their source code for "length *= *0" and  
 make the change on their own pace and with appropriate testing.
Arguably this is already possible within a project, such as Sociomantic's. They haven't done it yet. I don't know why. -Steve
It has not been mentioned as example of blocker but example of most horrible breaking change that can happen if dmd user is not aware of it before upgrade.
I think that is very much a matter of your coding style. Many people wouldn't notice any changes, and might even notice an improvement.
 I think only practical thing that blocks D2 transition right now is that  
 no one can afford to work on it full-time and amount of changes needed  
 is too big to be done as a side work - array stomping is not only  
 problematic area, I personally think const will cause much more problems.

 Really hope that with recent hires doing a focused porting effort will  
 soon become a real option. D1 is so D1 :P
I think with the example Don showed, the above fix is not going to catch all the problems. There is no easy grep fix. I totally get that the effort is not worth the benefit currently. Just let us know if we can help in any way when it becomes more of a realistic priority. I would love to see the most obvious commercial D success story on D2 :) -Steve
Mar 14 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 3/14/14, Dicebot <public dicebot.lv> wrote:
 Really hope that with recent hires doing a focused porting effort
 will soon become a real option. D1 is so D1 :P
OT and maybe NDAd: how many people have been hired? I know of at least two. :)
Mar 14 2014
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, March 14, 2014 14:50:43 Dicebot wrote:
 It has not been mentioned as example of blocker but example of
 most horrible breaking change that can happen if dmd user is not
 aware of it before upgrade.
 
 I think only practical thing that blocks D2 transition right now
 is that no one can afford to work on it full-time and amount of
 changes needed is too big to be done as a side work - array
 stomping is not only problematic area, I personally think const
 will cause much more problems.
 
 Really hope that with recent hires doing a focused porting effort
 will soon become a real option. D1 is so D1 :P
Just a thought for this particular case. You could write a clearForAppend for D1 which simply sets the array length to 0 and slowly change your current code to use that instead of setting the array length to 0 directly. Then when you do go to port to D2, you can simply change clearForAppend's behavior so that it also calls assumeSafeAppend. That would allow you to do this particular change as a side project rather than focusing on it full time, which obviously isn't enough on its own, but it could help. - Jonathan M Davis
Mar 19 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
If I may, and as author of dcollection, you'll certainly agree, I
think the way forward is improving the language when it comes to
collections.

That way, any behavior can be provided as user type. Patching the
language in such a way is simply moving the problem around.


qualifier for template type parameters. I consider this to be on


The problem is that T!ElementType is a ompletely different type
than T!const(ElementType) . Because of this, if is almost
impossible to provide anything that behave anything close to
arrays. That is th reason why collection are so scarce in D, this
is the reason why efforts to implement AA without the black magic
involved right now are failing.

This problem is a killer for the viability of D2 on the long run.
I'm not saying that lightly.
Mar 13 2014
next sibling parent reply "sclytrack" <sclytrack sexy.com> writes:
On Thursday, 13 March 2014 at 21:28:46 UTC, deadalnix wrote:
 If I may, and as author of dcollection, you'll certainly agree, 
 I
 think the way forward is improving the language when it comes to
 collections.

 That way, any behavior can be provided as user type. Patching 
 the
 language in such a way is simply moving the problem around.


 qualifier for template type parameters. I consider this to be on


 The problem is that T!ElementType is a ompletely different type
 than T!const(ElementType) . Because of this, if is almost
 impossible to provide anything that behave anything close to
 arrays. That is th reason why collection are so scarce in D, 
 this
 is the reason why efforts to implement AA without the black 
 magic
 involved right now are failing.

 This problem is a killer for the viability of D2 on the long 
 run.
 I'm not saying that lightly.
Language changes? Like this? ------------- Delayed D Language D Normal immutable int * a; int * a; immutable qual(int) * a; immutable(int) * a; immutable int qual(*) a; immutable int * a; ------------- struct A { int a; qual(int) b; } A vari; const A cvari; vari.a = 1; cvari = vari; cvari.a = 2; cvari.b = 3; //error ------------- qual class Test { } Delayed D D struct language immutable Test t; immutable(Test) * t; immutable qual(Test) t; immutable Test * t; ------------- struct Array(T) { qual(T) * data; size_t length; } Array!(int) a; const Array!(int) b; void routine(const Array!(int) b) { } b = a; routine(a);
Mar 13 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 13 March 2014 at 22:21:56 UTC, sclytrack wrote:
 	Language changes? Like this?

 	-------------

 	Delayed D Language			D Normal

 	immutable int * a;			int * a;
 	immutable qual(int) * a;		immutable(int) * a;
 	immutable int qual(*) a;		immutable int * a;

 	-------------

 	struct A
 	{
 		int a;
 		qual(int) b;
 	}
 			

 	A vari;
 	const A cvari;
 	vari.a = 1;
 	cvari = vari;
 	cvari.a = 2;
 	cvari.b = 3;	//error

 	-------------

 	qual
 	class Test
 	{
 	}


 	Delayed D				D struct language

 	immutable Test t;			immutable(Test) * t;
 	immutable qual(Test) t; 		immutable Test * t;

 	-------------


 	struct Array(T)
 	{
 		qual(T) * data;
 		size_t length;
 	}


 	Array!(int) a;	
 	const Array!(int) b;

 	void routine(const Array!(int) b)
 	{

 	}

 	b = a;
 	routine(a);
I'm not sure how to understand your change proposal, you should explicit it more. But it seems that you get what the problem is (or at least, you are poking the right language constructs in your proposal).
Mar 13 2014
parent "sclytrack" <sclytrack fake.com> writes:
On Thursday, 13 March 2014 at 22:21:56 UTC, sclytrack wrote:

 	Delayed D Language			D Normal

 	immutable int * a;			int * a;
 	immutable qual(int) * a;		immutable(int) * a;
 	immutable int qual(*) a;		immutable int * a;

 	-------------

 	struct A
 	{
 		int a;
 		qual(int) b;
 	}
 			

 	A vari;
 	const A cvari;
 	vari.a = 1;
 	cvari = vari;
 	cvari.a = 2;
 	cvari.b = 3;	//error

 	-------------

 	qual
 	class Test
 	{
 	}


 	Delayed D				D struct language

 	immutable Test t;			immutable(Test) * t;
 	immutable qual(Test) t; 		immutable Test * t;

 	-------------


 	struct Array(T)
 	{
 		qual(T) * data;
 		size_t length;
 	}


 	Array!(int) a;	
 	const Array!(int) b;

 	void routine(const Array!(int) b)
 	{

 	}

 	b = a;
 	routine(a);
I'm not sure how to understand your change proposal, you should explicit it more. But it seems that you get what the problem is (or at least, you are poking the right language constructs in your proposal). qual would be the entry point of the qualifier so immutable int * * * * qual(*) * * * data; So everything to the left would be immutable and everything to the right would be mutable. struct A { char [] a; qual(char) [] b; } When defining const A data; Again the const is delayed until the entry qual(int). So ... A data; struct A { char [] a; char [] b; } const A cdata; struct A { char [] a; const(char) [] b; } immutable A idata; struct A { char [] a; immutable(char) [] b; } All this A has a single memory layout and a single qualifier that modifies them. cdata = data; cdata = idata; --- struct B { A data; qual(A) qdata; } B bdata; struct B { A data; A qdata; } const B cbdata; struct B { A data; const(A) qdata; //struct A { char [] a; const(char) [] b; } } immutable B ibdata; struct B { A data; immutable(A) qdata; //struct A { char [] a; immutable(char) [] b; } } cbdata = bdata cbdata = ibdata For functions it the qualifier would also be delayed which means having the word const in front of it doesn't necessarily mean it is const. void routine(const A * data) { data.a = "modifiable"; } If we are forcing a delayed qualifier on A we are forcing the same one as the one on the pointer. const A qual(*) data; It is the same const applied to the pointer and the struct A. --- Okay now to more conventional D. These two are more or less the same type. Container!(PayloadType) Container!(PayloadType, const) The following two are different types. Container!(PayloadType) Container!(const(PayloadType)) --- Container!(PayloadType, const) The qualifier can be applied to your payload field and even none-payload fields. The thing is only PayLoadType is part of the type definition and the qualifier at the end is not. The qualifier at the end must not participate in the type definition in any way. Not in static if and so forth. By some magic needs to be prevented. Having to deal with a single qualifier is much easier than having to deal with the qualifier of every payload. Again a bit of qual code. struct Container!(PayloadType) { qual(char) [] a; PayloadType payload; } PayloadType participates in the type and sets the memory layout of the type. Here the qualifier isn't even on the payload. --- Container!(const(int), int, char [], const) The first 3 define the type. (const(int), int, char []) The last bit. The qualifier (const) of the container does not participate in the definition of the type. --- For overloading there are too many options. void routine( immutable Container!(int, const) data); //a void routine( immutable Container!(int, immutable) data); //b Here the first one (a) is not needed. In delayed qualifier D this would be. void routine(imutable qual(Container!(int)) data) //b Setting the qual in right in front of the container means that immutable applies to all the fields of the Container!(int) and not only those marked with qual. --- In Normal D style Existing style. Already possible. void routine( Container!(int, ) data); //a void routine( const Container!(int, const) data); //b void routine( immutable Container!(int, immutable) data); //c The new ones. void routine( Container!(int, const ) data); void routine( Container!(int, immutable) data); The nonsense ones. void routine( const Container!(int, ) data); void routine( immutable Container!(int, const) data); --- Anyways. Thank you for your time reading this mess.
Mar 17 2014
prev sibling parent "sclytrack" <sclytrack collector.com> writes:
On Thursday, 13 March 2014 at 21:28:46 UTC, deadalnix wrote:
 If I may, and as author of dcollection, you'll certainly agree, 
 I
 think the way forward is improving the language when it comes to
 collections.

 That way, any behavior can be provided as user type. Patching 
 the
 language in such a way is simply moving the problem around.
------------------------------------------ If it is not in the language then maybe a design pattern. ------------------------------------------

 qualifier for template type parameters. I consider this to be on

------------------------------------------ Number 1 problem. Noted. ------------------------------------------
 The problem is that T!ElementType is a ompletely different type
 than T!const(ElementType) . Because of this, if is almost
------------------------------------------ I grasp that you need some sort of an implicit cast from T!ElementType to T!const(ElementType). void routine(T!const(ElementType) p) {} T!ElementType a; routine(a); ------------------------------------------ Collection!(const(Element1), immutable(Element2), Element3); There are a lot of different implicit conversions that stacking up. Now separating the qualifier like this. T!(ElementType, immutable) The implicit conversions would then be based on that sole qualifier at the end. Would this not be sufficient for your collections? (Not rethorical). Collection!(const(Element1), immutable(Element2), Element3, const); ------------------------------------------ void routine(immutable Collection!(ElementType, const)); //Meh void routine(immutable Collection!(ElementType, immutable)); I also feel like the above //Meh should not be there. ------------------------------------------
 impossible to provide anything that behave anything close to
 arrays. That is th reason why collection are so scarce in D, 
 this
 is the reason why efforts to implement AA without the black 
 magic
------------------------------------------ I dunno. Not a magician. Have to think. ------------------------------------------
 involved right now are failing.
 This problem is a killer for the viability of D2 on the long 
 run.
 I'm not saying that lightly.
------------------------------------------ Everybody wants a solution to this. That is why everybody is jumping on this like crazy instead of responding to the finalize by default thread. (irony) Some more "Collection and Const Noise". Collection!const(Noise)
Mar 15 2014
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 arr.length = 0;
...
 3. Don's company uses D1 as its language, I highly recommend watching  
 Don's Dconf13 presentation (and look forward to his Dconf14 one!) to see  
 how effective D code can create unbelievable speed, especially where  
 array slices are concerned. But to the above line, in D2, they must add  
 the following code to get the same behavior:

 arr.assumeSafeAppend();
Just a quick note, buried in same thread that Don mentioned, he outlined a more specific case, and this does not involve setting length to 0, but to any arbitrary value. This means my approach does not help them, and although it makes sense, the idea that it would help Sociomantic move to D2 is not correct. -Steve
Mar 14 2014
parent reply "Don" <x nospam.com> writes:
On Friday, 14 March 2014 at 14:48:13 UTC, Steven Schveighoffer 
wrote:
 On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer 
 <schveiguy yahoo.com> wrote:


 arr.length = 0;
...
 3. Don's company uses D1 as its language, I highly recommend 
 watching Don's Dconf13 presentation (and look forward to his 
 Dconf14 one!) to see how effective D code can create 
 unbelievable speed, especially where array slices are 
 concerned. But to the above line, in D2, they must add the 
 following code to get the same behavior:

 arr.assumeSafeAppend();
Just a quick note, buried in same thread that Don mentioned, he outlined a more specific case, and this does not involve setting length to 0, but to any arbitrary value. This means my approach does not help them, and although it makes sense, the idea that it would help Sociomantic move to D2 is not correct. -Steve
Actually it would help a great deal. In most cases, we do set the length to 0. That example code is unusual. FYI: In D1, this was the most important idiom in the language. In the first D conference in 2007, a feature T[new] was described, specifically to support this idiom in a safe manner. Implementation was begun in the compiler. Unfortunately, it didn't happen in the end. I'm not sure if it would actually have worked or not. BTW you said somewhere that concatenation always allocates. What I actually meant was ~=, not ~. In our code it is always preceded by .length = 0 though. It's important that ~= should not allocate, when the existing capacity is large enough.
Mar 14 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/14/14, 8:58 AM, Don wrote:
 Actually it would help a great deal. In most cases, we do set the length
 to 0. That example code is unusual.

 FYI: In D1, this was the most important idiom in the language.
 In the first D conference in 2007, a feature T[new] was described,
 specifically to support this idiom in a safe manner. Implementation was
 begun in the compiler. Unfortunately, it didn't happen in the end. I'm
 not sure if it would actually have worked or not.
Some perspective would be interesting. 1. Do you think eliminating stomping was a strategic mistake? (Clearly it had tactical issues with regard to portability.) 2. If you could do it over again, would you go with T[new]? Something else? Andrei
Mar 14 2014
parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 14 March 2014 at 16:38:23 UTC, Andrei Alexandrescu
wrote:
 Some perspective would be interesting.

 1. Do you think eliminating stomping was a strategic mistake? 
 (Clearly it had tactical issues with regard to portability.)
We should stomp isolated arrays. We must not stomp immutable arrays. That mean we must not stomp const ones as well.
 2. If you could do it over again, would you go with T[new]? 
 Something else?
No much choixe here, because of the problem mentioned earlier in this thread. i'm baffled that this doesn't grab more attention.
Mar 14 2014
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 14 Mar 2014 11:58:06 -0400, Don <x nospam.com> wrote:

 On Friday, 14 March 2014 at 14:48:13 UTC, Steven Schveighoffer wrote:
 On Thu, 13 Mar 2014 11:24:01 -0400, Steven Schveighoffer  
 <schveiguy yahoo.com> wrote:


 arr.length = 0;
...
 3. Don's company uses D1 as its language, I highly recommend watching  
 Don's Dconf13 presentation (and look forward to his Dconf14 one!) to  
 see how effective D code can create unbelievable speed, especially  
 where array slices are concerned. But to the above line, in D2, they  
 must add the following code to get the same behavior:

 arr.assumeSafeAppend();
Just a quick note, buried in same thread that Don mentioned, he outlined a more specific case, and this does not involve setting length to 0, but to any arbitrary value. This means my approach does not help them, and although it makes sense, the idea that it would help Sociomantic move to D2 is not correct. -Steve
Actually it would help a great deal. In most cases, we do set the length to 0. That example code is unusual.
If that example is not usual, it means that case is easy to fix (as I stated in the other post). Can you estimate how many ways your code contracts the length of an array? I assume all of them must be fully referencing the block, since that was the requirement in D1 (the slice had to point at the beginning of the block for append to work). I imagine that adding this 'feature' of setting length = 0 would help, but maybe just adding a new, but similar symbol for length that means "Do what D1 length would have done" would be less controversial for adding to druntime. Example strawman: arr.slength = 0; // effectively the same as arr.length = 0; arr.assumeSafeAppend(); It would do the same thing, but the idea is it would work for extension too -- if arr points at the beginning of the block, and slength *grows* into the block, it would work the same as D1 as well -- adjusting the "used" length and not reallocating. Essentially, you would have to s/.length/.slength, and everything would work. Of course, length is not a keyword, so there may be other cases where length is used (read property for instance) where slength would not necessarily have to be used. However, one thing we cannot fix is: arr = arr[0..$-1]; arr ~= x; This would reallocate due to stomp prevention, and I can't fix that. Do you have any cases like this, or do you always use .length?
 FYI: In D1, this was the most important idiom in the language.
 In the first D conference in 2007, a feature T[new] was described,  
 specifically to support this idiom in a safe manner. Implementation was  
 begun in the compiler. Unfortunately, it didn't happen in the end. I'm  
 not sure if it would actually have worked or not.
I think the benefits would have been very minimal. T[new] would be almost exactly like T[]. And you still would have to have updated all your code to use T[new]. When to use T[new] or T[] would have driven most people mad. I was around back then, and I also remember Andrei, when writing TDPL, stating that the T[new] and T[] differences were so subtle and confusing that he was glad he didn't have to specify that.
 BTW you said somewhere that concatenation always allocates. What I  
 actually meant was ~=, not ~.
OK
 In our code it is always preceded by .length = 0 though.
 It's important that ~= should not allocate, when the existing capacity  
 is large enough.
That is the case for D2, as long as the slice is qualified as ending at the end of the valid data (setting length to 0 doesn't do this, obviously). This is a slight departure from D1, which required the slice to point at the *beginning* of the data. -Steve
Mar 14 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 03/14/2014 06:38 PM, Steven Schveighoffer wrote:
 I imagine that adding this 'feature' of setting length = 0 would help,
 but maybe just adding a new, but similar symbol for length that means
 "Do what D1 length would have done" would be less controversial for
 adding to druntime.  Example strawman:

 arr.slength = 0; // effectively the same as arr.length = 0;
 arr.assumeSafeAppend();

 It would do the same thing, but the idea is it would work for extension
 too -- if arr points at the beginning of the block, and slength *grows*
 into the block, it would work the same as D1 as well -- adjusting the
 "used" length and not reallocating.

 Essentially, you would have to s/.length/.slength, and everything would
 work. Of course, length is not a keyword, so there may be other cases
 where length is used (read property for instance) where slength would
 not necessarily have to be used.

 However, one thing we cannot fix is:

 arr = arr[0..$-1];
 arr ~= x;

 This would reallocate due to stomp prevention, and I can't fix that.
auto sslice(T)(T[] arr, size_t s, size_t e){ auto r=arr[s..e]; r.assumeSafeAppend(); return r; } arr = arr.sslice(0, $-1); arr ~= x;
 Do you have any cases like this, or do you always use .length?
I think just replacing all usages of features that have changed behaviour with library implementations that simulate the old behaviour would be quite easy.
Mar 16 2014