www.digitalmars.com         C & C++   DMDScript  

D - .length modification question

reply John Boucher <John_member pathlink.com> writes:
If 
args.length = args.length - 1 ;
is OK, why do
args.length -= 1 ;
and
args.length-- ;
produce the compilation error
'args.length' is not an lvalue
?

I hope it's a bug rather than a (poor) design decision.

John Boucher
The King had Humpty pushed.
Sep 19 2003
parent reply J Anderson <anderson badmama.com.au.REMOVE> writes:
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

John Boucher wrote:

If 
args.length = args.length - 1 ;
is OK, why do
args.length -= 1 ;
and
args.length-- ;
produce the compilation error
'args.length' is not an lvalue
?

I hope it's a bug rather than a (poor) design decision.

John Boucher
The King had Humpty pushed.
  

debate about this. It's really to keep programmers from doing: for (int n=0; n<x; x++) { args.length++; //or args.length--; ... } Which is less much less efficient then. args.length = args.length + x; for (int n=0; n<x; x++) { ... } As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one. The longer syntax is to discourage bad programming. -Anderson
Sep 19 2003
next sibling parent reply John Boucher <John_member pathlink.com> writes:
Ah, so the latter. I guess I'll stick with C# then.

In article <bkfq6p$5mt$1 digitaldaemon.com>, J Anderson says...
This is a multi-part message in MIME format.
--------------060601020803030507000701
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

John Boucher wrote:

If 
args.length = args.length - 1 ;
is OK, why do
args.length -= 1 ;
and
args.length-- ;
produce the compilation error
'args.length' is not an lvalue
?

I hope it's a bug rather than a (poor) design decision.

John Boucher
The King had Humpty pushed.
  

debate about this. It's really to keep programmers from doing: for (int n=0; n<x; x++) { args.length++; //or args.length--; ... } Which is less much less efficient then. args.length = args.length + x; for (int n=0; n<x; x++) { ... } As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one. The longer syntax is to discourage bad programming. -Anderson --------------060601020803030507000701 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"> <title></title> </head> <body text="#000000" bgcolor="#ffffff"> John Boucher wrote:<br> <blockquote type="cite" cite="midbkfms1$2uag$1 digitaldaemon.com"> <pre wrap="">If args.length = args.length - 1 ; is OK, why do args.length -= 1 ; and args.length-- ; produce the compilation error 'args.length' is not an lvalue ? I hope it's a bug rather than a (poor) design decision. John Boucher The King had Humpty pushed. </pre> </blockquote> It's not nessarily a poor design decision.&nbsp; There's been quite a bit of debate about this.&nbsp; It's really to keep programmers from doing:<br> <br> for (int n=0; n&lt;x; x++)<br> {<br> &nbsp;&nbsp;&nbsp; args.length++; //or args.length--;<br> &nbsp;&nbsp;&nbsp; ...<br> }<br> <br> Which is less much less efficient then.<br> <br> args.length = args.length + x;<br> for (int n=0; n&lt;x; x++)<br> {<br> &nbsp;&nbsp;&nbsp; ...<br> }<br> <br> As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one.&nbsp; The longer syntax is to discourage bad programming.<br> <br> -Anderson<br> </body> </html> --------------060601020803030507000701--

Sep 19 2003
next sibling parent "Charles Sanders" <sanders-consulting comcast.net> writes:
You guess you'll stick with C#, what is that supposed to mean ?

C

"John Boucher" <John_member pathlink.com> wrote in message
news:bkfr7m$8qm$1 digitaldaemon.com...
 Ah, so the latter. I guess I'll stick with C# then.

 In article <bkfq6p$5mt$1 digitaldaemon.com>, J Anderson says...
This is a multi-part message in MIME format.
--------------060601020803030507000701
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

John Boucher wrote:

If
args.length = args.length - 1 ;
is OK, why do
args.length -= 1 ;
and
args.length-- ;
produce the compilation error
'args.length' is not an lvalue
?

I hope it's a bug rather than a (poor) design decision.

John Boucher
The King had Humpty pushed.

debate about this. It's really to keep programmers from doing: for (int n=0; n<x; x++) { args.length++; //or args.length--; ... } Which is less much less efficient then. args.length = args.length + x; for (int n=0; n<x; x++) { ... } As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one. The longer syntax is to discourage bad programming. -Anderson --------------060601020803030507000701 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"> <title></title> </head> <body text="#000000" bgcolor="#ffffff"> John Boucher wrote:<br> <blockquote type="cite" cite="midbkfms1$2uag$1 digitaldaemon.com"> <pre wrap="">If args.length = args.length - 1 ; is OK, why do args.length -= 1 ; and args.length-- ; produce the compilation error 'args.length' is not an lvalue ? I hope it's a bug rather than a (poor) design decision. John Boucher The King had Humpty pushed. </pre> </blockquote> It's not nessarily a poor design decision.&nbsp; There's been quite a bit


debate about this.&nbsp; It's really to keep programmers from doing:<br>
<br>
for (int n=0; n&lt;x; x++)<br>
{<br>
&nbsp;&nbsp;&nbsp; args.length++; //or args.length--;<br>
&nbsp;&nbsp;&nbsp; ...<br>
}<br>
<br>
Which is less much less efficient then.<br>
<br>
args.length = args.length + x;<br>
for (int n=0; n&lt;x; x++)<br>
{<br>
&nbsp;&nbsp;&nbsp; ...<br>
}<br>
<br>
As I see it, most of the time, you should be increasing/decreasing an
array size in large blocks. Code should very rarely need to increase an
array size by one.&nbsp; The longer syntax is to discourage bad


<br>
-Anderson<br>
</body>
</html>

--------------060601020803030507000701--


Sep 19 2003
prev sibling parent "Andrew Edwards" <edwardsac spamfreeusa.com> writes:
"John Boucher" <John_member pathlink.com> wrote in message
news:bkfr7m$8qm$1 digitaldaemon.com...
 Ah, so the latter. I guess I'll stick with C# then.

Andrew
Sep 19 2003
prev sibling parent reply Helmut Leitner <helmut.leitner chello.at> writes:
 J Anderson wrote:
 
 John Boucher wrote:
 
 If
 args.length = args.length - 1 ;
 is OK, why do
 args.length -= 1 ;
 and
 args.length-- ;
 produce the compilation error
 'args.length' is not an lvalue
 ?

 I hope it's a bug rather than a (poor) design decision.

 John Boucher
 The King had Humpty pushed.

really to keep programmers from doing: for (int n=0; n<x; x++) { args.length++; //or args.length--; ... } Which is less much less efficient then. args.length = args.length + x; for (int n=0; n<x; x++) { ... } As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one. The longer syntax is to discourage bad programming. -Anderson

Is it really "to discourage bad programming"? I would have assumed, that this is an arbitrary implementation detail. Any programming language that is sufficiently complex, will have all doors open for "bad programing" and there is no way to stop this. Grandmothering was never C's style, at least. Is it in D? == One idea that hounts me, is the Java String/Stringbuffer problem, that is somehow reflected in the D OutBuffer class. It basically means, that there are situations were you want to reserve space for an array which will then grow and shrink below this limits without reallocations. In Java - and currently in D - you need to create special classes for this purpose. If any array had a .reserve field (Walter's C++ Array class even has), lots of ways would open up for efficient programming. And we could drop the OutBuffer class completly. We could also write args.reserve=100; args.length++; without being inefficient. The cost of 4 byte per array seems steeper than it is. It would only hurt with small strings. Even with them (allocated with N*16 byte) the effect would be small. On the positive side, you could just write: alias char [] string; string buffer; buffer.reserve=100000; foreach(file; sourcefiles) { FileGetStr(file,buffer); ... } without any reallocation inefficiencies in typical conditions. This would also go a long way towards solving the problem of efficient formatted output, which is still hindered be the fact that string reallocation is unavoidable. With a .reserve you could just write string s; s.length=100; StrFormat(s,"Name=",name); StrCatFormat(s,"Age=%d",age); so that you can handle this like a normal string, while no reallocation or object creation has to happen inside. ==== Therefore, please consider the suggestion to add a property .reserve to the array type in this way: - reserve can be read and set like length - reallocations happen according to max(reserve,length) advantage: - if length changes below or equal reserve, no reallocations need to happen. -- Helmut Leitner leitner hls.via.at Graz, Austria www.hls-software.com
Sep 19 2003
next sibling parent J Anderson <anderson badmama.com.au.REMOVE> writes:
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Helmut Leitner wrote:

  

J Anderson wrote:

John Boucher wrote:

    

If
args.length = args.length - 1 ;
is OK, why do
args.length -= 1 ;
and
args.length-- ;
produce the compilation error
'args.length' is not an lvalue
?

I hope it's a bug rather than a (poor) design decision.

John Boucher
The King had Humpty pushed.


      

really to keep programmers from doing: for (int n=0; n<x; x++) { args.length++; //or args.length--; ... } Which is less much less efficient then. args.length = args.length + x; for (int n=0; n<x; x++) { ... } As I see it, most of the time, you should be increasing/decreasing an array size in large blocks. Code should very rarely need to increase an array size by one. The longer syntax is to discourage bad programming. -Anderson

Is it really "to discourage bad programming"? I would have assumed, that this is an arbitrary implementation detail. Any programming language that is sufficiently complex, will have all doors open for "bad programing" and there is no way to stop this. Grandmothering was never C's style, at least. Is it in D? == One idea that hounts me, is the Java String/Stringbuffer problem, that is somehow reflected in the D OutBuffer class. It basically means, that there are situations were you want to reserve space for an array which will then grow and shrink below this limits without reallocations. In Java - and currently in D - you need to create special classes for this purpose. If any array had a .reserve field (Walter's C++ Array class even has), lots of ways would open up for efficient programming. And we could drop the OutBuffer class completly.

We could also write
   args.reserve=100;
   args.length++;
without being inefficient.

The cost of 4 byte per array seems steeper than it is. It would only hurt
with small strings. Even with them (allocated with N*16 byte) the 
effect would be small.

On the positive side, you could just write:

   alias char [] string; 
   string buffer;
   buffer.reserve=100000;
   foreach(file; sourcefiles) {
      FileGetStr(file,buffer);
      ...
   }

without any reallocation inefficiencies in typical conditions.

This would also go a long way towards solving the problem
of efficient formatted output, which is still hindered be the
fact that string reallocation is unavoidable. 

With a .reserve you could just write
   string s;
   s.length=100;
   StrFormat(s,"Name=",name);
   StrCatFormat(s,"Age=%d",age);
so that you can handle this like a normal string, while
no reallocation or object creation has to happen inside.

====

Therefore, please consider the suggestion to add a property
   .reserve
to the array type in this way:
   -  reserve can be read and set like length
   -  reallocations happen according to max(reserve,length)
advantage:
   -  if length changes below or equal reserve,
      no reallocations need to happen.

  

where nine or ten different "reserve" techniques offered. Actually a couple of "reserve" techniques required no extra memory at all. However, they all had overhead (not just memory overhead). There's an extra check at each resize. The resulting consensus was to have another type, defined in the standard lib. A programmer should be given the option of using a reserve or not. That's exactly what leaving it out of the standard D array does because they can use the standard lib one, or any other allocator scheme they like. -Anderson
Sep 20 2003
prev sibling next sibling parent reply "Walter" <walter digitalmars.com> writes:
Actually, the implementation of D arrays does have a bit of a 'reserve',
since the garbage collector allocates by using power of 2 buckets. This is
an implementation detail, though. It still makes for much more efficient
code to set the .length to some reasonably expected reserve value, then fill
up the array, then reset the .length to the final size.
Sep 22 2003
next sibling parent reply J C Calvarese <jcc7 cox.net> writes:
Walter wrote:
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets. This is
 an implementation detail, though. It still makes for much more efficient
 code to set the .length to some reasonably expected reserve value, then fill
 up the array, then reset the .length to the final size.

But how do you keep up with the currently-filled length using this method? I'm guessing you have an extra variable whose use may not be obvious. I'd rather use .length for the current length and use .reserve to set a reasonable expected length. I think Helmut's idea makes a lot of sense. Would it be difficult to implement? Justin
Sep 22 2003
parent reply "Walter" <walter digitalmars.com> writes:
"J C Calvarese" <jcc7 cox.net> wrote in message
news:bko3oe$l8q$1 digitaldaemon.com...
 Walter wrote:
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets. This


 an implementation detail, though. It still makes for much more efficient
 code to set the .length to some reasonably expected reserve value, then


 up the array, then reset the .length to the final size.

method? I'm guessing you have an extra variable whose use may not be obvious. I'd rather use .length for the current length and use .reserve to set a reasonable expected length. I think Helmut's idea makes a lot of sense. Would it be difficult to implement?

After setting .length to the max reasonable size, then set .length back to zero. You now have a reserve that is something like the max reasonable size rounded up to the next bucket size.
Sep 23 2003
next sibling parent reply Hauke Duden <H.NS.Duden gmx.net> writes:
Walter wrote:
 After setting .length to the max reasonable size, then set .length back to
 zero. You now have a reserve that is something like the max reasonable size
 rounded up to the next bucket size.

Hmmm. Does that mean that the amount of memory used up by arrays can never decrease? I think you should at least leave the door open for later optimization of the array memory handling so that some of the memory is freed if the used length is below a certain threshold. That would mean, though, that the kind of reserving memory that you proposed may not work in the future. Increasing the length and then decreasing it again also looks a bit like a dirty hack - the semantics depend on the internal compiler architecture. Nothing I would want to use and certainly nothing that is easy to understand for an uninitiated programmer reading someone else's code. Is there a specific reason why you don't want to expose the reserve field? It looks like a much cleaner solution to me. Hauke
Sep 23 2003
next sibling parent Antti =?iso-8859-1?Q?Syk=E4ri?= <jsykari gamma.hut.fi> writes:
In article <bkq1lf$r5v$1 digitaldaemon.com>, Hauke Duden wrote:
 Walter wrote:
 After setting .length to the max reasonable size, then set .length back to
 zero. You now have a reserve that is something like the max reasonable size
 rounded up to the next bucket size.

That would mean, though, that the kind of reserving memory that you proposed may not work in the future. Increasing the length and then decreasing it again also looks a bit like a dirty hack - the semantics depend on the internal compiler architecture.

 Is there a specific reason why you don't want to expose the reserve 
 field? It looks like a much cleaner solution to me.

reserve(int i) { int old_length = length; length = i; length = old_length; } Voilà! A self-documenting, reusable solution. -Antti
Sep 23 2003
prev sibling parent reply "Sean L. Palmer" <palmer.sean verizon.net> writes:
"Hauke Duden" <H.NS.Duden gmx.net> wrote in message
news:bkq1lf$r5v$1 digitaldaemon.com...
 Walter wrote:
 After setting .length to the max reasonable size, then set .length back


 zero. You now have a reserve that is something like the max reasonable


 rounded up to the next bucket size.

Hmmm. Does that mean that the amount of memory used up by arrays can never decrease? I think you should at least leave the door open for later optimization of the array memory handling so that some of the memory is freed if the used length is below a certain threshold.

I suppose the GC might (if fed enough info) know that a block holds an array, know where to get the size, and free up blocks beyond the power of two above the current use.
 That would mean, though, that the kind of reserving memory that you
 proposed may not work in the future. Increasing the length and then
 decreasing it again also looks a bit like a dirty hack - the semantics
 depend on the internal compiler architecture. Nothing I would want to
 use and certainly nothing that is easy to understand for an uninitiated
 programmer reading someone else's code.

 Is there a specific reason why you don't want to expose the reserve
 field? It looks like a much cleaner solution to me.

Yes, I dislike depending on implicit semantics. Walter, are you going to set this reserve behavior in stone, or is it unspecified? Sean
Sep 24 2003
next sibling parent reply Hauke Duden <H.NS.Duden gmx.net> writes:
Sean L. Palmer wrote:
 I suppose the GC might (if fed enough info) know that a block holds an
 array, know where to get the size, and free up blocks beyond the power of
 two above the current use.

But what happens if the GC does this in the first iteration of the loop? Then we lose our "reservation" and end up having the inefficient case again. Hauke
Sep 24 2003
parent reply "Sean L. Palmer" <palmer.sean verizon.net> writes:
What, you want cake and want to eat it too?  ;)

I don't have an answer at present.  D could store the reserved size;  this
would take a bit of space, nothing that would kill anyone.

Sean

"Hauke Duden" <H.NS.Duden gmx.net> wrote in message
news:bkrml2$4a7$2 digitaldaemon.com...
 Sean L. Palmer wrote:
 I suppose the GC might (if fed enough info) know that a block holds an
 array, know where to get the size, and free up blocks beyond the power


 two above the current use.

But what happens if the GC does this in the first iteration of the loop? Then we lose our "reservation" and end up having the inefficient case

 Hauke

Sep 24 2003
parent Hauke Duden <H.NS.Duden gmx.net> writes:
Sean L. Palmer wrote:
 What, you want cake and want to eat it too?  ;)

<homer> Mmmmmh. Caaake. </homer>
 I don't have an answer at present.  D could store the reserved size;  this
 would take a bit of space, nothing that would kill anyone.

As I understand it, the current implementation already does this. Also, I cannot think of an efficient implementation of dynamic arrays that can work with only a length field. You could only allocate exactly the length and would have to reallocate with each change of the length. Not what I'd call efficient. So if a reserve field will be needed anyway, why not expose it to the programmer? Hauke
Sep 24 2003
prev sibling parent J Anderson <anderson badmama.com.au.REMOVE> writes:
Sean L. Palmer wrote:

"Hauke Duden" <H.NS.Duden gmx.net> wrote in message
news:bkq1lf$r5v$1 digitaldaemon.com...
  

Walter wrote:
    

After setting .length to the max reasonable size, then set .length back
      


zero. You now have a reserve that is something like the max reasonable
      


rounded up to the next bucket size.
      

never decrease? I think you should at least leave the door open for later optimization of the array memory handling so that some of the memory is freed if the used length is below a certain threshold.

I suppose the GC might (if fed enough info) know that a block holds an array, know where to get the size, and free up blocks beyond the power of two above the current use.
That would mean, though, that the kind of reserving memory that you
proposed may not work in the future. Increasing the length and then
decreasing it again also looks a bit like a dirty hack - the semantics
depend on the internal compiler architecture. Nothing I would want to
use and certainly nothing that is easy to understand for an uninitiated
programmer reading someone else's code.

Is there a specific reason why you don't want to expose the reserve
field? It looks like a much cleaner solution to me.
    

Yes, I dislike depending on implicit semantics. Walter, are you going to set this reserve behavior in stone, or is it unspecified? Sean

field is the most optimal technique. Although, if there was a reserve field (but the implementation technique wasn't set in stone), then it could be ignored, I suppose.
Sep 25 2003
prev sibling parent J C Calvarese <jcc7 cox.net> writes:
Walter wrote:
 
 After setting .length to the max reasonable size, then set .length back to
 zero. You now have a reserve that is something like the max reasonable size
 rounded up to the next bucket size.

OK. That sounds like just the kind of feature I was requesting. Thanks. Justin
Sep 23 2003
prev sibling next sibling parent reply Helmut Leitner <leitner hls.via.at> writes:
Walter wrote:
 
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets. 

This will help little. Lets assume, someone decides to read a file line by line (or read it in one sweep but ends with a line in a string instead of a slice). The line will be reallocated hundredes of times. char line[]; line.reserve=1024; would set aside enough space so that reallocations would happen almost never.
 This is
 an implementation detail, though. It still makes for much more efficient
 code to set the .length to some reasonably expected reserve value, then fill
 up the array, then reset the .length to the final size.

Yes, I know that. But very often this is a repeated process and reallocation can only be avoided with a .reserve. How does your dmd compiler avoid reallocation of the "current source buffer"? If he does, he must use some kind of .reserve . If he doesn't there would be room for making it faster. This reserve thing is not theoretical. After 20 years of C-programming I've based all my C dynamic array handling on a "generic" Buffer structure, that looks like (translated to fit): typedef struct buffer { void *ptr; int length; int reserve; (int element_size; sometimes implicit) } BUFFER ; There are generic functions for reallocations, insertions, deletions, sorting... The reserve is an essential feature for optimization, for you can often use heuristics. For example you build a hash-bucket-table for a dictionary. You don't know how many entries you will have (perhaps 2000 or 10000), but you definitely don't want to start at the default (maybe 50?) to reallocate and rehash 5-8 times to reach final size. A simple .reserve=1000 will give you a much better start and give you the feeling that you have incorporated your partial knowledge as good as possible. How do you (Walter) build symbol tables during DMD compilation? Do you reallocate them when the symbols come? You can't know before how large these tables will grow. But I suppose you won't think much about the small sources and to mimize the compiler memory need for these. So you will reserve moderate space for these arrays, don't you? ==== I see two problems with reserve. The first is an implementation problem. An array is not an object. It is stored and passed as a (ptr, size) duple. And this is also how slices are handled (I think). So reserve it is not justing adding a field to an object. The second is a performance issue. Typically compiler builders have different performance interests than programmers. Compiler builders want to look good in benchmarks. Programmers want to write fast applications easily. A .reserve might worsen benchmarks (I don't think so, but we don't know), but would be an enormous benefit for the application programmer, because he needn't write special classes (which he often won't do) to get optimum performance. An example of this conflict of interest is the wc official sample program, which will perform but not scale, because if (inword) { char[] word = input[wstart .. input.length]; dictionary[word]++; } will keep the basic input buffer (the whole file) in memory. So you would be surprised about this code, if you would extend it to count the symbols in a larger set of files. It will eat all your memory. And it would be a good "test for experienced D programming capability" to change this to a performing and scaling example. ==== I also want to restate, that "basic IO" is still missing in D. This means, you can't write a tutorial that makes sense, because no sensible IO is available (you have to go back to printf and its friends) that you want to show anyone new to the language. Why is this so? Because "char []" can't be used for IO in a performing way! So this decision and implementation has been pushed and pushed .... And you can't work with "char buffer[N]" because - you can never be sure about the buffer size - you will need conversions all the time The obvious solution, the final, powerful and performing String class is a pure vision. I suppose it will never come. .reserve would be a pragmatic solution to this. -- Helmut Leitner leitner hls.via.at Graz, Austria www.hls-software.com
Sep 23 2003
parent reply "Walter" <walter digitalmars.com> writes:
"Helmut Leitner" <leitner hls.via.at> wrote in message
news:3F700109.7EDDD804 hls.via.at...
 Walter wrote:
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets.

line by line (or read it in one sweep but ends with a line in a string instead of a slice). The line will be reallocated hundredes of times.

No. For each line, maybe 2 to 4 times.
   char line[];
   line.reserve=1024;
 would set aside enough space so that reallocations would happen almost

You can do the same thing with: char line[]; line.length = 1024; line.length = 0; Now fill the array.
 How does your dmd compiler avoid reallocation of the "current source

The compiler doesn't. The garbage collector runtime does. It does it by figuring out by the address which bucket it is in, and how much room is left in the bucket.
 This reserve thing is not theoretical. After 20 years of C-programming

 based all my C dynamic array handling on a "generic" Buffer structure,

 looks like (translated to fit):

You're right. I use something almost exactly the same in my C programming. But with the garbage collector, I don't need to carry around the extra 'reserve' size, as it is implicit in the address of the buffer.
 An example of this conflict of interest is the wc official sample program,
 which will perform  but not scale, because
         if (inword)
         {   char[] word = input[wstart .. input.length];
             dictionary[word]++;
         }
 will keep the basic input buffer (the whole file) in memory. So you would
 be surprised about this code, if you would extend it to count the symbols
 in a larger set of files. It will eat all your memory. And it would be a
 good "test for experienced D programming capability" to change this to a
 performing and scaling example.

To process hundreds of megabytes of source with wc, just change the line to: char[] word = input[wstart .. input.length].dup;
 I also want to restate, that "basic IO" is still missing in D.

 This means, you can't write a tutorial that makes sense, because no
 sensible IO is available (you have to go back to printf and its friends)

 you want to show anyone new to the language.

 Why is this so? Because "char []" can't be used for IO in a performing

I think the reason is more my sloth than that!
 So this decision and implementation has been pushed and pushed ....
 And you can't work with "char buffer[N]" because
   - you can never be sure about the buffer size
   - you will need conversions all the time
 The obvious solution, the final, powerful and performing String class
 is a pure vision. I suppose it will never come.

 .reserve would be a pragmatic solution to this.

Sep 23 2003
next sibling parent reply Helmut Leitner <helmut.leitner chello.at> writes:
Walter wrote:
 An example of this conflict of interest is the wc official sample program,
 which will perform  but not scale, because
         if (inword)
         {   char[] word = input[wstart .. input.length];
             dictionary[word]++;
         }
 will keep the basic input buffer (the whole file) in memory. So you would
 be surprised about this code, if you would extend it to count the symbols
 in a larger set of files. It will eat all your memory. And it would be a
 good "test for experienced D programming capability" to change this to a
 performing and scaling example.

To process hundreds of megabytes of source with wc, just change the line to: char[] word = input[wstart .. input.length].dup;

It may not be that simple, because I would hope that char[] word = input[wstart .. input.length]; if(dictionary[word]!=NULL) { dictionary[word]++; } else { dictionary[word.dup]++; } performs better, because only those words are created as objects that are actually needed. But it may also be, that char[] word = input[wstart .. input.length]; int *p=dictionary[word]; if(p) { dictionary[word]++; } else { dictionary[word.dup]++; } performs even better, because it avoids an hash access that we can't be sure about whether the compiler optimizes it away or not. -- Helmut Leitner leitner hls.via.at Graz, Austria www.hls-software.com
Sep 23 2003
parent Helmut Leitner <helmut.leitner chello.at> writes:
Sorry for:

Helmut Leitner wrote:
 But it may also be, that
 
       char[] word = input[wstart .. input.length];
       int *p=dictionary[word];
       if(p) {
           dictionary[word]++;

This should read: *p++;
       } else {
           dictionary[word.dup]++;
       }
 
 performs even better, because it avoids an hash access that we can't
 be sure about whether the compiler optimizes it away or not.

-- Helmut Leitner leitner hls.via.at Graz, Austria www.hls-software.com
Sep 23 2003
prev sibling parent reply "Julio César Carrascal Urquijo" <adnoctum phreaker.net> writes:
 You can do the same thing with:
     char line[];
     line.length = 1024;
     line.length = 0;

 Now fill the array.

A question: Does the GC respects any number of buckets allocated like this? char line1[]; line1.length = 4096; line1.length = 0; char line2[]; line2.length = 4096; line2.length = 0; I mean, if I reserve more than 1 bucket (of 1024) will still be available after allocating another array?
Sep 23 2003
parent "Walter" <walter digitalmars.com> writes:
"Julio César Carrascal Urquijo" <adnoctum phreaker.net> wrote in message
news:bkqd4j$1c4m$1 digitaldaemon.com...
 A question: Does the GC respects any number of buckets allocated like

 char line1[];
 line1.length = 4096;
 line1.length = 0;

 char line2[];
 line2.length = 4096;
 line2.length = 0;

Yes.
 I mean, if I reserve more than 1 bucket (of 1024) will still be available
 after allocating another array?

Dec 10 2003
prev sibling parent reply "Vathix" <vathix dprogramming.com> writes:
"Walter" <walter digitalmars.com> wrote in message
news:bko2u7$is2$1 digitaldaemon.com...
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets. This is
 an implementation detail, though. It still makes for much more efficient
 code to set the .length to some reasonably expected reserve value, then

 up the array, then reset the .length to the final size.

What about this: char[] s = new char[100]; s = s[0 .. 5]; would .length changes create a new buffer or use that 100 up first?
Sep 23 2003
parent "Walter" <walter digitalmars.com> writes:
"Vathix" <vathix dprogramming.com> wrote in message
news:bkp8j2$2pmi$1 digitaldaemon.com...
 "Walter" <walter digitalmars.com> wrote in message
 news:bko2u7$is2$1 digitaldaemon.com...
 Actually, the implementation of D arrays does have a bit of a 'reserve',
 since the garbage collector allocates by using power of 2 buckets. This


 an implementation detail, though. It still makes for much more efficient
 code to set the .length to some reasonably expected reserve value, then

 up the array, then reset the .length to the final size.

char[] s = new char[100]; s = s[0 .. 5]; would .length changes create a new buffer or use that 100 up first?

The latter. It would not reallocate it.
Sep 23 2003
prev sibling parent "Riccardo De Agostini" <riccardo.de.agostini email.it> writes:
"Helmut Leitner" <helmut.leitner chello.at> ha scritto nel messaggio
news:3F6BF6C5.AB02DBDD chello.at...

 Therefore, please consider the suggestion to add a property
    .reserve
 to the array type in this way:
    -  reserve can be read and set like length
    -  reallocations happen according to max(reserve,length)
 advantage:
    -  if length changes below or equal reserve,
       no reallocations need to happen.

You have my vote! BTW, what if length grows above reserve? Should there also be a property to specify subsequent allocation "steps"? Say, reserve 100 elements at first, then grow by 10 at a time? Ric
Sep 23 2003