www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Possible rewrite of array operation spec

reply Stewart Gordon <smjg_1998 yahoo.com> writes:
This'll probably get people asking the prospect of array operations for 
1.0 to be resurrected, but still....

Here is a possible specification for array operations that I feel is 
better-defined than the one in the current out-of-date spec.  Of course, 
there are still some open questions, which I've put at the bottom.


Array operations
----------------
Arithmetic and bitwise operators are defined on array operands.  An 
expression involving an array evaluates to a new array in which the 
operator has been applied to the elements of the operands in turn.

In essence, when an expression contains more than one array operation, a 
new array is created to hold the result of each operation.  However, a 
quality implementation will optimize the evaluation of the expression to 
eliminate temporaries where possible.

Unary operations
~~~~~~~~~~~~~~~~
For the unary operators +, - and ~, the expression evaluates to a new 
array containing the result of applying the operator to each element. 
For example, with the declaration

	int[] x, y;

then the statement

	y = -x;

is simply equivalent to

	y = new int[x.length];
	for (int i = 0; i < y.length; i++) {
		y[i] = -x[i];
	}

Binary operations
~~~~~~~~~~~~~~~~~
The binary operations supported are +, -, *, /, %, &, |, ^, <<, >> and >>>.

If the two arrays are of the same dimension and of compatible types, 
then the expression evaluates to a new array in which each element is 
the result of applying the operator to corresponding elements of the 
operands.  For example, with the declarations

     int[] x, y, z;

the statement

     z = x + y;

is equivalent to

	z = new int[x.length];
	for (int i = 0; i < z.length; i++) {
		z[i] = x[i] + y[i];
	}

Both operands must be of the same length.  If they are not, an 
ArrayBoundsError is thrown.

For higher dimensions, this definition is applied recursively.  For 
example, with

     int[][] x, y, z;

the statement

     z = x * y;

is equivalent to

	z = new int[x.length];
	for (int i = 0; i < z.length; i++) {
		z[i] = x[i] * y[i];
	}

which is in turn equivalent to

	z = new int[x.length];
	for (int i = 0; i < z.length; i++) {
		z[i] = new int[x[i].length];
		for (int j = 0; j < z[i].length; j++) {
			z[i][j] = x[i][j] * y[i][j];
		}
	}

If the operands do not match in dimension, then the operator is applied 
to each element of the higher-dimension operation with the whole of the 
lower-dimension one.  For example, with

	int[] x, z;
	int y;

the statement

	z = x - y;

is equivalent to

	z = new int[x.length];
	for (int i = 0; i < z.length; i++) {
		z[i] = x[i] - y;
	}

Similarly,

	z = y - x;

is equivalent to

	z = new int[x.length];
	for (int i = 0; i < z.length; i++) {
		z[i] = y - x[i];
	}

This definition is applied recursively if the dimensions differ by two 
or more.

Assignment operations
~~~~~~~~~~~~~~~~~~~~~
When x is an array, the assignment

	x op= y;

is taken as equivalent to

	x = x op y;

whether y is an array of matching dimension, an array of lower dimension 
or a scalar.  Thus the operation creates a new array and assigns it to 
x.  If a sliced lvalue is used, the array is modified in place, so that

	x[] op= y;

is equivalent to

	x[] = x[] op y;

The preincrement and predecrement operators are handled in the same way.

User-defined types
~~~~~~~~~~~~~~~~~~
A class, struct or union type may have operators overloaded with array 
types as parameters.  To avoid conflicts between overloaded operators 
and array operations, binary operations involving both array and 
user-defined types are resolved as follows:

1. The normal operator overloading rules are applied.
2. If no match is found, the array operation rules are applied until 
both operands are reduced to scalar type; operator overloading rules are 
then applied to the result.
3. If the expression still does not resolve, it is an error.


Open questions
~~~~~~~~~~~~~~
Should postincrement and postdecrement be allowed?  How should they be 
handled?

Should we generalise the concept to function calls?  If so, I guess that 
overload resolution would work in much the same way as for operations on 
user-defined types.

If we do allow it on function calls, should we allow it to work on 
functions of three or more parameters?  In this case, the 
highest-dimension argument would be reduced to the dimension of the 
second highest, and then these two reduced together to match the third 
highest, and so on.

Of course, these questions raise one more: how easy or hard would these 
ideas be to implement?


Any thoughts?

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Feb 15 2005
next sibling parent reply xs0 <xs0 xs0.com> writes:
Nice job on the spec (especially the different dimensions handling). I 
do feel, however, that it should not be necessary to always create new 
arrays for result -- I'd say that if the left-hand side is non-null and 
of the appropriate size (i.e. exactly equal to size of result), it need 
not be reallocated. Consider just iterating through a list of vectors to 
find a specific one (for example, by multiplying each with some constant 
vector (i.e. dot product))- you wouldn't want a gazillion arrays created..

 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they be 
 handled?

I don't think so.. It would look weird and you can achieve the same by typing a+=1..
 Should we generalise the concept to function calls?  If so, I guess that 
 overload resolution would work in much the same way as for operations on 
 user-defined types.

Well, AFAIK, the whole point of having language-supported ops is run-time efficiency - you can otherwise easily wrap your arrays in classes like Vectors and Matrices and overload ops. Functions just don't make much sense in that respect (assuming you mean calling func(int, int) for each element of int[], int[]?)
 Of course, these questions raise one more: how easy or hard would these 
 ideas be to implement?

No idea :) If this is done, I'd also suggest adding a few util functions to arrays, like .min, .minIndex, .max, .maxIndex, .sum and .vectorLength (as in sqrt(a[0]^2 + a[1]^2 + ...)). Dot product is then simply (a*b).sum, and the compiler can detect this is being done and not even produce the result array.. xs0
Feb 15 2005
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
 Nice job on the spec (especially the different dimensions handling). I 
 do feel, however, that it should not be necessary to always create new 
 arrays for result -- I'd say that if the left-hand side is non-null and 
 of the appropriate size (i.e. exactly equal to size of result), it need 
 not be reallocated. Consider just iterating through a list of vectors to 
 find a specific one (for example, by multiplying each with some constant 
 vector (i.e. dot product))- you wouldn't want a gazillion arrays created..

That's basically what I said here:
 However, a quality implementation will optimize the evaluation of 
 the expression to eliminate temporaries where possible.


The essence of the essence is to keep the spec relatively simple and the semantics clear. <snip>
 If this is done, I'd also suggest adding a few util functions to arrays, 
 like .min, .minIndex, .max, .maxIndex, .sum and .vectorLength (as in 
 sqrt(a[0]^2 + a[1]^2 + ...)). Dot product is then simply (a*b).sum, and 
 the compiler can detect this is being done and not even produce the 
 result array..

I started to mention this idea before. D/21671 But calling them .min and .max is asking for confusion with the minimum and maximum allowable values of a type. Maybe .minValue and .maxValue, making them parallel with the Index counterparts? And if there is a tie for maximum or minimum, which index should minIndex and maxIndex return? The first? The last? Any old one? What would be worth any difference in computational cost? Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 15 2005
next sibling parent reply xs0 <xs0 xs0.com> writes:
 [snip - don't reallocate result array if it is already the 
 correct size (and not null, of course)]

That's basically what I said here:
 However, a quality implementation will optimize the evaluation of the 
 expression to eliminate temporaries where possible.



But it's not just an issue of optimization - if the spec says it must be allocated, you can't change the semantics of that by optimization. Consider two objects holding a reference to the same array; if one of them does a vector op with the result stored in the variable holding the reference and if the spec says you must reallocate, they mustn't have the same reference after the op is finished (or actually just after it starts), and that can't be optimized away (at least according to such spec)..
 But calling them .min and .max is asking for confusion with the minimum 
 and maximum allowable values of a type.  Maybe .minValue and .maxValue, 
 making them parallel with the Index counterparts?

or .minEl and .maxEl (El for element)? and perhaps min/maxElIndex would be more obvious, although verbosity doesn't seem to be preferred in D :)
 And if there is a tie for maximum or minimum, which index should 
 minIndex and maxIndex return?  The first?  The last?  Any old one?  What 
 would be worth any difference in computational cost?

When implemented by hand, this is usually the first matching index, I think, so that'd be an acceptable spec. Any old one is not very good - if it's the first (or the last), you can easily check for multiples with the slice syntax, otherwise you can't.. xs0
Feb 15 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
 That's basically what I said here:

 However, a quality implementation will optimize the evaluation of 
 the expression to eliminate temporaries where possible.



But it's not just an issue of optimization - if the spec says it must be allocated, you can't change the semantics of that by optimization.

Well, the spec doesn't say that it _must_ be allocated. The spec states what rather than how. So what matters is that the resulting behaviour is consistent.
 Consider two objects holding a reference to the same array; if one of 
 them does a vector op with the result stored in the variable holding the 
 reference and if the spec says you must reallocate, they mustn't have 
 the same reference after the op is finished (or actually just after it 
 starts), and that can't be optimized away (at least according to such 
 spec)..

The distinction between reference assignment and copying is already covered by the current spec. In-place vector modification follows directly from copying. For example, consider int[] x, y, z; ... y = x; ... y = z * 2; then y refers to a new array, and x still refers to the original. If you want to modify/repopulate y in place, you would do y[] = z * 2; by which you are indicating that you want to preserve the state of x and y referring to the same data. The same applies if op= is used instead of just =. Of course, if x is never used again and has never been assigned to anything else by reference, the two would be effectively equivalent. <snip>
 And if there is a tie for maximum or minimum, which index should 
 minIndex and maxIndex return?  The first?  The last?  Any old one?  
 What would be worth any difference in computational cost?

When implemented by hand, this is usually the first matching index, I think, so that'd be an acceptable spec. Any old one is not very good - if it's the first (or the last), you can easily check for multiples with the slice syntax, otherwise you can't..

How could I check for multiples with the slice syntax? Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 16 2005
parent reply xs0 <xs0 xs0.com> writes:
 Well, the spec doesn't say that it _must_ be allocated.  The spec states 
 what rather than how.  So what matters is that the resulting behaviour 
 is consistent.

Well, I read that in "a new array is created to hold the result"; however, having re-read the paragraph, I see that it can be read both ways.. It should be clear, though, and I think that it should explicitly say that the result array will be reused if it exists (is non-null) and is exactly the right size. If you want a new array, you can always use (a+b*c).dup or set result=null before the expression (or we can introduce a new syntax: result = new(a+b*c) :). You should have the option of reusing the array, though, so it should be supported in the spec.
 The distinction between reference assignment and copying is already 
 covered by the current spec.  In-place vector modification follows 
 directly from copying.  For example, consider
 
     int[] x, y, z;
     ...
     y = x;
     ...
     y = z * 2;
 
 then y refers to a new array, and x still refers to the original.  If 
 you want to modify/repopulate y in place, you would do
 
     y[] = z * 2;
 
 by which you are indicating that you want to preserve the state of x and 
 y referring to the same data.  The same applies if op= is used instead 
 of just =.

But why would you prefer a new array instead of in-place (when possible)? Considering arrays are references (with static arrays, you can't even create a new one, right?), it should be exactly the same if you say y=z*2 or x=z*2, as it is with objects, otherwise a whole lot of confusion and bugs will result from this, if you ask me..
 When implemented by hand, this is usually the first matching index, I 
 think, so that'd be an acceptable spec. Any old one is not very good - 
 if it's the first (or the last), you can easily check for multiples 
 with the slice syntax, otherwise you can't..

How could I check for multiples with the slice syntax?

int[] data; int idx=data.minElIndex; if (data[idx+1..length].minEl == data[idx]) { // more than one } xs0
Feb 16 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
<snip>
 Well, I read that in "a new array is created to hold the result"; 
 however, having re-read the paragraph, I see that it can be read both 
 ways.. It should be clear, though, and I think that it should explicitly 
 say that the result array will be reused if it exists (is non-null) and 
 is exactly the right size. If you want a new array, you can always use 
 (a+b*c).dup or set result=null before the expression (or we can 
 introduce a new syntax: result = new(a+b*c) :). You should have the 
 option of reusing the array, though, so it should be supported in the spec.

It's already supported. At the moment, int[] a, b; ... a = b; does a reference assignment. To reuse the result array, you use a[] = b; This would remain the same when array operations are involved.
 The distinction between reference assignment and copying is already 
 covered by the current spec.  In-place vector modification follows 
 directly from copying.  For example, consider

     int[] x, y, z;
     ...
     y = x;
     ...
     y = z * 2;

 then y refers to a new array, and x still refers to the original.  If 
 you want to modify/repopulate y in place, you would do

     y[] = z * 2;

 by which you are indicating that you want to preserve the state of x 
 and y referring to the same data.  The same applies if op= is used 
 instead of just =.

But why would you prefer a new array instead of in-place (when possible)?

Because you want x to still contain the same old data, of course.
 Considering arrays are references (with static arrays, you 
 can't even create a new one, right?), it should be exactly the same if 
 you say y=z*2 or x=z*2, as it is with objects, otherwise a whole lot of 
 confusion and bugs will result from this, if you ask me..

Of course not. Why would you declare two references to the same array in the same scope if (x === y) is going to remain true throughout? <snip>
 How could I check for multiples with the slice syntax?

int[] data; int idx=data.minElIndex; if (data[idx+1..length].minEl == data[idx]) { // more than one }

Oh yes, that makes perfect sense. But can the idiom be made efficient? A further idea might be to add minElCount and maxElCount. The compiler could optimise so that minEl, minElIndex and minElCount are calculated with one pass (a bit like common subexps) if two or more of them are used without the array being modified between retrievals. (Or maybe there could be something that returns a structure of value, index and count.) Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 16 2005
parent reply xs0 <xs0 xs0.com> writes:
Stewart Gordon wrote:
 To reuse the result array, you use
 
     a[] = b;
 
 This would remain the same when array operations are involved.

but a[]=b doesn't look like a can be [re]allocated (and it can be, if it is the wrong size or null). anyhow, it's not that important :)
 But why would you prefer a new array instead of in-place (when possible)?

Because you want x to still contain the same old data, of course.

But in this case you'd normally do x=y.dup; y=...; Except in case of array expressions, where you suggest x=y; y=...; // y is new
 Of course not.  Why would you declare two references to the same array 
 in the same scope if (x === y) is going to remain true throughout?

Who said anything about the same scope? Any anyhow, when I asked about your preference, I meant why do you think it's better to lose references as easily as possible, than the opposite? If there's only one reference, there is no argument - it's obviously better to reuse the array to avoid allocation/gc costs. If there's more than one reference, I'd definitely argue that the intention is usually to have them point to the same something (array or class) than to different somethings. If a snapshot of data is required, there's .dup (which is there in any case and serves exactly this purpose)
 Oh yes, that makes perfect sense.  But can the idiom be made efficient?
 
 A further idea might be to add minElCount and maxElCount.  The compiler 
 could optimise so that minEl, minElIndex and minElCount are calculated 
 with one pass (a bit like common subexps) if two or more of them are 
 used without the array being modified between retrievals.  (Or maybe 
 there could be something that returns a structure of value, index and 
 count.)

Sure, a struct would work, it might needlessly complicate things, though.. In most cases you just want the min/max value and checking for multiples is much rarer. I'd then prefer a .count(el) method which counts the number of occurences of _any_ value, not just min/max. The compiler can still easily optimize occurrences like data.count(data.maxEl) or even int firstMinIdx; int minValue; int reps=data.count(minValue=data[firstMinIdx=data.minElIndex]); while always avoiding counting and/or keeping min value/index when it's not necessary (and having these done as fast as possible is the whole point of compiler support - writing functions that do these things is trivial; and just consider how much faster a .count(1) can be on a bit[] than fetching each bit separately :). There's another problem with a struct - if you'd want the whole struct, it would have to be templatized to hold any kind of value, producing a new template instance (costing disk, memory, compilation time etc..) on each type of array where you'd want access to these properties. xs0
Feb 16 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
 Stewart Gordon wrote:
 
 To reuse the result array, you use
 
     a[] = b;
 
 This would remain the same when array operations are involved.

but a[]=b doesn't look like a can be [re]allocated (and it can be, if it is the wrong size or null). anyhow, it's not that important :)

What would be the sense of assigning in-place if the destination array is going to be either the wrong size or null? I see two motives for assigning in-place: - to keep the data in sync when there are multiple references to it. But this already isn't going to work if you need to keep resizing the array, unless you wrap the array in a pointer or class to make it work. - to save the overhead of memory allocation during calculations. But when the arrays can arbitrarily change in size, this overhead just comes back.
 But why would you prefer a new array instead of in-place (when 
 possible)?

Because you want x to still contain the same old data, of course.

But in this case you'd normally do x=y.dup; y=...;

Only if there are other references to the same data somewhere in the program, and I want x to break away so that I can modify it independently. Otherwise, the .dup is just wasteful.
 Except in case of array expressions, where you suggest
 
 x=y;
 y=...; // y is new

No "except" to it. The semantics of a = ... and a[] = ... are completely independent of the form of the right operand.
 Of course not.  Why would you declare two references to the same 
 array in the same scope if (x === y) is going to remain true 
 throughout?

Who said anything about the same scope? Any anyhow, when I asked about your preference, I meant why do you think it's better to lose references as easily as possible, than the opposite?

I'm not sure what you mean by this. My spec was written with three basic intentions: (a) to be well-defined (b) to keep D consistent within itself (c) to be backward compatible with the current assignment semantics Do you actually understand how my spec works? Let's look at some examples mechanically. Current D (semantics will be preserved): int[] x, y, z; ... z = x; assign x by reference to z z[] = x; copy the contents of x into z (requires that z and x are already the same size) z = x.dup; make a copy of x, and assign this copy by reference to z (The array previously referenced by z may still have references somewhere, or otherwise it would become eligible for GC.) With array ops: z = x + y; create an array of x[i] + y[i] assign this new array by reference to z z[] = x + y; create an array of x[i] + y[i] copy the contents of this new array in z (A decent compiler would implement this by adding the elements directly into z, bypassing the temporary array. But this is part of compiler optimisations, rather than of the basic semantics.) z = x * 2 + y; create an array of x[i] * 2 create an array of result[i] + y[i] assign this new result by reference to z (x * 2 is a temporary, and so the compiler can optimise it away.) z[] = x * 2 + y; create an array of x[i] * 2 create an array of result[i] + y[i] copy the contents of this new array into z (Here there are two temporaries: x * 2 and x * 2 + y.) See how they fit together now? <snip>
 Sure, a struct would work, it might needlessly complicate things, 
 though. In most cases you just want the min/max value and checking 
 for multiples is much rarer.

Indeed, there's no reason we can't have both the struct and plain minEl and maxEl. Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 17 2005
parent reply xs0 <xs0 xs0.com> writes:
 but a[]=b doesn't look like a can be [re]allocated (and it can be, if
 it is the wrong size or null). anyhow, it's not that important :)

What would be the sense of assigning in-place if the destination array is going to be either the wrong size or null?

that would, of course, make no sense :) that's why I don't like a[]=b+c; a[] implies that a is already allocated and the right size (you even state that yourself below).. With what you're suggesting, any code that cares about performance will look like: while (...) { if (a==null || a.length!=x.length) a=new int[x.length]; a[]=x+y; ... } And I'd like it to simply look like while (...) { a=x+y; ... } while having the same thing done performance..
 I see two motives for assigning in-place:
 
 - to keep the data in sync when there are multiple references to it.
 But this already isn't going to work if you need to keep resizing the 
 array, unless you wrap the array in a pointer or class to make it work.

I don't think most cases will be working with arrays of multiple sizes; when they do, my guess that the actual arrays will already be wrapped with their meta-data.. And like I suggested, you can alternatively leave other references alone by doing a=(x+y).dup where it's at least obvious you want the allocation done.
 - to save the overhead of memory allocation during calculations.  But
 when the arrays can arbitrarily change in size, this overhead just comes
 back.

True, but when the arrays don't arbitrarily change (like I said, IMO that is the majority), there's a chance to save a lot of overhead (with small arrays, memory allocation will easily take more time than calculations, so what's the point of having compiler support anyway then?).
 I'm not sure what you mean by this.  My spec was written with three 
 basic intentions:
 
 (a) to be well-defined
 (b) to keep D consistent within itself
 (c) to be backward compatible with the current assignment semantics

I don't see how either of these is broken by reusing the result array even when [] is not used. The spec can easily be well-defined by "if the result is assigned to an array reference which is non-null and of exactly the same dimensions as the result, its allocated memory will be reused; otherwise, a new array will be allocated to store the result." And that's far more similar to how slices work now (memory is reused, unless you extend a slice beyond the original size). As for consistency, consider: SomeClass a=new ...; SomeClass b=a; a+=5; // now, a is still the same object and a===b ---------- int[] a=new ...; int[] b=a; a+=5; // with my suggestion, a is still the same object and a===b // according to your spec a!==b OK, granted, reuse also breaks some consistency, but so does reallocating always.
 Do you actually understand how my spec works?

Completely :)
 Let's look at some  examples mechanically.
 
 Current D (semantics will be preserved):

These all stay the same, even if arrays are reused.
 With array ops:
 
     z = x + y;
 
         create an array of x[i] + y[i]
         assign this new array by reference to z

I really fail to see why a new array is necessary here, if it already exists. To me it looks the same, as if: int a=3; int* b=&a; a=2+3; // or even a+=2 if (*b==5) { // this is normal } else { // this is what you're suggesting with arrays } Now that I think about it, I don't even think the result should be completely new, instead its length can be adjusted. Then, you can allocate a big chunk of memory (to hold even your biggest inputs) and just use it without any allocation whatsoever. Basically, instead of what you wrote, z=x+y should translate to: -------------------------- assert(x.length=y.length); if (z==null) z=new int[x.length]; else if (z.length!=x.length) { // unless this is already checked z.length=x.length; } for(...) { ... } -------------------------- and z[]=x+y should translate to -------------------------- assert(z!=null && z.length==x.length && x.length==y.length); for(...) { ... } -------------------------- This means that z[] is expected to be non-null and the same size as x and y (and it is an error if it is not so), exactly like it is now when arrays are just copied (and that's the only current array op, afaik).
     z = x * 2 + y;
     z[] = x * 2 + y;

This is exactly the same case.
 <snip>
 
 Indeed, there's no reason we can't have both the struct and plain minEl 
 and maxEl.

Have you even read what I wrote? What type would the struct be? It'd have to be MinMaxValues!(int) or something. Like I said, that needlessly instantiates templates and having .count(value) is more useful as it can be used for things other than counting min/max values.. Actually, min/maxElIndex and count are all that is necessary, although data.minEl does looks better than data[data.minElIndex]. To finish this post, I'd like to reiterate again that having array ops is not important just because of prettier syntax but because of the speed the compiler can achieve by exploiting the knowledge it gets when you use such operations (e.g. the whole point of Expression templates in C++ is to have the compiler see what it is you want to do, rather than just seeing parts of it). I assume you agree. Now, if speed is important, you don't want to waste time allocating memory (it costs allocation, collection of the old array, breaks cache on processor etc. etc). So, it is better to reuse the array even when the programmer forgots to do so (or is too lazy to write the checks), as most probably, it was still the effect he wanted. If it is not, he can still easily create new arrays explicitly just by typing .dup, which is far less characters than those nulls and lengths checking :) Finally, consider all the potential users that D will lose without this. I bet that someone that wants to evaluate D will not read enough to know better and will write a=b+c instead of a[]=b+c. Running that in a loop (of, say, 5_000_000_000 iterations :) just to see performance will make him run away, as billions of new vectors will get allocated. Why is it so slow? Why did something so simple allocate MBs of memory? I thought native code was supposed to be fast, so I guess this D and its compiler really suck. I'd better learn Fortran! :) xs0
Feb 17 2005
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
<snip>
 that's why I don't like a[]=b+c; a[] implies that a is already allocated 
 and the right size (you even state that yourself below).. With what 
 you're suggesting, any code that cares about performance will look like:

Code that cares about performance would probably be written with at least some preconception of whether the array is going to be the same size.
 while (...) {
     if (a==null || a.length!=x.length)
         a=new int[x.length];
     a[]=x+y;
     ...
 }
 
 And I'd like it to simply look like
 
 while (...) {
     a=x+y;
     ...
 }

How about this? while (...) { a.length = x.length; a[] = x + y; } (Assuming that the contents of ... have some side effect on the length of x - otherwise how about making it even more efficient by putting the length assignment outside the loop?)
 while having the same thing done performance..
 
 I see two motives for assigning in-place:

 - to keep the data in sync when there are multiple references to it.
 But this already isn't going to work if you need to keep resizing the 
 array, unless you wrap the array in a pointer or class to make it work.

I don't think most cases will be working with arrays of multiple sizes;

Exactly. So it seems silly to convolute the language semantics for this minority of cases. <snip>
 True, but when the arrays don't arbitrarily change (like I said, IMO 
 that is the majority), there's a chance to save a lot of overhead (with 
 small arrays, memory allocation will easily take more time than 
 calculations, so what's the point of having compiler support anyway then?).

Exactly. Then one would use the [].
 I'm not sure what you mean by this.  My spec was written with three 
 basic intentions:

 (a) to be well-defined
 (b) to keep D consistent within itself
 (c) to be backward compatible with the current assignment semantics

I don't see how either of these is broken by reusing the result array even when [] is not used.

Where a is a dynamic array, a = ... by definition does a reference assignment. <snip>
 As for consistency, consider:
 
 SomeClass a=new ...;
 SomeClass b=a;
 
 a+=5;
 
 // now, a is still the same object and a===b

Yes, iff the class defines opAddAssign. (At least from what I can make of the spec.)
 ----------
 
 int[] a=new ...;
 int[] b=a;
 
 a+=5;
 
 // with my suggestion, a is still the same object and a===b
 // according to your spec a!==b
 
 OK, granted, reuse also breaks some consistency, but so does 
 reallocating always.

Hmm....
 Do you actually understand how my spec works?

Completely :)
 Let's look at some examples mechanically.

 Current D (semantics will be preserved):

These all stay the same, even if arrays are reused.

Just think about it. Suppose z = x; does a reference assignment, but z = x + y; does an in-place modification. Then you are changing the semantics of the = operator by changing the right operand. And you're not even changing the type of the right operand, only its form. This is a cause of confusion that is bound to lead to bugs, never mind trying to generalise the semantics of = applied to arbitrary expressions.
 With array ops:

     z = x + y;

         create an array of x[i] + y[i]
         assign this new array by reference to z

I really fail to see why a new array is necessary here, if it already exists. To me it looks the same, as if:

Because this is what the programmer asked for. <snip>
 So, it is better to reuse the array even when the programmer 
 forgots to do so (or is too lazy to write the checks), as most probably, 
 it was still the effect he wanted. If it is not, he can still easily 
 create new arrays explicitly just by typing .dup, which is far less 
 characters than those nulls and lengths checking :)

As has been said plenty of times, a program is only as good as the person who wrote it. And so if the programmer is lazy, he/she/it shouldn't be too surprised if the program is lazy. Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 17 2005
parent reply xs0 <xs0 xs0.com> writes:
 How about this?
 
     while (...) {
         a.length = x.length;
         a[] = x + y;
     }

Again, why would this not be the default case (i.e. done by the compiler?) Also consider that if a is int[][][], you'll also need to allocate all the int[][]'s and really really quickly that a=x+y becomes totally invisible, all because you didn't want to change your spec (which is completely fine, aside from the issue we're arguing over). And I'd guess that only the outer array gets reused with a[]=x+y? So what would you have the user type then? a[][][]=x+y? That's probably not even legal.. And what if a is of type T as in template(T)?
 Exactly.  So it seems silly to convolute the language semantics for this 
 minority of cases.

There's nothing convoluted.. both a= and a[]= are defined only for things that are currently supported. x+y on arrays is not defined currently, so you can define = for array ops as you want. = doesn't do the same for structs and classes either, and no one seems to mind (even though they're really similar otherwise).
 <snip>

you tend to snip my best arguments, methinks :P
 Where a is a dynamic array, a = ... by definition does a reference 
 assignment.

I don't agree there's any such definition, but let's not go there. What if it is a static array, you'll force the users to type [] every time? When they (this time really) obviously want it in-place. Or would you have it work differently in each case?
 Yes, iff the class defines opAddAssign.  (At least from what I can make 
 of the spec.)

obviously.. the point was, however, that a and b still point to the same object. a+=b acts like in-place for classes, is in-place for primitives, so why would it be different for arrays?
 int[] a=new ...;
 int[] b=a;

 a+=5;

 // with my suggestion, a is still the same object and a===b
 // according to your spec a!==b

 <snip>

Hmm....

Hmm what?
 Just think about it.  Suppose
 
     z = x;
 
 does a reference assignment, but
 
     z = x + y;
 
 does an in-place modification.  

Maybe, but z+=x certainly indicates in-place modification, so it would seem you can't have it totally consistent anyhow.. If that is the case I'd certainly prefer in-place whenever possible.
 <snip>

The semantics of the = operator are not always the same even without array ops. x+y is also not defined to be anything in particular (currently). Why do you keep insisting that they are? Considering that your spec itself says that the effect is the same as if you wrote the indicated code, you can just change the expansion in the spec to not reallocate when not necessary and there you have it.
     z = x + y;

exists. To me it looks the same, as if:

Because this is what the programmer asked for.

No, he didn't. I can't see from that statement that the programmer wishes to allocate a new z. _You_ say that is what he asked for; all I can see is he wanted z to contain the sum and if he really wanted a new array, he'd write (x+y).dup. It's the same with slices - sometimes you get a completely new memory space, and sometimes you don't, and the default is what's most efficient ; how come it doesn't bother anyone that setting subslice.length can have the effect of losing the reference to original memory? And that's exactly the same thing - you get to reuse memory when possible, and other cases are handled as well as they can be automatically. When you're certain you want a stand-alone copy, you use .dup, and that's it.
 <snip on newbies>


Well, considering that everyone is a newbie at first, and that D does need more users, saying "they suck so why deal with them" is not helping anything.. class Sumator(T) { T sum; void set(T val) { sum=val; } void add(T val) { sum+=val; } T get() { return sum; } } Newbie or not, Sumator!(int) is fine, while Sumator!(int[]) totally sucks (unless, of course, you prefer code that runs really slowly) and you can't even make it better, unless you specialize it for all array dimensions in which you use it, which makes using a template pointless.. For what reason? And this is just a trivial case... To sum up, you'd have stuff defined like this: a = b + c; // create a new array (really bad performance-wise, // but still desired behavior like 1% of time) a[] = b + c; // fail if a is null/wrong size, otherwise work in-place // useful when the operands are really known to be the same // size all the time (that would be like 20% of time) a = (b+c).dup; // same as a=b+c (and as useless) And I'd have it defined like this: a = b + c; // work in-place when possible (useful like 80% of the time) a[] = b + c; // same as above a = (b+c).dup; // same as above So you'd have two syntaxes for the same crappy-most-of-the-time behavior, and you'd have the programmer write additional code for the most common case (for handling arbitrary lengths with memory reuse). Would you agree that is a good summary? xs0
Feb 17 2005
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
 How about this?

     while (...) {
         a.length = x.length;
         a[] = x + y;
     }

Again, why would this not be the default case (i.e. done by the compiler?) Also consider that if a is int[][][], you'll also need to allocate all the int[][]'s and really really quickly that a=x+y becomes totally invisible, all because you didn't want to change your spec (which is completely fine, aside from the issue we're arguing over). And I'd guess that only the outer array gets reused with a[]=x+y? So what would you have the user type then? a[][][]=x+y? That's probably not even legal.. And what if a is of type T as in template(T)?

At the moment, a[][][] is just the same as a[], since it's just slicing the outermost dimension again and again. Well done, you do have a point there. Maybe someone else has an idea....
 Exactly.  So it seems silly to convolute the language semantics for 
 this minority of cases.

There's nothing convoluted.. both a= and a[]= are defined only for things that are currently supported. x+y on arrays is not defined currently, so you can define = for array ops as you want. = doesn't do the same for structs and classes either, and no one seems to mind (even though they're really similar otherwise).

If you simply mean that structs are contained by value and classes by reference, then the semantics _are_ the same: copy what is represented by the right operand into the piece of memory represented by the left operand, whether this is a value or a reference.
 <snip>

you tend to snip my best arguments, methinks :P
 Where a is a dynamic array, a = ... by definition does a reference 
 assignment.

I don't agree there's any such definition, but let's not go there. What if it is a static array, you'll force the users to type [] every time? When they (this time really) obviously want it in-place. Or would you have it work differently in each case?

I wouldn't consider it at all obvious that someone who types z = x + y; really meant z[] = x + y; If I wrote the former, I would consider it obvious that it is what I meant. <snip>
 a+=b acts like in-place for classes,

Like I said, only if the class defines opAddAssign. Otherwise, it's equivalent to a = a + b.
 is in-place for primitives,

Because primitives are contained by value.
 so why would it be different for arrays?

My thought is that it would be just like a class on which opAddAssign isn't defined.
 int[] a=new ...;
 int[] b=a;

 a+=5;

 // with my suggestion, a is still the same object and a===b
 // according to your spec a!==b

 <snip>

Hmm....

Hmm what?

Good question....
 Just think about it.  Suppose

     z = x;

 does a reference assignment, but

     z = x + y;

 does an in-place modification.  

Maybe, but z+=x certainly indicates in-place modification, so it would seem you can't have it totally consistent anyhow.. If that is the case I'd certainly prefer in-place whenever possible.

Again, there's always the option of using z[] += x.
 <snip>

The semantics of the = operator are not always the same even without array ops. x+y is also not defined to be anything in particular (currently). Why do you keep insisting that they are? Considering that your spec itself says that the effect is the same as if you wrote the indicated code, you can just change the expansion in the spec to not reallocate when not necessary and there you have it.

It already does, in the second paragraph of the first heading. Except that it appears your use of the word "necessary" might be inconsistent with mine.
     z = x + y;

I really fail to see why a new array is necessary here, if it already exists. To me it looks the same, as if:

Because this is what the programmer asked for.

No, he didn't. I can't see from that statement that the programmer wishes to allocate a new z.

The basic request is that any other references to the data pointed to by z before the assignment will still be pointing to the same data as they were. <snip>
 To sum up, you'd have stuff defined like this:
 
 a = b + c; // create a new array (really bad performance-wise,
            // but still desired behavior like 1% of time)
 a[] = b + c; // fail if a is null/wrong size, otherwise work in-place
              // useful when the operands are really known to be the same
              // size all the time (that would be like 20% of time)
 a = (b+c).dup; // same as a=b+c (and as useless)
 
 And I'd have it defined like this:
 
 a = b + c; // work in-place when possible (useful like 80% of the time)
 a[] = b + c; // same as above
 a = (b+c).dup; // same as above
 
 So you'd have two syntaxes for the same crappy-most-of-the-time 
 behavior, and you'd have the programmer write additional code for the 
 most common case (for handling arbitrary lengths with memory reuse).
 
 Would you agree that is a good summary?

Good at summarising, yes. Good at convincing me, I'm not sure. Maybe we should disallow a = b + c; altogether, instead requiring the coder to specify explicitly a[] = b + c; or a = (b + c).dup; Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 18 2005
parent reply xs0 <xs0 xs0.com> writes:
 At the moment, a[][][] is just the same as a[], since it's just slicing 
 the outermost dimension again and again.  Well done, you do have a point 
 there.  Maybe someone else has an idea....

Wow .. You actually agreed with something :)
 If you simply mean that structs are contained by value and classes by 
 reference, then the semantics _are_ the same: copy what is represented 
 by the right operand into the piece of memory represented by the left 
 operand, whether this is a value or a reference.

Well, I guess we understand semantics differently.. This is how I see it for the = operator: int a=b; // copy value of b to a struct a=b; // copy contents of b to a (might seem the same as // the previous case, but struct is a composite // and furthermore is different from a class) class a=b; // make a point to b array a=b; // make a point to b array a[]=(int)b; // set all elements of a to b array a[]=b; // copy data from b to a array a=b[c..d]; // create a new slice with same contents as b[c..d] // (memory is reused when possible, so watch out) array a=b.dup; // create a copy of b and make a point to it These explanations are not all equal, so you can't say that the semantics are the same. IMHO, of course :)
 I wouldn't consider it at all obvious that someone who types
 
     z = x + y;
 
 really meant
 
     z[] = x + y;
 
 If I wrote the former, I would consider it obvious that it is what I meant.

If you knew that all the vars are arrays, that z=x+y means that a new array gets created and you wrote that, you would indeed consider it obvious. It's not obvious in itself; if x, y and z are ints, you don't expect a new int to get allocated.. You also don't expect (*a)=2+3 to change the pointer, yet it is just as reference to int, as the above z is reference to array..
 Like I said, only if the class defines opAddAssign.  Otherwise, it's 
 equivalent to a = a + b.

But regardless of whether defined opAddAssign is defined, when you see a+=b, you interpret it as in-place, even though behind the scenes something else happens. The difference is also that you write (or not) opAddAssign yourself, so you can change what is going on, which is not the case for arrays.
 My thought is that it would be just like a class on which opAddAssign 
 isn't defined.

But why would it not be like a class on which opAddAssign _is_ defined? At this point it can go either way and I don't see any real benefits in your case and I do see real benefits in my case.
 Again, there's always the option of using z[] += x.

If you know in advance z is an array (and possibly even an array of dimension exactly 1). What would you write in all other cases?
 It already does, in the second paragraph of the first heading.  Except 
 that it appears your use of the word "necessary" might be inconsistent 
 with mine.

Well, the whole spec could just as easily be defined only with loop expansions of such expressions and there'd be no need to create new arrays and then optimize them away (isn't that kind of pointless?). (a+b*c).dup can be defined as a special case that actually creates a new array and there you have it. The only issue is with what checks are done, where I again suggest: a=b+c (or any other expression with arrays) means that a is made to be a valid array to hold the result of b+c (the "it just works in 99% of cases" principle); if it already is, nothing is done; if it's null, it gets allocated; if it's the wrong size, its .length is adjusted. this is really useful functionality and saves a lot of typing, errors, CPU time and whatnot, even though it doesn't change a as reference, only a's length/values/nullness.. a[]=b+c, on the other hand, means that a is already expected to be the right size and it is an error, if it is not (like it currently is with a[]=b or a[]=b[]). Just like a=b+c except an added assert(a!=null && a.length=b.length).
 No, he didn't. I can't see from that statement that the programmer 
 wishes to allocate a new z.

The basic request is that any other references to the data pointed to by z before the assignment will still be pointing to the same data as they were.

Why is this a basic request? Slices also have perfectly useful behavior, but it's not required that either they point to the same memory or to different memory; it works in a way that's efficient (memory gets reused) yet still user-friendly (if reuse is not possible, it automatically gets reallocated). Why not have the same with array ops? Automatically being the keyword - so it just works.
 Maybe we should disallow
 
     a = b + c;
 
 altogether, instead requiring the coder to specify explicitly
 
     a[] = b + c;
 
 or
 
     a = (b + c).dup;

Well, that's an option, but I don't think it's a good one, unless we get a new operator that does what I'm trying to suggest (automatic handling of stuff that really can be automatically handled). Like <== or something, and it can be equal to = in all cases but arrays (so templates are possible). With templates you can then use = if you don't expect arrays (as in you feel that they will not work), and <== if you do expect/support them. xs0
Feb 18 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
<snip>
 I wouldn't consider it at all obvious that someone who types

     z = x + y;

 really meant

     z[] = x + y;

 If I wrote the former, I would consider it obvious that it is what I 
 meant.

If you knew that all the vars are arrays, that z=x+y means that a new array gets created and you wrote that, you would indeed consider it obvious. It's not obvious in itself; if x, y and z are ints, you don't expect a new int to get allocated.. You also don't expect (*a)=2+3 to change the pointer, yet it is just as reference to int, as the above z is reference to array..

Of course not, here it's obvious that I'm changing the contents of an already allocated piece of memory. Just like it is when I use [] to force in-place assignment. <snip>
 My thought is that it would be just like a class on which opAddAssign 
 isn't defined.

But why would it not be like a class on which opAddAssign _is_ defined? At this point it can go either way and I don't see any real benefits in your case and I do see real benefits in my case.
 Again, there's always the option of using z[] += x.

If you know in advance z is an array (and possibly even an array of dimension exactly 1). What would you write in all other cases?

Again, you have a point or two.
 It already does, in the second paragraph of the first heading.  Except 
 that it appears your use of the word "necessary" might be inconsistent 
 with mine.

Well, the whole spec could just as easily be defined only with loop expansions of such expressions and there'd be no need to create new arrays and then optimize them away (isn't that kind of pointless?).0

This would seem a step back towards the poorly-defined old spec. But I'm not sure. Moreover, we should certainly be able to pass an array expression to a function that expects an array. <snip>
 The basic request is that any other references to the data pointed to 
 by z before the assignment will still be pointing to the same data as 
 they were.

Why is this a basic request? Slices also have perfectly useful behavior, but it's not required that either they point to the same memory or to different memory; it works in a way that's efficient (memory gets reused) yet still user-friendly (if reuse is not possible, it automatically gets reallocated). Why not have the same with array ops? Automatically being the keyword - so it just works.

Yes, it should reuse memory where possible. But only where the compiler can determine that this doesn't affect the behaviour of the program - and hence again this would be part of optimisation rather than language spec. After all, D is designed with the optimisation technology of modern compilers in mind.
 Maybe we should disallow

     a = b + c;

 altogether, instead requiring the coder to specify explicitly

     a[] = b + c;

 or

     a = (b + c).dup;

Well, that's an option, but I don't think it's a good one, unless we get a new operator that does what I'm trying to suggest (automatic handling of stuff that really can be automatically handled). Like <== or something, and it can be equal to = in all cases but arrays (so templates are possible). With templates you can then use = if you don't expect arrays (as in you feel that they will not work), and <== if you do expect/support them.

Yes, generic programming implications are something else to consider. Which option are you suggesting that <== should be equivalent to? The "don't expect arrays" case could also cover instances that don't do arithmetic on the parameter type at all, and as such will work implicitly. Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 18 2005
parent reply xs0 <xs0 xs0.com> writes:
 Of course not, here it's obvious that I'm changing the contents of an 
 already allocated piece of memory.  Just like it is when I use [] to 
 force in-place assignment.

Well, I should've written {{ int a; int *b=&a; a=2+3; }} and b still points to a.. But anyhow, we now seem to agree that there are issues, so let's leave = alone :)
 Yes, it should reuse memory where possible.  But only where the compiler 
 can determine that this doesn't affect the behaviour of the program - 
 and hence again this would be part of optimisation rather than language 
 spec.

Well, but this is exactly the problem - the compiler can't determine that (except for inner operations, as in + and * in (a+b*c) and possibly for local variables that are created within the function and don't get passed as parameters and don't get sliced or casted to pointers or whatnot..) But that totally ruins reuse between function calls, for example, and also a bunch of other cases.. (even if no other method touches an object var and it is protected from the outside world, you can still extend the class and write a method that does, for example). I don't believe it is a problem to reuse even when it could affect the behavior, if it is documented. First, in many cases it will not really affect the behavior, even if the compiler can't determine that. Second, in a lot of other cases, this will actually be desired. In the few cases it will not be desired, the programmer will know to take care of that, as it can be the first thing written in the description of <== operator and assuming that a=b+c is made illegal for arrays, all cases are covered.. And, for the umpteenth time, slices also reuse memory without causing major problems :)
 Yes, generic programming implications are something else to consider. 
 Which option are you suggesting that <== should be equivalent to?

a <== expr means variable a gets allocated (if null) or resized (if non-null but wrong size) so it matches expr, including all potential dimensions. This also means that it will be useful as a replacement of a[]=b (as in a<==b), when you'll want lazy allocation and/or automatic handling of different sizes.. For non-array types, it behaves exactly like =.
 The "don't expect arrays" case could also cover instances that don't do 
 arithmetic on the parameter type at all, and as such will work implicitly.

Well, if all you do is a=b, it will still work with arrays, although if that's really all you do, it shouldn't be an issue? Of course, there's also op=. Which one should it be ( op= or op<== ), assuming not both get supported (which will probably not happen.. think about <=<== or >>><== or !<>=<== :)? I'd vote for <==, as op= does mean in-place in the case where it is normally used (i.e. primitives). To further improve templating it might also make sense to extend .dup to value-based types (to handle cases where you do want copies made). Unlike most other properties, it does make sense on all types - it says I want a copy. On primitive types and structs, .dup would just be their value itself (and that is their copy already). Classes can also implement .dup to return their clone. This only leaves a slight inefficiency when you do something like {{ a=(b+c*d).dup }} with classes, but when this becomes an issue, you can handle all class types with a single template specialization for Object. With all that, .minIdx, .maxIdx, .sum and .count and decent compiler support, some really neat things could be done, effective both in amount of code needed and resulting speed. xs0
Feb 18 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:
 Of course not, here it's obvious that I'm changing the contents of an 
 already allocated piece of memory.  Just like it is when I use [] to 
 force in-place assignment.

Well, I should've written {{ int a; int *b=&a; a=2+3; }} and b still points to a.. But anyhow, we now seem to agree that there are issues, so let's leave = alone :)

 Well, if all you do is a=b, it will still work with arrays, although if 
 that's really all you do, it shouldn't be an issue?
 
 Of course, there's also op=. Which one should it be ( op= or op<== ), 
 assuming not both get supported (which will probably not happen.. think 
 about <=<== or >>><== or !<>=<== :)? I'd vote for <==, as op= does mean 
 in-place in the case where it is normally used (i.e. primitives).
 
 To further improve templating it might also make sense to extend .dup to 
 value-based types (to handle cases where you do want copies made). 

What if we had an assignment operator that forces copy? That would certainly stand out better in the code than a .dup hidden somewhere in a long expression. (I know that mentioning this probably causes a riot (if anyone else is actually still reading this thread)), so the case would have to be really strong. Is it?
 Unlike most other properties, it does make sense on all types - it says 
 I want a copy. On primitive types and structs, .dup would just be their 
 value itself (and that is their copy already). Classes can also 
 implement .dup to return their clone. This only leaves a slight 
 inefficiency when you do something like {{ a=(b+c*d).dup }} with 
 classes, but when this becomes an issue, you can handle all class types 
 with a single template specialization for Object.
 
 
 With all that, .minIdx, .maxIdx, .sum and .count and decent compiler 
 support, some really neat things could be done, effective both in amount 
 of code needed and resulting speed.
 
 
 xs0

Feb 21 2005
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
xs0 wrote:
<snip>
 And I'd guess that only the outer array gets reused with a[]=x+y? So 
 what would you have the user type then? a[][][]=x+y? That's probably not 
 even legal.. And what if a is of type T as in template(T)?

Just thinking about this, at the moment we don't seem to have a means of deep in-place copying of nested dynamic arrays. If we're going to support deep in-place assignment when doing arithmetic, we should have a corresponding means of simply copying an array in this way. And so I guess the syntax would correspond. <snip>
 I don't agree there's any such definition, but let's not go there. What 
 if it is a static array, you'll force the users to type [] every time? 
 When they (this time really) obviously want it in-place. Or would you 
 have it work differently in each case?

Oops, I kind of slipped in my last reply. With static arrays, always assigning by value. I didn't get round to covering static arrays in my proposal, but figured it would be straightforward. Indeed, when assigning to a static array, obviously it would be in place, since static arrays are by value. And if you have int[6][] x = new int[6][42]; x[] = ...; then the assignment would be fully in place. But at least one question remains: how should assignment of an expression on static arrays to a dynamic array be handled? (Is this a reference assignment at the moment? I should expect so.) Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 18 2005
parent Stewart Gordon <smjg_1998 yahoo.com> writes:
Stewart Gordon wrote:
 xs0 wrote:
 <snip>
 
 And I'd guess that only the outer array gets reused with a[]=x+y? So 
 what would you have the user type then? a[][][]=x+y? That's probably 
 not even legal.. And what if a is of type T as in template(T)?

Just thinking about this, at the moment we don't seem to have a means of deep in-place copying of nested dynamic arrays. If we're going to support deep in-place assignment when doing arithmetic, we should have a corresponding means of simply copying an array in this way. And so I guess the syntax would correspond.

Just thinking about it again. Is it really worth having deep in-place assignment, whether doing arithmetic or simple copying? Or might it just as well wait until we get true multidimensional arrays? Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 21 2005
prev sibling parent "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 17 Feb 2005 17:29:44 +0100, xs0 <xs0 xs0.com> wrote:
 but a[]=b doesn't look like a can be [re]allocated (and it can be, if
 it is the wrong size or null). anyhow, it's not that important :)

is going to be either the wrong size or null?

that would, of course, make no sense :) that's why I don't like a[]=b+c; a[] implies that a is already allocated and the right size (you even state that yourself below).. With what you're suggesting, any code that cares about performance will look like: while (...) { if (a==null || a.length!=x.length) a=new int[x.length]; a[]=x+y; ... } And I'd like it to simply look like while (...) { a=x+y; ... } while having the same thing done performance..

Doesn't array bounds checking handle this? With bounds checking on, it will be handled at runtime with an exception. With it off, it will be fast as it has no checking. Regan
Feb 17 2005
prev sibling parent xs0 <xs0 xs0.com> writes:
One more thing I just thought of - there's quite a difference between these:

a = b;  // copy _reference_
a = -b; // copy _contents_ and negate

Which might not be good.. However, using slice syntax doesn't quite look 
that good:

a[] = b[] + c[] * d[]; // abcd are somewhat lost in there

Perhaps a slightly modified syntax could be used? Something like

a <-- b + c * d;

I'm sure not many people will like that :) It does clearly differentiate 
between the usual meaning of = and the looping on elements done with 
arrays, though.. Any thoughts?


xs0
Feb 15 2005
prev sibling parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Tue, 15 Feb 2005 19:15:45 +0100, xs0 <xs0 xs0.com> wrote:
 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they be  
 handled?

I don't think so.. It would look weird and you can achieve the same by typing a+=1..

I think they should be allowed, I don't think they look weird, and I think they're useful eg. int[] a; int[] b; int[] c; b[] = a[]--; //assigns b[x] = a[x], then a[x] = a[x]-1; b[] = c[] + a[]--; //assigns b[x] = c[x] + a[x], then a[x] = a[x]-1; Basically it's done as 2 operations, same as: int i; int j; j = i--; Regan
Feb 15 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Regan Heath wrote:
 On Tue, 15 Feb 2005 19:15:45 +0100, xs0 <xs0 xs0.com> wrote:
 
 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they 
 be  handled?



 I think they should be allowed, I don't think they look weird, and I 
 think  they're useful eg.
 
 int[] a;
 int[] b;
 int[] c;
 
 b[] = a[]--;       //assigns b[x] = a[x], then a[x] = a[x]-1;
 b[] = c[] + a[]--; //assigns b[x] = c[x] + a[x], then a[x] = a[x]-1;

If you've got the []s to indicate in-place assignment, then this makes sense. But what about b = a--; ? Two possible interpretations: b = a; a = a.dup; foreach (inout x; a) x--; or b = a.dup; foreach (inout x; a) x--; Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 16 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Wed, 16 Feb 2005 12:58:46 +0000, Stewart Gordon <smjg_1998 yahoo.com>  
wrote:
 Regan Heath wrote:
 On Tue, 15 Feb 2005 19:15:45 +0100, xs0 <xs0 xs0.com> wrote:

 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they  
 be  handled?



 I think they should be allowed, I don't think they look weird, and I  
 think  they're useful eg.
  int[] a;
 int[] b;
 int[] c;
  b[] = a[]--;       //assigns b[x] = a[x], then a[x] = a[x]-1;
 b[] = c[] + a[]--; //assigns b[x] = c[x] + a[x], then a[x] = a[x]-1;

If you've got the []s to indicate in-place assignment, then this makes sense.

Which is exactly why I had them :)
 But what about

      b = a--;

 ?

I probably should have thought about those too...
 Two possible interpretations:

      b = a;
      a = a.dup;
      foreach (inout x; a) x--;

Not this, because "b = a" is a copy i.e. "b = a.dup"
 or

      b = a.dup;
      foreach (inout x; a) x--;

This is probably the most sensible. or Error: postdecrement is not allowed on 'a' did you mean 'a[]' Regan
Feb 16 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Regan Heath wrote:
 On Wed, 16 Feb 2005 12:58:46 +0000, Stewart Gordon 
 <smjg_1998 yahoo.com>  wrote:

      b = a;
      a = a.dup;
      foreach (inout x; a) x--;

Not this, because "b = a" is a copy i.e. "b = a.dup"

What do you mean? b = a on dynamic arrays is, by definition, a reference assignment.
 or

      b = a.dup;
      foreach (inout x; a) x--;

This is probably the most sensible.

And it's also consistent with what can be done with opPostInc and opPostDec on a class. Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 17 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 17 Feb 2005 11:10:13 +0000, Stewart Gordon <smjg_1998 yahoo.com>  
wrote:
 Regan Heath wrote:
 On Wed, 16 Feb 2005 12:58:46 +0000, Stewart Gordon  
 <smjg_1998 yahoo.com>  wrote:

      b = a;
      a = a.dup;
      foreach (inout x; a) x--;


What do you mean? b = a on dynamic arrays is, by definition, a reference assignment.

Gargh, you're totally right. If "b = a;" is: - b reference a If "a--;" is: - create a new array - set new[x] = a[x] - 1; Then "b = a--;" is: - b reference a - create new array - set new[x] = a[x] - 1 - a reference new which is exactly what you had above :)
 or

      b = a.dup;
      foreach (inout x; a) x--;


And it's also consistent with what can be done with opPostInc and opPostDec on a class.

At the time I was thinking it was consistent with: int i = 5; int j; j = i--; The result in both cases is however the same, b contains what a did, and a contains a[x]-1, eg.
      b = a;
      a = a.dup;
      foreach (inout x; a) x--;



- b reference a - create new array - new[x] = a[x] - 1 - a reference new ----
      b = a.dup;
      foreach (inout x; a) x--;



- b reference a.dup - a[x] = a[x] - 1 ---- I think the top one is more consistent with D's reference semantics, the bottom one is consistent with D's value type semantics i.e. it's what happens with ints. So I reckon the top one is the 'correct' one. Regan
Feb 17 2005
parent "Regan Heath" <regan netwin.co.nz> writes:
On Fri, 18 Feb 2005 11:55:09 +1300, Regan Heath <regan netwin.co.nz> wrote:
 On Thu, 17 Feb 2005 11:10:13 +0000, Stewart Gordon <smjg_1998 yahoo.com>  
 wrote:
 Regan Heath wrote:
 On Wed, 16 Feb 2005 12:58:46 +0000, Stewart Gordon  
 <smjg_1998 yahoo.com>  wrote:

      b = a;
      a = a.dup;
      foreach (inout x; a) x--;


What do you mean? b = a on dynamic arrays is, by definition, a reference assignment.

Gargh, you're totally right. If "b = a;" is: - b reference a If "a--;" is: - create a new array - set new[x] = a[x] - 1;

forgot - a reference new
 Then "b = a--;" is:
   - b reference a
   - create new array
   - set new[x] = a[x] - 1
   - a reference new

 which is exactly what you had above :)

 or

      b = a.dup;
      foreach (inout x; a) x--;


And it's also consistent with what can be done with opPostInc and opPostDec on a class.

At the time I was thinking it was consistent with: int i = 5; int j; j = i--; The result in both cases is however the same, b contains what a did, and a contains a[x]-1, eg.
      b = a;
      a = a.dup;
      foreach (inout x; a) x--;



- b reference a - create new array - new[x] = a[x] - 1 - a reference new ----
      b = a.dup;
      foreach (inout x; a) x--;



- b reference a.dup - a[x] = a[x] - 1 ---- I think the top one is more consistent with D's reference semantics, the bottom one is consistent with D's value type semantics i.e. it's what happens with ints. So I reckon the top one is the 'correct' one. Regan

Feb 17 2005
prev sibling next sibling parent reply pragma <pragma_member pathlink.com> writes:
In article <cussnt$fe8$1 digitaldaemon.com>, Stewart Gordon says...
Should we generalise the concept to function calls?  If so, I guess that 
overload resolution would work in much the same way as for operations on 
user-defined types.

That makes sense to me. Just allow arrays of function pointers, delegates and anything with opCall() defined to be callable en masse. At a minimum, it would make event-based (1 to n messaging) programming a snap. I would also think that it could be extended to object members and methods as well?
 interface Foo{
  int foo();
  void bar();
 }

 Foo[] test;
 test.foo(); // calls .foo() for each object
 int[] result = test.bar(); // calls .bar() for each object, result is an array.

Seems exotic at first, but it does lend to D's mantra of being 'context free'.
If we do allow it on function calls, should we allow it to work on 
functions of three or more parameters?  In this case, the 
highest-dimension argument would be reduced to the dimension of the 
second highest, and then these two reduced together to match the third 
highest, and so on.

I'm not sure I follow you here. Sounds like you're talking about using the function syntax for mapping to the array dimension space... is this correct?
Of course, these questions raise one more: how easy or hard would these 
ideas be to implement?

Not hard, but would require a certain familiarity with the frontend that only Walter and an intrepid few programmers share. There are also some holes, that would have to be handled, if one adhered to just this spec by itself. It would be much easier to implement the above as templates first (as a proof-of-concept) simply because each of your examples are merely transforms based on type and dimension. It could help with refining the spec. One question: What about associative arrays? - EricAnderton at yahoo
Feb 15 2005
next sibling parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
Foo[] test;
test.foo(); // calls .foo() for each object
int[] result = test.bar(); // calls .bar() for each object, result is an array.

Seems exotic at first, but it does lend to D's mantra of being 'context free'.

That, and the events, looks very nice in my opinion. Then again, would it conflict with the current array overloading? int[] bar(Foo[] test); That's the only flaw I can see in it at this point. But it would make arrays of objects much more useful imho.
 One question: What about associative arrays?

Well, if the two had matching sets of keys it might be possible: int[int] x, y; x = -y; foreach (int /* or char, etc. */ k; x.keys) x[k] = -y[k]; The question is, does this have to check that the keys all exist in both arrays, too? Meaning, x.keys.length == y.keys.length and that none of the evaluations cause an exception? Seems rather dangerous, to me... -[Unknown]
Feb 15 2005
prev sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
pragma wrote:
<snip>
 That makes sense to me.  Just allow arrays of function pointers, delegates and 
 anything with opCall() defined to be callable en masse.  
 
 At a minimum, it would make event-based (1 to n messaging) programming a snap.
 
 I would also think that it could be extended to object members and methods as 
 well?
 
 interface Foo{
     int foo();
     void bar();
 }

 Foo[] test;
 test.foo(); // calls .foo() for each object
 int[] result = test.bar(); // calls .bar() for each object, result is an array.


That would lead to such troubles as class Foo { int length; } Foo[] test; and then is test.length the length of the array, or an array of lengths of the Foo objects? <snip>
 I'm not sure I follow you here.  Sounds like you're talking about using the 
 function syntax for mapping to the array dimension space... is this correct?

I'm not sure I follow you either. You seem to be talking about the ability to call arrays of functions. I was actually talking about the ability to pass array arguments to functions defined with scalar parameters. <snip>
 One question: What about associative arrays?

Good question. I guess we could extend the concept to AAs. An operation on an AA would return a new AA in which the keys remain the same and the operation is applied to the values. Trying to do it on binary ops, let alone functions of three or more parameters, would require that the two arrays have the same set of keys ... but is this likely to happen in the real world? Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 16 2005
parent reply pragma <pragma_member pathlink.com> writes:
In article <cuvgv4$4fd$1 digitaldaemon.com>, Stewart Gordon says...
pragma wrote:
 Foo[] test;
 test.foo(); // calls .foo() for each object
 int[] result = test.bar(); // calls .bar() for each object, result is an array.


That would lead to such troubles as class Foo { int length; } Foo[] test; and then is test.length the length of the array, or an array of lengths of the Foo objects?

That is a problem. On the one hand, this is obviously a potential source of error, and could be flagged down by the compiler quite easily. On the other, it could be handled via some precedence given to array properties over array expression resolution (not a very good idea IMO). Perhaps an additional array pseudo-property could be added to avoid such conflicts? How about 'each' or 'every'?
 Foo[] test;
 test.each.foo(); // calls .foo() for each object
 int[] result = test.each.bar();

 One question: What about associative arrays?

Good question. I guess we could extend the concept to AAs. An operation on an AA would return a new AA in which the keys remain the same and the operation is applied to the values. Trying to do it on binary ops, let alone functions of three or more parameters, would require that the two arrays have the same set of keys ... but is this likely to happen in the real world?

I see what you mean. It reminds me of database operations, when performing arithmetic in SQL statements. Perhaps the standard ruleset for dealing with null (non-existant) values might come into play here: any op performed against null is null. So far, you've applied the spec to associative arrays already. The dimensions thus far have merely been int's, so this much is done. I think that if you were to rewrite your translations based on AA's of int keys, you'd see that its most of the way there. All that's really left is set-notation, which would be necessary the instant you get away from using scalars for your dimensions. Overloading '+' for superset, or union, for example would be a bad move because objects may have operator overloads that are applicable. So you'd need to extend the array properties to include operations apart form minValue() and maxValue() like union(arr), superset(arr), exclusion(arr) and so forth. Fore example, assume that ValueType and KeyType can be *anything*, not just int or char[].
 alias ValueType[KeyType] ExampleAA;
 ExampleAA a,b,c;
 a = b + c;

Which translates to:
 a = b.dup;
 foreach(KeyType key,ValueType  value; c){
   a[key] += value;
 }

The above fails if ValueType is an object w/o opAdd() or a string, so the compiler would have to see the types invovled ahead of time and generate a compiler error (Cannot add char[][int] to char[][int]). This syntax would give the needed behavior:
 a = b.superset(c);

Which translates to:
 a = b.dup;
 foreach(KeyType key,ValueType  value; c){
   a[key] = value; // subtle, but important
 }

- EricAnderton at yahoo
Feb 16 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Wed, 16 Feb 2005 16:10:40 +0000 (UTC), pragma  
<pragma_member pathlink.com> wrote:
 In article <cuvgv4$4fd$1 digitaldaemon.com>, Stewart Gordon says...
 pragma wrote:
 Foo[] test;
 test.foo(); // calls .foo() for each object
 int[] result = test.bar(); // calls .bar() for each object, result is  
 an array.


That would lead to such troubles as class Foo { int length; } Foo[] test; and then is test.length the length of the array, or an array of lengths of the Foo objects?

That is a problem. On the one hand, this is obviously a potential source of error, and could be flagged down by the compiler quite easily. On the other, it could be handled via some precedence given to array properties over array expression resolution (not a very good idea IMO). Perhaps an additional array pseudo-property could be added to avoid such conflicts? How about 'each' or 'every'?

What about: test.length //length of array test[].length //length of each element in array I think this idea could possibly be a source of bugs. Regan
Feb 16 2005
parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
Regan Heath schrieb:
 What about:
 
 test.length    //length of array
 test[].length  //length of each element in array

Bad idea: 'test[]' is an array by itself, so it has a length property by itself...
Feb 16 2005
parent reply "Regan Heath" <regan netwin.co.nz> writes:
On Thu, 17 Feb 2005 05:58:39 +0100, Norbert Nemec  
<Norbert Nemec-online.de> wrote:
 Regan Heath schrieb:
 What about:
  test.length    //length of array
 test[].length  //length of each element in array

Bad idea: 'test[]' is an array by itself, so it has a length property by itself...

Is it? I thought it was a class, with a length property. # class Foo { # int length; # } now, if it had an [] operator (opIndex?) then that would be a problem. I have no bright ideas yet... If it was an array... Foo[][] test; test.length == length of [][] test[].length == length of each [] in [] test[][].length == foo length property. Regan.
Feb 17 2005
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
Regan Heath wrote:
<snip>
 If it was an array...
 
 Foo[][] test;
 
 test.length     == length of [][]
 test[].length   == length of each [] in []
 test[][].length == foo length property.

The trouble is that, at the moment, test[][] is equivalent to test[]. It's just the whole array taken a slice of again. Another catch is that if test is a struct/class, by writing test[].length you could be asking to take the length of the array returned by its opSlice method. And so either some existing code may well be broken or the semantics wouldn't be context-free.... (OK, so one would be an lvalue, and one an rvalue, but I recall commenting on that PCC before....) Stewart. -- My e-mail is valid but not my primary mailbox. Please keep replies on the 'group where everyone may benefit.
Feb 18 2005
parent Norbert Nemec <Norbert Nemec-online.de> writes:
Stewart Gordon schrieb:
 Regan Heath wrote:
 <snip>
 
 If it was an array...

 Foo[][] test;

 test.length     == length of [][]
 test[].length   == length of each [] in []
 test[][].length == foo length property.

The trouble is that, at the moment, test[][] is equivalent to test[]. It's just the whole array taken a slice of again.

Why "at the moment"? If you want to change this, I believe you are asking for deep trouble. (Not only compatibility, but consistency of the language) [] simply is the full-slice operator. If a regular slice operator returns an array, so should the full-slice operator.
Feb 18 2005
prev sibling next sibling parent Norbert Nemec <Norbert Nemec-online.de> writes:
Nice work, Stewart! This is definitely much more than I have come up 
with over the past months. If only, I could get my head together, sit 
down and write up the complete proposal on arrays, array operations, 
etc. that I have in my mind...

The proposal that you bring up seems mostly similar to the one that was 
in the specs over a long time - except of course, that it is a lot 
clearer and more detailed. For the time being, I see no fundamental flaw 
with it. However, the question that are left for me are:


* How does one extend these array operations?

Interpreting every function call on an array as an elementwise operation 
is not a good idea. The other extreme would be to limit array operations 
to the fixed set, predefined by the language. This would give you a toy 
solution that is nice for the occasional user but does not hold up for 
someone digging in deeper.


* How does this extend to multidimensional arrays?

The kind of nested arrays that you mention (like int[][]) is all we have 
at the moment, but I still feel that D needs true multidimensional (i.e. 
rectangular) arrays. And for these, there should be some more 
flexibility to specify which dimension should be looped over etc.

I know that rectangular arrays are a thing of the future, but I still 
think that they are needed, and I see a certain danger in specifying 
array operations without rectangular arrays in mind.
Feb 16 2005
prev sibling next sibling parent Stewart Gordon <smjg_1998 yahoo.com> writes:
Another open question: should we allow all this on char arrays?

I can imagine someone wanting to do Caesar or Vigenère cipher stuff with 
this.  But does it really make any sense to do it under the UTF constraints?

Stewart.

-- 
My e-mail is valid but not my primary mailbox.  Please keep replies on 
the 'group where everyone may benefit.
Feb 16 2005
prev sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
One more detail: it should be clear, that the order of evaluation is not 
defined and that the temporary array is not guaranteed to be created. 
Any code that depends on the order of evaluation or the existance of 
temporaries is to be considered erraneous, like:

	a[1:9] = a[0:8]+1;

(The compiler may not always be able to reliably detect such errors. 
Maybe a case for warnings?)

If the order is unnecessarily defined in the specs, this might seriously 
limit the optimizability of the code. Currently, the wording "is 
equivalent to" sounds very dangerous in that respect.

Ciao,
Norbert



Stewart Gordon schrieb:
 This'll probably get people asking the prospect of array operations for 
 1.0 to be resurrected, but still....
 
 Here is a possible specification for array operations that I feel is 
 better-defined than the one in the current out-of-date spec.  Of course, 
 there are still some open questions, which I've put at the bottom.
 
 
 Array operations
 ----------------
 Arithmetic and bitwise operators are defined on array operands.  An 
 expression involving an array evaluates to a new array in which the 
 operator has been applied to the elements of the operands in turn.
 
 In essence, when an expression contains more than one array operation, a 
 new array is created to hold the result of each operation.  However, a 
 quality implementation will optimize the evaluation of the expression to 
 eliminate temporaries where possible.
 
 Unary operations
 ~~~~~~~~~~~~~~~~
 For the unary operators +, - and ~, the expression evaluates to a new 
 array containing the result of applying the operator to each element. 
 For example, with the declaration
 
     int[] x, y;
 
 then the statement
 
     y = -x;
 
 is simply equivalent to
 
     y = new int[x.length];
     for (int i = 0; i < y.length; i++) {
         y[i] = -x[i];
     }
 
 Binary operations
 ~~~~~~~~~~~~~~~~~
 The binary operations supported are +, -, *, /, %, &, |, ^, <<, >> and >>>.
 
 If the two arrays are of the same dimension and of compatible types, 
 then the expression evaluates to a new array in which each element is 
 the result of applying the operator to corresponding elements of the 
 operands.  For example, with the declarations
 
     int[] x, y, z;
 
 the statement
 
     z = x + y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] + y[i];
     }
 
 Both operands must be of the same length.  If they are not, an 
 ArrayBoundsError is thrown.
 
 For higher dimensions, this definition is applied recursively.  For 
 example, with
 
     int[][] x, y, z;
 
 the statement
 
     z = x * y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] * y[i];
     }
 
 which is in turn equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = new int[x[i].length];
         for (int j = 0; j < z[i].length; j++) {
             z[i][j] = x[i][j] * y[i][j];
         }
     }
 
 If the operands do not match in dimension, then the operator is applied 
 to each element of the higher-dimension operation with the whole of the 
 lower-dimension one.  For example, with
 
     int[] x, z;
     int y;
 
 the statement
 
     z = x - y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] - y;
     }
 
 Similarly,
 
     z = y - x;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = y - x[i];
     }
 
 This definition is applied recursively if the dimensions differ by two 
 or more.
 
 Assignment operations
 ~~~~~~~~~~~~~~~~~~~~~
 When x is an array, the assignment
 
     x op= y;
 
 is taken as equivalent to
 
     x = x op y;
 
 whether y is an array of matching dimension, an array of lower dimension 
 or a scalar.  Thus the operation creates a new array and assigns it to 
 x.  If a sliced lvalue is used, the array is modified in place, so that
 
     x[] op= y;
 
 is equivalent to
 
     x[] = x[] op y;
 
 The preincrement and predecrement operators are handled in the same way.
 
 User-defined types
 ~~~~~~~~~~~~~~~~~~
 A class, struct or union type may have operators overloaded with array 
 types as parameters.  To avoid conflicts between overloaded operators 
 and array operations, binary operations involving both array and 
 user-defined types are resolved as follows:
 
 1. The normal operator overloading rules are applied.
 2. If no match is found, the array operation rules are applied until 
 both operands are reduced to scalar type; operator overloading rules are 
 then applied to the result.
 3. If the expression still does not resolve, it is an error.
 
 
 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they be 
 handled?
 
 Should we generalise the concept to function calls?  If so, I guess that 
 overload resolution would work in much the same way as for operations on 
 user-defined types.
 
 If we do allow it on function calls, should we allow it to work on 
 functions of three or more parameters?  In this case, the 
 highest-dimension argument would be reduced to the dimension of the 
 second highest, and then these two reduced together to match the third 
 highest, and so on.
 
 Of course, these questions raise one more: how easy or hard would these 
 ideas be to implement?
 
 
 Any thoughts?
 
 Stewart.
 

Feb 16 2005
parent reply Dave <Dave_member pathlink.com> writes:
In article <cv17tg$22nb$1 digitaldaemon.com>, Norbert Nemec says...
One more detail: it should be clear, that the order of evaluation is not 
defined and that the temporary array is not guaranteed to be created. 
Any code that depends on the order of evaluation or the existance of 
temporaries is to be considered erraneous, like:

	a[1:9] = a[0:8]+1;

(The compiler may not always be able to reliably detect such errors. 
Maybe a case for warnings?)

If the order is unnecessarily defined in the specs, this might seriously 
limit the optimizability of the code. Currently, the wording "is 
equivalent to" sounds very dangerous in that respect.

Ciao,
Norbert

Norbert - a few questions on all this if you have the time. How has the F95 FORALL statement gone over in numerical computing? Is it being used in a lot of codes and/or has there been a lot of software being re-written to use it? How are the optimizing compilers dealing with it - are they using it to good advantage? I gather there has been at least a little movement towards C/++ from Fortran, especially since more and more numerical software specialists have gotten into the field and pried the keyboard out of the hands of the people with the white lab coats. Problem has been that Fortran is so darn good at what it does. So, Is it possible for D (done correctly) to attract a good number of numerical computing people, or is Fortran so entrenched that this would be difficult with even a great language implementation? Is there/has there been a general feeling that it's time to "move past" Fortran in that field? Thanks, - Dave
Stewart Gordon schrieb:
 This'll probably get people asking the prospect of array operations for 
 1.0 to be resurrected, but still....
 
 Here is a possible specification for array operations that I feel is 
 better-defined than the one in the current out-of-date spec.  Of course, 
 there are still some open questions, which I've put at the bottom.
 
 
 Array operations
 ----------------
 Arithmetic and bitwise operators are defined on array operands.  An 
 expression involving an array evaluates to a new array in which the 
 operator has been applied to the elements of the operands in turn.
 
 In essence, when an expression contains more than one array operation, a 
 new array is created to hold the result of each operation.  However, a 
 quality implementation will optimize the evaluation of the expression to 
 eliminate temporaries where possible.
 
 Unary operations
 ~~~~~~~~~~~~~~~~
 For the unary operators +, - and ~, the expression evaluates to a new 
 array containing the result of applying the operator to each element. 
 For example, with the declaration
 
     int[] x, y;
 
 then the statement
 
     y = -x;
 
 is simply equivalent to
 
     y = new int[x.length];
     for (int i = 0; i < y.length; i++) {
         y[i] = -x[i];
     }
 
 Binary operations
 ~~~~~~~~~~~~~~~~~
 The binary operations supported are +, -, *, /, %, &, |, ^, <<, >> and >>>.
 
 If the two arrays are of the same dimension and of compatible types, 
 then the expression evaluates to a new array in which each element is 
 the result of applying the operator to corresponding elements of the 
 operands.  For example, with the declarations
 
     int[] x, y, z;
 
 the statement
 
     z = x + y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] + y[i];
     }
 
 Both operands must be of the same length.  If they are not, an 
 ArrayBoundsError is thrown.
 
 For higher dimensions, this definition is applied recursively.  For 
 example, with
 
     int[][] x, y, z;
 
 the statement
 
     z = x * y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] * y[i];
     }
 
 which is in turn equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = new int[x[i].length];
         for (int j = 0; j < z[i].length; j++) {
             z[i][j] = x[i][j] * y[i][j];
         }
     }
 
 If the operands do not match in dimension, then the operator is applied 
 to each element of the higher-dimension operation with the whole of the 
 lower-dimension one.  For example, with
 
     int[] x, z;
     int y;
 
 the statement
 
     z = x - y;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = x[i] - y;
     }
 
 Similarly,
 
     z = y - x;
 
 is equivalent to
 
     z = new int[x.length];
     for (int i = 0; i < z.length; i++) {
         z[i] = y - x[i];
     }
 
 This definition is applied recursively if the dimensions differ by two 
 or more.
 
 Assignment operations
 ~~~~~~~~~~~~~~~~~~~~~
 When x is an array, the assignment
 
     x op= y;
 
 is taken as equivalent to
 
     x = x op y;
 
 whether y is an array of matching dimension, an array of lower dimension 
 or a scalar.  Thus the operation creates a new array and assigns it to 
 x.  If a sliced lvalue is used, the array is modified in place, so that
 
     x[] op= y;
 
 is equivalent to
 
     x[] = x[] op y;
 
 The preincrement and predecrement operators are handled in the same way.
 
 User-defined types
 ~~~~~~~~~~~~~~~~~~
 A class, struct or union type may have operators overloaded with array 
 types as parameters.  To avoid conflicts between overloaded operators 
 and array operations, binary operations involving both array and 
 user-defined types are resolved as follows:
 
 1. The normal operator overloading rules are applied.
 2. If no match is found, the array operation rules are applied until 
 both operands are reduced to scalar type; operator overloading rules are 
 then applied to the result.
 3. If the expression still does not resolve, it is an error.
 
 
 Open questions
 ~~~~~~~~~~~~~~
 Should postincrement and postdecrement be allowed?  How should they be 
 handled?
 
 Should we generalise the concept to function calls?  If so, I guess that 
 overload resolution would work in much the same way as for operations on 
 user-defined types.
 
 If we do allow it on function calls, should we allow it to work on 
 functions of three or more parameters?  In this case, the 
 highest-dimension argument would be reduced to the dimension of the 
 second highest, and then these two reduced together to match the third 
 highest, and so on.
 
 Of course, these questions raise one more: how easy or hard would these 
 ideas be to implement?
 
 
 Any thoughts?
 
 Stewart.
 


Feb 17 2005
parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
Dave schrieb:
 Norbert - a few questions on all this if you have the time.
 
 How has the F95 FORALL statement gone over in numerical computing? Is it being
 used in a lot of codes and/or has there been a lot of software being re-written
 to use it? How are the optimizing compilers dealing with it - are they using it
 to good advantage?
 
 I gather there has been at least a little movement towards C/++ from Fortran,
 especially since more and more numerical software specialists have gotten into
 the field and pried the keyboard out of the hands of the people with the white
 lab coats. Problem has been that Fortran is so darn good at what it does. So,
Is
 it possible for D (done correctly) to attract a good number of numerical
 computing people, or is Fortran so entrenched that this would be difficult with
 even a great language implementation? Is there/has there been a general feeling
 that it's time to "move past" Fortran in that field?

Very good questions indeed. I cannot easily give an answer to them. Fact is that people move slowly. The use what they know and what has proven to work. Before they even think about changing, you'll have to demonstrate without a doubt that the alternative is superior. And even then, most will make a decision to change and need years before they find the time to actually do the step. In my immediate environment, people use * plain C, because they had some course as a junior * Fortran, because it has great libraries * Matlab, because it is so well documented and easy to start with * Python, because it is a beautiful wrapper around ugly C/Fortran code To pull any of them, it will take years. The only hope that I have is, that D gains momentum for its general purpose qualities, and when the Numerics people finally arrive, they realize that everything is prepared for them to dive in. As far as I can observe, numerics people usually have a very pragmatic approach and take what's there. To invest in their support is a very long-term investment, but once we have them, we can be sure that it will pay back by their investment in the quality of compilers. About the practical use of the vectorizing F95 features - I do not really know. I know that F95 is an awfully complicated language and that the implementations are far behind C++. I know that many people use F90 compilers only because of the nicer syntax but stick to their well-known programming style. The world is changing awfully slow...
Feb 17 2005
parent Dave <Dave_member pathlink.com> writes:
In article <cv27t5$2k6$1 digitaldaemon.com>, Norbert Nemec says...
Dave schrieb:
 Norbert - a few questions on all this if you have the time.
 
 How has the F95 FORALL statement gone over in numerical computing? Is it being
 used in a lot of codes and/or has there been a lot of software being re-written
 to use it? How are the optimizing compilers dealing with it - are they using it
 to good advantage?
 
 I gather there has been at least a little movement towards C/++ from Fortran,
 especially since more and more numerical software specialists have gotten into
 the field and pried the keyboard out of the hands of the people with the white
 lab coats. Problem has been that Fortran is so darn good at what it does. So,
Is
 it possible for D (done correctly) to attract a good number of numerical
 computing people, or is Fortran so entrenched that this would be difficult with
 even a great language implementation? Is there/has there been a general feeling
 that it's time to "move past" Fortran in that field?

Very good questions indeed. I cannot easily give an answer to them. Fact is that people move slowly. The use what they know and what has proven to work. Before they even think about changing, you'll have to demonstrate without a doubt that the alternative is superior. And even then, most will make a decision to change and need years before they find the time to actually do the step. In my immediate environment, people use * plain C, because they had some course as a junior * Fortran, because it has great libraries * Matlab, because it is so well documented and easy to start with * Python, because it is a beautiful wrapper around ugly C/Fortran code To pull any of them, it will take years. The only hope that I have is, that D gains momentum for its general purpose qualities, and when the Numerics people finally arrive, they realize that everything is prepared for them to dive in. As far as I can observe, numerics people usually have a very pragmatic approach and take what's there. To invest in their support is a very long-term investment, but once we have them, we can be sure that it will pay back by their investment in the quality of compilers. About the practical use of the vectorizing F95 features - I do not really know. I know that F95 is an awfully complicated language and that the implementations are far behind C++. I know that many people use F90 compilers only because of the nicer syntax but stick to their well-known programming style. The world is changing awfully slow...

Aye, in some ways at least. Maybe D can change some of that - I think the array operations that you, Walter and Stewart (sorry if I forgot anyone) are hashing out may be a big part of it. Even if the numerics developers are slow on the uptake, I bet the rest of us will find some use for what you're discussing! Thanks, - Dave
Feb 17 2005