www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Appender bug, or impossible?

reply Jonathan Marler <johnnymarler gmail.com> writes:
It looks like an Appender field of a struct doesn't work when its 
element type is the containing struct, i.e.

struct Foo
{
    Appender!(Foo[]) fooAppender;
}

My guess is the problem is that the Appender needs to know how 
big "Foo" is since it would be storing each element by value, but 
since Foo's size depends on the size of the appender, we have a 
circular dependency on knowing the storage size.

This theory is reinforced because modifying the element type of 
fooAppender to "Foo*" fixes the problem.
struct Foo
{
    Appender!(Foo*[]) fooAppender; // Works fine
}

One odd thing is that this error doesn't manifest until the 
appender is used.  It compiles just fine so long as you don't 
actually use the appender, but once you put something in it or 
access the data, you will get a compiler error at the location 
where it was used, and not at the place where the appender was 
defined.  I'm not sure how it does this, but from the error 
messages I've seen, it looks like the appender may be setting the 
element type to "void" and silently continuing on as if this 
isn't a problem.

Does anything know if this is a bug, or if it is as designed?  If 
it turns out that the functionality can't be supported, I think 
that not throwing an error at the location of the appender 
definition is a problem that should be addressed.
Nov 13 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 11/13/16 11:29 AM, Jonathan Marler wrote:
 It looks like an Appender field of a struct doesn't work when its
 element type is the containing struct, i.e.

 struct Foo
 {
    Appender!(Foo[]) fooAppender;
 }

 My guess is the problem is that the Appender needs to know how big "Foo"
 is since it would be storing each element by value, but since Foo's size
 depends on the size of the appender, we have a circular dependency on
 knowing the storage size.

 This theory is reinforced because modifying the element type of
 fooAppender to "Foo*" fixes the problem.
 struct Foo
 {
    Appender!(Foo*[]) fooAppender; // Works fine
 }

 One odd thing is that this error doesn't manifest until the appender is
 used.  It compiles just fine so long as you don't actually use the
 appender, but once you put something in it or access the data, you will
 get a compiler error at the location where it was used, and not at the
 place where the appender was defined.  I'm not sure how it does this,
 but from the error messages I've seen, it looks like the appender may be
 setting the element type to "void" and silently continuing on as if this
 isn't a problem.
This is probably because Appender's functions are all templates. A Foo[] can be stored in a Foo, because it doesn't need the size. But yes, as soon as you start needing Appender functions, then the compiler chokes. It is a forward reference bug, but still a bug IMO. If you can store the appender, then the compiler knows how big it has to be. So it should be fine at that point. Paging Timon, I'm betting your front end handles this just fine ;) -Steve
Nov 13 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 14.11.2016 00:32, Steven Schveighoffer wrote:
 A Foo[] can be stored in a Foo, because it doesn't need the size. But
 yes, as soon as you start needing Appender functions, then the compiler
 chokes.

 It is a forward reference bug, but still a bug IMO. If you can store the
 appender, then the compiler knows how big it has to be. So it should be
 fine at that point.

 Paging Timon, I'm betting your front end handles this just fine ;)

 -Steve
It does. :) Minimal example: struct Appender(A){ alias T = typeof({ A a; return a[0]; }()); T[] data; void put(T item){ data~=item; } } struct Foo{ Appender!(Foo[]) fooAppender; } Foo[] test(){ Foo f; f.fooAppender.put(Foo()); return f.fooAppender.data; } static assert(test().length==1); Error with DMD, works with my front end.
Nov 14 2016
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 11/14/2016 12:26 PM, Timon Gehr wrote:

 Error with DMD, works with my front end.
What are the steps to use your front end instead of dmd's? Is the awesome combo of Timon front-end plus ldc back-end possible? Ali
Nov 15 2016
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.2016 10:10, Ali Çehreli wrote:
 On 11/14/2016 12:26 PM, Timon Gehr wrote:

 Error with DMD, works with my front end.
What are the steps to use your front end instead of dmd's?
The first step is I need to implement all remaining language features. :)
 Is the
 awesome combo of Timon front-end plus ldc back-end possible?

 Ali
Hopefully it is possible eventually.
Nov 16 2016
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 14.11.2016 21:26, Timon Gehr wrote:
 On 14.11.2016 00:32, Steven Schveighoffer wrote:
 A Foo[] can be stored in a Foo, because it doesn't need the size. But
 yes, as soon as you start needing Appender functions, then the compiler
 chokes.

 It is a forward reference bug, but still a bug IMO. If you can store the
 appender, then the compiler knows how big it has to be. So it should be
 fine at that point.

 Paging Timon, I'm betting your front end handles this just fine ;)

 -Steve
It does. :) Minimal example: struct Appender(A){ alias T = typeof({ A a; return a[0]; }()); T[] data; void put(T item){ data~=item; } } struct Foo{ Appender!(Foo[]) fooAppender; } Foo[] test(){ Foo f; f.fooAppender.put(Foo()); return f.fooAppender.data; } static assert(test().length==1); Error with DMD, works with my front end.
Coincidentally, I've tried compiling your front end with latest dmd just yesterday. There is one file missing, though: cent_.d. Could you please commit this, too? The cent code commented out, I noticed your code is suffering from the same issue as the OP (also happens for AAs, e.g. T[int]). A workaround is to use void[] and wrap it in property functions. I've fixed/worked around a number of issues in dmd with respect to incomplete semantic analysis that happen with template mixins, but more still pop up afterwards. Are there limitations to what the front end understands? Is it able to digest itself? That would be pretty impressive ;-)
Nov 15 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 15.11.2016 14:11, Rainer Schuetze wrote:
 On 14.11.2016 21:26, Timon Gehr wrote:
 On 14.11.2016 00:32, Steven Schveighoffer wrote:
 A Foo[] can be stored in a Foo, because it doesn't need the size. But
 yes, as soon as you start needing Appender functions, then the compiler
 chokes.

 It is a forward reference bug, but still a bug IMO. If you can store the
 appender, then the compiler knows how big it has to be. So it should be
 fine at that point.

 Paging Timon, I'm betting your front end handles this just fine ;)

 -Steve
It does. :) Minimal example: struct Appender(A){ alias T = typeof({ A a; return a[0]; }()); T[] data; void put(T item){ data~=item; } } struct Foo{ Appender!(Foo[]) fooAppender; } Foo[] test(){ Foo f; f.fooAppender.put(Foo()); return f.fooAppender.data; } static assert(test().length==1); Error with DMD, works with my front end.
Coincidentally, I've tried compiling your front end with latest dmd just yesterday. There is one file missing, though: cent_.d. Could you please commit this, too? ...
Oops. Thanks for pointing this out, I have committed it. (It's just a stub anyway: It's a wrapper around a long.)
 The cent code commented out, I noticed your code is suffering from the
 same issue as the OP (also happens for AAs, e.g. T[int]). A workaround
 is to use void[] and wrap it in property functions.

 I've fixed/worked around a number of issues in dmd with respect to
 incomplete semantic analysis that happen with template mixins, but more
 still pop up afterwards.
 ...
DMD 2.060 is the only version of DMD that builds the code.
 Are there limitations to what the front end understands?
Yup. There are many features missing that are quite easy to implement but require work, and a few that are somewhat messy to specify (e.g. 'protected'). An incomplete list: * UDA's * Built-in members (init, stringof, min, max, ...) * various forms of import statements - static, selective, renaming, ... * Initialization of union fields * anonymous structs & unions - Analysis & Lowering * additional import paths & implicit object.d * implicit inheritance from Object * version declarations * Associative arrays/Associative array literals * module declarations * (implicit) super constructor calls - default constructor call insertion - flow analysis * Destructor and postblit calls - use flow analysis to optimize moves * with statements * associative arrays * foreach statements - automatic decoding for string types - foreach over associative arrays - foreach over delegates - foreach over AliasSeq * pattern matching in old-style template constraints * explicit casts from/to class references - eg. to/from void* and to bool * try-catch-finally statements * scope guards * initialization crossing check * multi-argument struct constructors. * struct postblit & destructors * finish operator overloading support * opDispatch * member alias declarations aliasing members - correctly provide a this pointer * visibility - package, protected * alias this
 Is it able to digest itself?
Not yet. (Mostly because of missing language features.)
 That would be pretty impressive ;-)
I want to get there eventually. :) Unfortunately I haven't had a lot of time to spend on this lately. Also, DMD 2.060 has quite many annoying bugs that slow down development.
Nov 16 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-11-16 01:11, Timon Gehr wrote:

 Yup. There are many features missing that are quite easy to implement
 but require work, and a few that are somewhat messy to specify (e.g.
 'protected'). An incomplete list:

 * UDA's
 * Built-in members (init, stringof, min, max, ...)
 * various forms of import statements
    - static, selective, renaming, ...
 * Initialization of union fields
 * anonymous structs & unions
    - Analysis & Lowering
 * additional import paths & implicit object.d
 * implicit inheritance from Object
 * version declarations
 * Associative arrays/Associative array literals
 * module declarations
 * (implicit) super constructor calls
    - default constructor call insertion
    - flow analysis
 * Destructor and postblit calls
    - use flow analysis to optimize moves
 * with statements
 * associative arrays
 * foreach statements
    - automatic decoding for string types
    - foreach over associative arrays
    - foreach over delegates
    - foreach over AliasSeq
 * pattern matching in old-style template constraints
 * explicit casts from/to class references
    - eg. to/from void* and to bool
 * try-catch-finally statements
 * scope guards
 * initialization crossing check
 * multi-argument struct constructors.
 * struct postblit & destructors
 * finish operator overloading support
 * opDispatch
 * member alias declarations aliasing members
    - correctly provide a this pointer
 * visibility
    - package, protected
 * alias this
Is it "only" semantic analysis that is missing for these features or lexing and/or parsing as well? -- /Jacob Carlborg
Nov 16 2016
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 16.11.2016 19:17, Jacob Carlborg wrote:
 On 2016-11-16 01:11, Timon Gehr wrote:

 Yup. There are many features missing that are quite easy to implement
 but require work, and a few that are somewhat messy to specify (e.g.
 'protected'). An incomplete list:

 * UDA's
 * Built-in members (init, stringof, min, max, ...)
 * various forms of import statements
    - static, selective, renaming, ...
 * Initialization of union fields
 * anonymous structs & unions
    - Analysis & Lowering
 * additional import paths & implicit object.d
 * implicit inheritance from Object
 * version declarations
 * Associative arrays/Associative array literals
 * module declarations
 * (implicit) super constructor calls
    - default constructor call insertion
    - flow analysis
 * Destructor and postblit calls
    - use flow analysis to optimize moves
 * with statements
 * associative arrays
 * foreach statements
    - automatic decoding for string types
    - foreach over associative arrays
    - foreach over delegates
    - foreach over AliasSeq
 * pattern matching in old-style template constraints
 * explicit casts from/to class references
    - eg. to/from void* and to bool
 * try-catch-finally statements
 * scope guards
 * initialization crossing check
 * multi-argument struct constructors.
 * struct postblit & destructors
 * finish operator overloading support
 * opDispatch
 * member alias declarations aliasing members
    - correctly provide a this pointer
 * visibility
    - package, protected
 * alias this
Is it "only" semantic analysis that is missing for these features or lexing and/or parsing as well?
Mostly semantic analysis, but parsing is very easy anyway. I think UDAs are the only listed feature still missing in the parser, but that's just because they were introduced after I had already written the parser.
Nov 16 2016
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 16 November 2016 at 09:11:50 UTC, Timon Gehr wrote:
 I want to get there eventually. :)
 Unfortunately I haven't had a lot of time to spend on this 
 lately. Also, DMD 2.060 has quite many annoying bugs that slow 
 down development.
Sounds to me like you should rewrite the compiler a little to get it working under 2.07x.
Nov 16 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.11.2016 03:52, Stefan Koch wrote:
 On Wednesday, 16 November 2016 at 09:11:50 UTC, Timon Gehr wrote:
 I want to get there eventually. :)
 Unfortunately I haven't had a lot of time to spend on this lately.
 Also, DMD 2.060 has quite many annoying bugs that slow down development.
Sounds to me like you should rewrite the compiler a little to get it working under 2.07x.
"A little" is probably not enough.
Nov 17 2016
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Thursday, 17 November 2016 at 21:39:15 UTC, Timon Gehr wrote:
 On 17.11.2016 03:52, Stefan Koch wrote:
 On Wednesday, 16 November 2016 at 09:11:50 UTC, Timon Gehr 
 wrote:
 I want to get there eventually. :)
 Unfortunately I haven't had a lot of time to spend on this 
 lately.
 Also, DMD 2.060 has quite many annoying bugs that slow down 
 development.
Sounds to me like you should rewrite the compiler a little to get it working under 2.07x.
"A little" is probably not enough.
I'm interested to know what the big differences were that make updating particularly hard?
Nov 17 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 17.11.2016 22:41, John Colvin wrote:
 On Thursday, 17 November 2016 at 21:39:15 UTC, Timon Gehr wrote:
 On 17.11.2016 03:52, Stefan Koch wrote:
 On Wednesday, 16 November 2016 at 09:11:50 UTC, Timon Gehr wrote:
 I want to get there eventually. :)
 Unfortunately I haven't had a lot of time to spend on this lately.
 Also, DMD 2.060 has quite many annoying bugs that slow down
 development.
Sounds to me like you should rewrite the compiler a little to get it working under 2.07x.
"A little" is probably not enough.
I'm interested to know what the big differences were that make updating particularly hard?
The forward reference bugs are different.
Nov 17 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 17 November 2016 at 22:46:33 UTC, Timon Gehr wrote:
 On 17.11.2016 22:41, John Colvin wrote:
 On Thursday, 17 November 2016 at 21:39:15 UTC, Timon Gehr 
 wrote:
 On 17.11.2016 03:52, Stefan Koch wrote:
 On Wednesday, 16 November 2016 at 09:11:50 UTC, Timon Gehr 
 wrote:
 I want to get there eventually. :)
 Unfortunately I haven't had a lot of time to spend on this 
 lately.
 Also, DMD 2.060 has quite many annoying bugs that slow down
 development.
Sounds to me like you should rewrite the compiler a little to get it working under 2.07x.
"A little" is probably not enough.
I'm interested to know what the big differences were that make updating particularly hard?
The forward reference bugs are different.
His compiler is filled with templates. I have never seen anything like it.
Nov 17 2016
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 16.11.2016 10:11, Timon Gehr wrote:
 On 15.11.2016 14:11, Rainer Schuetze wrote:
 Coincidentally, I've tried compiling your front end with latest dmd just
 yesterday. There is one file missing, though: cent_.d. Could you please
 commit this, too?
 ...
Oops. Thanks for pointing this out, I have committed it. (It's just a stub anyway: It's a wrapper around a long.)
Thanks.
 The cent code commented out, I noticed your code is suffering from the
 same issue as the OP (also happens for AAs, e.g. T[int]). A workaround
 is to use void[] and wrap it in property functions.

 I've fixed/worked around a number of issues in dmd with respect to
 incomplete semantic analysis that happen with template mixins, but more
 still pop up afterwards.
 ...
DMD 2.060 is the only version of DMD that builds the code.
I've read that, but I don't want to go back that much in time...
 Are there limitations to what the front end understands?
Yup. There are many features missing that are quite easy to implement but require work, and a few that are somewhat messy to specify (e.g. 'protected'). An incomplete list: * UDA's * Built-in members (init, stringof, min, max, ...) * various forms of import statements - static, selective, renaming, ... * Initialization of union fields * anonymous structs & unions - Analysis & Lowering * additional import paths & implicit object.d * implicit inheritance from Object * version declarations * Associative arrays/Associative array literals * module declarations * (implicit) super constructor calls - default constructor call insertion - flow analysis * Destructor and postblit calls - use flow analysis to optimize moves * with statements * associative arrays * foreach statements - automatic decoding for string types - foreach over associative arrays - foreach over delegates - foreach over AliasSeq * pattern matching in old-style template constraints * explicit casts from/to class references - eg. to/from void* and to bool * try-catch-finally statements * scope guards * initialization crossing check * multi-argument struct constructors. * struct postblit & destructors * finish operator overloading support * opDispatch * member alias declarations aliasing members - correctly provide a this pointer * visibility - package, protected * alias this
Quite a list, but the issues that are likely to affect symbol lookup might not be too complicated to implement. I'm looking for candidates that might replace the semantic engine in Visual D, as D_Parser is unlikely to be developed further at the moment.
 Is it able to digest itself?
Not yet. (Mostly because of missing language features.)
 That would be pretty impressive ;-)
I want to get there eventually. :) Unfortunately I haven't had a lot of time to spend on this lately. Also, DMD 2.060 has quite many annoying bugs that slow down development.
I hope to get the forward reference bugs solved in current dmd. That might also help your motivation to continue the front end ;-)
Nov 18 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 18 November 2016 at 10:18:21 UTC, Rainer Schuetze 
wrote:

 I'm looking for candidates that might replace the semantic 
 engine in Visual D, as D_Parser is unlikely to be developed 
 further at the moment.
Hmm I plan on cleaning DDMD up a bunch. Any chance you could use the DDMD frontend ?
Nov 18 2016
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 18.11.2016 11:33, Stefan Koch wrote:
 On Friday, 18 November 2016 at 10:18:21 UTC, Rainer Schuetze wrote:

 I'm looking for candidates that might replace the semantic engine in
 Visual D, as D_Parser is unlikely to be developed further at the moment.
Hmm I plan on cleaning DDMD up a bunch. Any chance you could use the DDMD frontend ?
That's also an option, but I don't think it is in a state (yet) to be used as a completion engine.
Nov 18 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 18 November 2016 at 11:37:40 UTC, Rainer Schuetze 
wrote:
 On 18.11.2016 11:33, Stefan Koch wrote:
 On Friday, 18 November 2016 at 10:18:21 UTC, Rainer Schuetze 
 wrote:

 I'm looking for candidates that might replace the semantic 
 engine in
 Visual D, as D_Parser is unlikely to be developed further at 
 the moment.
Hmm I plan on cleaning DDMD up a bunch. Any chance you could use the DDMD frontend ?
That's also an option, but I don't think it is in a state (yet) to be used as a completion engine.
Completion engine. Indeed it's not usable as that. What about DCD ?
Nov 18 2016
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 18.11.2016 12:39, Stefan Koch wrote:
 On Friday, 18 November 2016 at 11:37:40 UTC, Rainer Schuetze wrote:
 On 18.11.2016 11:33, Stefan Koch wrote:
 On Friday, 18 November 2016 at 10:18:21 UTC, Rainer Schuetze wrote:

 I'm looking for candidates that might replace the semantic engine in
 Visual D, as D_Parser is unlikely to be developed further at the
 moment.
Hmm I plan on cleaning DDMD up a bunch. Any chance you could use the DDMD frontend ?
That's also an option, but I don't think it is in a state (yet) to be used as a completion engine.
Completion engine. Indeed it's not usable as that. What about DCD ?
Reading its feature list, I suspect it would be a step backward. D_Parser comes with pretty good template, CTFE, UFCS and mixin support.
Nov 18 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 18 November 2016 at 12:50:30 UTC, Rainer Schuetze 
wrote:
 On 18.11.2016 12:39, Stefan Koch wrote:
 On Friday, 18 November 2016 at 11:37:40 UTC, Rainer Schuetze 
 wrote:
 On 18.11.2016 11:33, Stefan Koch wrote:
 On Friday, 18 November 2016 at 10:18:21 UTC, Rainer Schuetze 
 wrote:

[...]
Hmm I plan on cleaning DDMD up a bunch. Any chance you could use the DDMD frontend ?
That's also an option, but I don't think it is in a state (yet) to be used as a completion engine.
Completion engine. Indeed it's not usable as that. What about DCD ?
Reading its feature list, I suspect it would be a step backward. D_Parser comes with pretty good template, CTFE, UFCS and mixin support.
Then I'd say try to keep dparser :) I wonder how it does mixins though ...
Nov 18 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 18.11.2016 11:18, Rainer Schuetze wrote:
 ...
Quite a list, but the issues that are likely to affect symbol lookup might not be too complicated to implement.
I think 'protected' is the only remaining feature whose implementation is not "obvious". Otherwise, finishing support for all features is mostly a question of spending the time. Some performance tuning will be necessary too.
 I'm looking for candidates
 that might replace the semantic engine in Visual D,
In theory, the frontend would be a good option because due to its design, it is quite easy to extend for incremental semantic analysis support. (It keeps track of all dependencies between nodes, all that is needed is to record those and to re-analyze dependent code as required.)
 as D_Parser is
 unlikely to be developed further at the moment.

 Is it able to digest itself?
Not yet. (Mostly because of missing language features.)
 That would be pretty impressive ;-)
I want to get there eventually. :) Unfortunately I haven't had a lot of time to spend on this lately. Also, DMD 2.060 has quite many annoying bugs that slow down development.
I hope to get the forward reference bugs solved in current dmd.
Nice initiative!
 That might also help your motivation to continue the front end ;-)
Certainly. :)
Nov 18 2016
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 18.11.2016 15:37, Timon Gehr wrote:
 I hope to get the forward reference bugs solved in current dmd.
Nice initiative!
It took some time, but here is a version of dmd: https://github.com/rainers/dmd/tree/tg_frontend that can compile an updated frontend: https://github.com/tgehr/d-compiler/pull/1 [The debug version of dmd hits 2 assertions during code generation, though (some real/ireal type mismatch, unlikely to make a lot of trouble in the release build).]
Nov 25 2016
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 25.11.2016 09:46, Rainer Schuetze wrote:
 On 18.11.2016 15:37, Timon Gehr wrote:
 I hope to get the forward reference bugs solved in current dmd.
Nice initiative!
It took some time, but here is a version of dmd: https://github.com/rainers/dmd/tree/tg_frontend ...
What needs to happen that this is merged into master? :)
 that can compile an updated frontend:

 https://github.com/tgehr/d-compiler/pull/1

 [The debug version of dmd hits 2 assertions during code generation,
 though (some real/ireal type mismatch, unlikely to make a lot of trouble
 in the release build).]
Merged! Thank you for taking care of this.
Nov 28 2016
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 28.11.2016 10:30, Timon Gehr wrote:
 On 25.11.2016 09:46, Rainer Schuetze wrote:
 On 18.11.2016 15:37, Timon Gehr wrote:
 I hope to get the forward reference bugs solved in current dmd.
Nice initiative!
It took some time, but here is a version of dmd: https://github.com/rainers/dmd/tree/tg_frontend ...
What needs to happen that this is merged into master? :)
I noticed it fails on a couple of tests from the testsuite. Will have to figure out, why... There are a couple of issues involved, these will likely have to be split up to get a chance of getting merged.
Nov 28 2016