www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Appender and CTFE

reply Trass3r <un known.com> writes:
replace() doesn't work in CTFE anymore cause it was modified to be based on
Appender.
According to klickverbot, other phobos functions share that fate.

I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?
Mar 03 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Trass3r:

 I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime. Bye, bearophile
Mar 03 2011
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 03.03.2011 14:12, schrieb bearophile:
 Trass3r:

  I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime.

but doesn't that kill a little bit the ctfe feature :) ... if i always pushed to the direct of using another branch, im also push to have another branch of errors :) - whats the point of ctfe then?
Mar 03 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-03-03 14:12, bearophile wrote:
 Trass3r:

 I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime. Bye, bearophile

So much for the "having the same implementation for the compile time function and the runtime function". -- /Jacob Carlborg
Mar 03 2011
next sibling parent David Nadlinger <see klickverbot.at> writes:
On 3/3/11 4:28 PM, Jacob Carlborg wrote:
 So much for the "having the same implementation for the compile time
 function and the runtime function".

Well, that's exactly what __ctfe allows for in this case – optimizing a small section of the code for performance using runtime-only stuff, while still being able to call all functions depending of it at compile time too… Do you think you will find time to work on this reasonably soon, Steve? David
Mar 03 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-03-03 16:35, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:

 On 2011-03-03 14:12, bearophile wrote:
 Trass3r:

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime. Bye, bearophile

So much for the "having the same implementation for the compile time function and the runtime function".

Actually, I think even if Appender wasn't implemented via pImpl style, it wouldn't be available to CTFE because it uses implementation details from the GC and runtime (for performance). I think a __ctfe thing is probably the right thing to do. Note that nobody cares about append performance in CTFE because it only affects compile time, not runtime, so the CTFE version can be really foolish and simple, and nobody cares if it's "implemented the same". -Steve

Yeah, but Walter and/or Andrei has mentioned that as a feature. Having the same code for compile time and runtime. -- /Jacob Carlborg
Mar 03 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 10:05 AM, Jacob Carlborg wrote:
 On 2011-03-03 16:35, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:

 On 2011-03-03 14:12, bearophile wrote:
 Trass3r:

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime. Bye, bearophile

So much for the "having the same implementation for the compile time function and the runtime function".

Actually, I think even if Appender wasn't implemented via pImpl style, it wouldn't be available to CTFE because it uses implementation details from the GC and runtime (for performance). I think a __ctfe thing is probably the right thing to do. Note that nobody cares about append performance in CTFE because it only affects compile time, not runtime, so the CTFE version can be really foolish and simple, and nobody cares if it's "implemented the same". -Steve

Yeah, but Walter and/or Andrei has mentioned that as a feature. Having the same code for compile time and runtime.

I see nothing wrong with the occasional forking conditioned by __ctfe. Even today, code may fork an optimized but nonportable implementation of some algorithm. The main requirement is that such forks are rare enough to not cause undue maintenance burden. Andrei
Mar 03 2011
parent reply Kevin Bealer <kevindangerbealer removedanger.gmail.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 I see nothing wrong with the occasional forking conditioned by __ctfe.
 Even today, code may fork an optimized but nonportable implementation of
 some algorithm. The main requirement is that such forks are rare enough
 to not cause undue maintenance burden.

 Andrei

Regarding maintenance burden, it should be easy to test the correctness of such code: in a unit test: enum a = f(...); assert(a == f(...)); Kevin
Mar 04 2011
parent dennis luehring <dl.soluz gmx.net> writes:
Am 04.03.2011 09:51, schrieb Kevin Bealer:
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
  I see nothing wrong with the occasional forking conditioned by __ctfe.
  Even today, code may fork an optimized but nonportable implementation of
  some algorithm. The main requirement is that such forks are rare enough
  to not cause undue maintenance burden.

  Andrei

Regarding maintenance burden, it should be easy to test the correctness of such code: in a unit test: enum a = f(...); assert(a == f(...)); Kevin

based on the complexity of the "function" is can be much much more then that... and the problem is that "normal" users tend to forget unit-tests... the _ctfe thing is great because it solves problems in the real world still better then before, but it easily opens the door to multi-branch development with many different error-szenarios the question is: is there a way to keep the _ctfe-branching under control - i think as long as the ctfe functionality is very near to the normal function world it will kept low by nature
Mar 04 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:

 On 2011-03-03 14:12, bearophile wrote:
 Trass3r:

 I think something should be done about this. Couldn't Appender be  
 implemented without using a pointer to a struct?

There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is usable in pure functions too), that allows to create two paths inside the Appender, one for CT and one for runtime. Bye, bearophile

So much for the "having the same implementation for the compile time function and the runtime function".

Actually, I think even if Appender wasn't implemented via pImpl style, it wouldn't be available to CTFE because it uses implementation details from the GC and runtime (for performance). I think a __ctfe thing is probably the right thing to do. Note that nobody cares about append performance in CTFE because it only affects compile time, not runtime, so the CTFE version can be really foolish and simple, and nobody cares if it's "implemented the same". -Steve
Mar 03 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be based  
 on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be  
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve
Mar 03 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 8:34 AM, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve

One broader question would be if Appender's optimization is still needed after Steve's improvements to ~=. What is the current performance gap between using ~= and Appender? Andrei
Mar 03 2011
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 10:54 AM, kenji hara wrote:
 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr = [];
 arr.put(1);  // invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.

 Kenji

Agreed. I was mostly hoping about implementation simplification. Andrei
Mar 03 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:03:31 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve

One broader question would be if Appender's optimization is still needed after Steve's improvements to ~=. What is the current performance gap between using ~= and Appender?

It is large. The Appender struct keeps track of its capacity in a local member, instead of having to look it up using the GC. The lookup cache helps, but it's still a convoluted lookup, vs a simple member dereference. Using trivial benchmarks, I found Appender is about 5x faster than ~=. -Steve
Mar 03 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
------------LqxzXNzTWKL2SHjRCpIVOU
Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes
Content-Transfer-Encoding: 8bit

On Thu, 03 Mar 2011 11:19:12 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 Using trivial benchmarks, I found Appender is about 5x faster than ~=.

code attached if anyone is interested. -Steve ------------LqxzXNzTWKL2SHjRCpIVOU Content-Disposition: attachment; filename=testappender.d Content-Type: application/octet-stream; name=testappender.d Content-Transfer-Encoding: Base64 aW1wb3J0IHN0ZC5hcnJheTsKaW1wb3J0IHN0ZC5yYW5nZTsKCmVudW0gbmVsZW1z ID0gMTBfMDAwXzAwMDsKCnZvaWQgbWFpbihzdHJpbmdbXSBhcmdzKSB7CiAgICBp bnRbXSB4OwogICAgaWYoYXJncy5sZW5ndGggPiAxICYmIGFyZ3NbMV0gPT0gImFw cGVuZGVyIikKICAgIHsKICAgICAgICBBcHBlbmRlciEoaW50W10pIGFwcDsKICAg ICAgICBmb3JlYWNoKGk7IDAuLm5lbGVtcykKICAgICAgICAgICAgYXBwLnB1dChp KTsKICAgICAgICB4ID0gYXBwLmRhdGE7CiAgICB9CiAgICBlbHNlIGlmKGFyZ3Mu bGVuZ3RoID4gMSAmJiBhcmdzWzFdID09ICJhcHBlbmRhcnIiKQogICAgewogICAg ICAgIEFwcGVuZGVyIShpbnRbXSkgYXBwOwogICAgICAgIGZvcmVhY2goaTsgMC4u bmVsZW1zLzQpCiAgICAgICAgewogICAgICAgICAgICBpbnRbNF0gZTsKICAgICAg ICAgICAgZVswXSA9IGkgKiA0OwogICAgICAgICAgICBlWzFdID0gaSAqIDQgKyAx OwogICAgICAgICAgICBlWzJdID0gaSAqIDQgKyAyOwogICAgICAgICAgICBlWzNd ID0gaSAqIDQgKyAzOwogICAgICAgICAgICBhcHAucHV0KGVbXSk7CiAgICAgICAg fQogICAgICAgIHggPSBhcHAuZGF0YTsKICAgIH0KICAgIGVsc2UKICAgIHsKICAg ICAgICBmb3JlYWNoKGk7IDAuLm5lbGVtcykKICAgICAgICAgICAgeCB+PSBpOwog ICAgfQogICAgYXNzZXJ0KHgubGVuZ3RoID09IG5lbGVtcyk7CiAgICBmb3JlYWNo KGk7IDAuLm5lbGVtcykKICAgICAgICBhc3NlcnQoeFtpXSA9PSBpLTEpOwp9Cg== ------------LqxzXNzTWKL2SHjRCpIVOU--
Mar 03 2011
prev sibling next sibling parent spir <denis.spir gmail.com> writes:
On 03/03/2011 05:03 PM, Andrei Alexandrescu wrote:
 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve

One broader question would be if Appender's optimization is still needed after Steve's improvements to ~=. What is the current performance gap between using ~= and Appender?

When I tried to use Appender, it was actually slightly slower than plain ~=. The reason was, IIRC, Appender is only efficient when bits added are simple elements, not mini-arrays to be concatenated (there was a thread about that). There was also an issue with reserve. Denis -- _________________ vita es estrany spir.wikidot.com
Mar 03 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:22:00 -0500, spir <denis.spir gmail.com> wrote:

 On 03/03/2011 05:03 PM, Andrei Alexandrescu wrote:
 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve

One broader question would be if Appender's optimization is still needed after Steve's improvements to ~=. What is the current performance gap between using ~= and Appender?

When I tried to use Appender, it was actually slightly slower than plain ~=. The reason was, IIRC, Appender is only efficient when bits added are simple elements, not mini-arrays to be concatenated (there was a thread about that). There was also an issue with reserve.

I believe those problems have been resolved. It was fixed in the latest release (2.052) -Steve
Mar 03 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:20:44 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Thu, 03 Mar 2011 11:19:12 -0500, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:

 Using trivial benchmarks, I found Appender is about 5x faster than ~=.

code attached if anyone is interested.

err... you can comment out the assert and sanity check at the end. It doesn't add significant time to the runtime, but I was testing my fixes for bug 5198. -Steve
Mar 03 2011
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
Even without performance issue, Appender is necessary because it
configures output range.

Currently std.array.put does not allow empty array as its argument.
int[] arr = [];
arr.put(1);  // invalid. this does not means appending.

Therefore building an array its length is not known beforehand
requires Appender.

Kenji

2011/3/4 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:
 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:
 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:

 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?

Yes, it is something I intend to do. I want to make an (un safe) scoped appender which does not use a pImpl, along with a safe appender which does use pImpl. It should be as easy as making the impl struct a public struct. -Steve

One broader question would be if Appender's optimization is still needed after Steve's improvements to ~=. What is the current performance gap between using ~= and Appender? Andrei

Mar 03 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:54:58 -0500, kenji hara <k.hara.pg gmail.com> wrote:

 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr = [];
 arr.put(1);  // invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.

Performance really is the only issue, the complexity of Appender is not necessary for your issue. One could make a very simple appender struct if creating a proper output range was the only problem: struct appender(T) { T[] data; void put(T t) { data ~= t; } } -Steve
Mar 03 2011
prev sibling parent kenji hara <k.hara.pg gmail.com> writes:
I'm sorry my point was irrelevant.

Kenji

2011/3/4 Steven Schveighoffer <schveiguy yahoo.com>:
 On Thu, 03 Mar 2011 11:54:58 -0500, kenji hara <k.hara.pg gmail.com> wrot=

 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr =3D [];
 arr.put(1); =A0// invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.

Performance really is the only issue, the complexity of Appender is not necessary for your issue. =A0One could make a very simple appender struct=

 creating a proper output range was the only problem:

 struct appender(T)
 {
 =A0 T[] data;
 =A0 void put(T t) { data ~=3D t; }
 }

 -Steve

Mar 03 2011