www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Appender and CTFE

reply Trass3r <un known.com> writes:
replace() doesn't work in CTFE anymore cause it was modified to be based on
Appender.
According to klickverbot, other phobos functions share that fate.

I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?
Mar 03 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Trass3r:


 I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?


There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is
usable in pure functions too), that allows to create two paths inside the
Appender, one for CT and one for runtime.

Bye,
bearophile
Mar 03 2011
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 03.03.2011 14:12, schrieb bearophile:

 Trass3r:


  I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?


 There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is
usable in pure functions too), that allows to create two paths inside the
Appender, one for CT and one for runtime.


but doesn't that kill a little bit the ctfe feature :) ... if i always 
pushed to the direct of using another branch, im also push to have 
another branch of errors :) - whats the point of ctfe then?
Mar 03 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-03-03 14:12, bearophile wrote:

 Trass3r:


 I think something should be done about this. Couldn't Appender be implemented
without using a pointer to a struct?


 There's no need for this, there is __ctfe (that thanks to fixing bug 4177 is
usable in pure functions too), that allows to create two paths inside the
Appender, one for CT and one for runtime.

 Bye,
 bearophile


So much for the "having the same implementation for the compile time 
function and the runtime function".

-- 
/Jacob Carlborg
Mar 03 2011
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:


 On 2011-03-03 14:12, bearophile wrote:

 Trass3r:


 I think something should be done about this. Couldn't Appender be  
 implemented without using a pointer to a struct?


 There's no need for this, there is __ctfe (that thanks to fixing bug  
 4177 is usable in pure functions too), that allows to create two paths  
 inside the Appender, one for CT and one for runtime.

 Bye,
 bearophile


 So much for the "having the same implementation for the compile time  
 function and the runtime function".


Actually, I think even if Appender wasn't implemented via pImpl style, it  
wouldn't be available to CTFE because it uses implementation details from  
the GC and runtime (for performance).

I think a __ctfe thing is probably the right thing to do.  Note that  
nobody cares about append performance in CTFE because it only affects  
compile time, not runtime, so the CTFE version can be really foolish and  
simple, and nobody cares if it's "implemented the same".

-Steve
Mar 03 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-03-03 16:35, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:


 On 2011-03-03 14:12, bearophile wrote:

 Trass3r:


 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 There's no need for this, there is __ctfe (that thanks to fixing bug
 4177 is usable in pure functions too), that allows to create two
 paths inside the Appender, one for CT and one for runtime.

 Bye,
 bearophile


 So much for the "having the same implementation for the compile time
 function and the runtime function".


 Actually, I think even if Appender wasn't implemented via pImpl style,
 it wouldn't be available to CTFE because it uses implementation details
 from the GC and runtime (for performance).

 I think a __ctfe thing is probably the right thing to do. Note that
 nobody cares about append performance in CTFE because it only affects
 compile time, not runtime, so the CTFE version can be really foolish and
 simple, and nobody cares if it's "implemented the same".

 -Steve


Yeah, but Walter and/or Andrei has mentioned that as a feature. Having 
the same code for compile time and runtime.

-- 
/Jacob Carlborg
Mar 03 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 10:05 AM, Jacob Carlborg wrote:

 On 2011-03-03 16:35, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 10:28:00 -0500, Jacob Carlborg <doob me.com> wrote:


 On 2011-03-03 14:12, bearophile wrote:

 Trass3r:


 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 There's no need for this, there is __ctfe (that thanks to fixing bug
 4177 is usable in pure functions too), that allows to create two
 paths inside the Appender, one for CT and one for runtime.

 Bye,
 bearophile


 So much for the "having the same implementation for the compile time
 function and the runtime function".


 Actually, I think even if Appender wasn't implemented via pImpl style,
 it wouldn't be available to CTFE because it uses implementation details
 from the GC and runtime (for performance).

 I think a __ctfe thing is probably the right thing to do. Note that
 nobody cares about append performance in CTFE because it only affects
 compile time, not runtime, so the CTFE version can be really foolish and
 simple, and nobody cares if it's "implemented the same".

 -Steve


 Yeah, but Walter and/or Andrei has mentioned that as a feature. Having
 the same code for compile time and runtime.


I see nothing wrong with the occasional forking conditioned by __ctfe. 
Even today, code may fork an optimized but nonportable implementation of 
some algorithm. The main requirement is that such forks are rare enough 
to not cause undue maintenance burden.

Andrei
Mar 03 2011
parent reply Kevin Bealer <kevindangerbealer removedanger.gmail.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

 I see nothing wrong with the occasional forking conditioned by __ctfe.
 Even today, code may fork an optimized but nonportable implementation of
 some algorithm. The main requirement is that such forks are rare enough
 to not cause undue maintenance burden.

 Andrei


Regarding maintenance burden, it should be easy to test the correctness of
such code:

in a unit test:

  enum a = f(...);
  assert(a == f(...));

Kevin
Mar 04 2011
parent dennis luehring <dl.soluz gmx.net> writes:
Am 04.03.2011 09:51, schrieb Kevin Bealer:

 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article

  I see nothing wrong with the occasional forking conditioned by __ctfe.
  Even today, code may fork an optimized but nonportable implementation of
  some algorithm. The main requirement is that such forks are rare enough
  to not cause undue maintenance burden.

  Andrei


 Regarding maintenance burden, it should be easy to test the correctness of
 such code:

 in a unit test:

    enum a = f(...);
    assert(a == f(...));

 Kevin


based on the complexity of the "function" is can be much much more then 
that... and the problem is that "normal" users tend to forget unit-tests...

the _ctfe thing is great because it solves problems in the real world 
still better then before, but it easily opens the door to multi-branch 
development with many different error-szenarios

the question is: is there a way to keep the _ctfe-branching under 
control - i think as long as the ctfe functionality is very near to the 
normal function world it will kept low by nature
Mar 04 2011
prev sibling parent David Nadlinger <see klickverbot.at> writes:
On 3/3/11 4:28 PM, Jacob Carlborg wrote:

 So much for the "having the same implementation for the compile time
 function and the runtime function".


Well, that's exactly what __ctfe allows for in this case – optimizing a 
small section of the code for performance using runtime-only stuff, 
while still being able to call all functions depending of it at compile 
time too…

Do you think you will find time to work on this reasonably soon, Steve?

David
Mar 03 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be based  
 on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be  
 implemented without using a pointer to a struct?


Yes, it is something I intend to do.  I want to make an (un safe) scoped  
appender which does not use a pImpl, along with a safe appender which does  
use pImpl.

It should be as easy as making the impl struct a public struct.

-Steve
Mar 03 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 8:34 AM, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 Yes, it is something I intend to do. I want to make an (un safe) scoped
 appender which does not use a pImpl, along with a safe appender which
 does use pImpl.

 It should be as easy as making the impl struct a public struct.

 -Steve


One broader question would be if Appender's optimization is still needed 
after Steve's improvements to ~=. What is the current performance gap 
between using ~= and Appender?

Andrei
Mar 03 2011
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:03:31 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 Yes, it is something I intend to do. I want to make an (un safe) scoped
 appender which does not use a pImpl, along with a safe appender which
 does use pImpl.

 It should be as easy as making the impl struct a public struct.

 -Steve


 One broader question would be if Appender's optimization is still needed  
 after Steve's improvements to ~=. What is the current performance gap  
 between using ~= and Appender?


It is large.  The Appender struct keeps track of its capacity in a local  
member, instead of having to look it up using the GC.  The lookup cache  
helps, but it's still a convoluted lookup, vs a simple member dereference.

Using trivial benchmarks, I found Appender is about 5x faster than ~=.

-Steve
Mar 03 2011
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:19:12 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 Using trivial benchmarks, I found Appender is about 5x faster than ~=.


code attached if anyone is interested.

-Steve
Mar 03 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:20:44 -0500, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 On Thu, 03 Mar 2011 11:19:12 -0500, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:


 Using trivial benchmarks, I found Appender is about 5x faster than ~=.


 code attached if anyone is interested.


err... you can comment out the assert and sanity check at the end.  It  
doesn't add significant time to the runtime, but I was testing my fixes  
for bug 5198.

-Steve
Mar 03 2011
prev sibling next sibling parent reply spir <denis.spir gmail.com> writes:
On 03/03/2011 05:03 PM, Andrei Alexandrescu wrote:

 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 Yes, it is something I intend to do. I want to make an (un safe) scoped
 appender which does not use a pImpl, along with a safe appender which
 does use pImpl.

 It should be as easy as making the impl struct a public struct.

 -Steve


 One broader question would be if Appender's optimization is still needed after
 Steve's improvements to ~=. What is the current performance gap between using
 ~= and Appender?


When I tried to use Appender, it was actually slightly slower than plain ~=. 
The reason was, IIRC, Appender is only efficient when bits added are simple 
elements, not mini-arrays to be concatenated (there was a thread about that). 
There was also an issue with reserve.

Denis
-- 
_________________
vita es estrany
spir.wikidot.com
Mar 03 2011
parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:22:00 -0500, spir <denis.spir gmail.com> wrote:


 On 03/03/2011 05:03 PM, Andrei Alexandrescu wrote:

 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 Yes, it is something I intend to do. I want to make an (un safe) scoped
 appender which does not use a pImpl, along with a safe appender which
 does use pImpl.

 It should be as easy as making the impl struct a public struct.

 -Steve


 One broader question would be if Appender's optimization is still  
 needed after
 Steve's improvements to ~=. What is the current performance gap between  
 using
 ~= and Appender?


 When I tried to use Appender, it was actually slightly slower than plain  
 ~=. The reason was, IIRC, Appender is only efficient when bits added are  
 simple elements, not mini-arrays to be concatenated (there was a thread  
 about that).  There was also an issue with reserve.


I believe those problems have been resolved.  It was fixed in the latest  
release (2.052)

-Steve
Mar 03 2011
prev sibling parent reply kenji hara <k.hara.pg gmail.com> writes:
Even without performance issue, Appender is necessary because it
configures output range.

Currently std.array.put does not allow empty array as its argument.
int[] arr = [];
arr.put(1);  // invalid. this does not means appending.

Therefore building an array its length is not known beforehand
requires Appender.

Kenji

2011/3/4 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 3/3/11 8:34 AM, Steven Schveighoffer wrote:

 On Thu, 03 Mar 2011 07:41:49 -0500, Trass3r <un known.com> wrote:


 replace() doesn't work in CTFE anymore cause it was modified to be
 based on Appender.
 According to klickverbot, other phobos functions share that fate.

 I think something should be done about this. Couldn't Appender be
 implemented without using a pointer to a struct?


 Yes, it is something I intend to do. I want to make an (un safe) scoped
 appender which does not use a pImpl, along with a safe appender which
 does use pImpl.

 It should be as easy as making the impl struct a public struct.

 -Steve


 One broader question would be if Appender's optimization is still needed
 after Steve's improvements to ~=. What is the current performance gap
 between using ~= and Appender?

 Andrei
Mar 03 2011
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/3/11 10:54 AM, kenji hara wrote:

 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr = [];
 arr.put(1);  // invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.

 Kenji


Agreed. I was mostly hoping about implementation simplification.

Andrei
Mar 03 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 03 Mar 2011 11:54:58 -0500, kenji hara <k.hara.pg gmail.com> wrote:


 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr = [];
 arr.put(1);  // invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.


Performance really is the only issue, the complexity of Appender is not  
necessary for your issue.  One could make a very simple appender struct if  
creating a proper output range was the only problem:

struct appender(T)
{
    T[] data;
    void put(T t) { data ~= t; }
}

-Steve
Mar 03 2011
parent kenji hara <k.hara.pg gmail.com> writes:
I'm sorry my point was irrelevant.

Kenji

2011/3/4 Steven Schveighoffer <schveiguy yahoo.com>:

 On Thu, 03 Mar 2011 11:54:58 -0500, kenji hara <k.hara.pg gmail.com> wrot=


e:

 Even without performance issue, Appender is necessary because it
 configures output range.

 Currently std.array.put does not allow empty array as its argument.
 int[] arr =3D [];
 arr.put(1); =A0// invalid. this does not means appending.

 Therefore building an array its length is not known beforehand
 requires Appender.


 Performance really is the only issue, the complexity of Appender is not
 necessary for your issue. =A0One could make a very simple appender struct=


 if

 creating a proper output range was the only problem:

 struct appender(T)
 {
 =A0 T[] data;
 =A0 void put(T t) { data ~=3D t; }
 }

 -Steve
Mar 03 2011