www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11138] New: "real" appender missing

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138

           Summary: "real" appender missing
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: Phobos
        AssignedTo: nobody puremagic.com
        ReportedBy: verylonglogin.reg gmail.com


--- Comment #0 from Denis Shelomovskij <verylonglogin.reg gmail.com> 2013-09-29
14:23:28 MSD ---
Lets call "real" an appender which will create the one and only array, i.e. it
will not randomly create a copy of array on appending. Such copy is bad as:
1. Almost no use.
    In 99% appender is used as "real" appender and the copy is not needed.
2. Performance.
    It will cause calls of postblits and then destructors on collection of copy
array.
3. Needless restriction on element type.
    One may want to create and array of uncopyable type which non-"real"
appender can't do by definition.

Obviously, "real" appender is uncopyable and don't give safe access to its
array until it finished.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Sep 29 2013
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138


monarchdodra gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |monarchdodra gmail.com


--- Comment #1 from monarchdodra gmail.com 2013-09-29 04:03:25 PDT ---
(In reply to comment #0)
 3. Needless restriction on element type.
     One may want to create and array of uncopyable type which non-"real"
 appender can't do by definition.
Appender supports immutable construction (mostly bug free, AFAIK). I don't think a "real" appender would do any better than appender could anyways, so this is a non-argument (IMO).
 2. Performance.
     It will cause calls of postblits and then destructors on collection of copy
 array.
Technically, the destructors don't need to be called, since the GC will leak them anyways :/ See below about "postblits"
 Lets call "real" an appender which will create the one and only array, i.e. it
 will not randomly create a copy of array on appending. Such copy is bad as:
 1. Almost no use.
     In 99% appender is used as "real" appender and the copy is not needed.
I've thought about this before. The *only* case this would really change anything, is on structs that have an elaborate CC. Arguably, these things have no business being in a GC managed array, since it never finalizes its elements (and arguably, if you have a CC, you have a destructor). If we ever get a finalizing GC, then things will be different. One "workaround" I've though about is that when the type has a CC, Appender could carry an extra "ownsArray" boolean. If appender must relocate its array, and "owns" the array, then postblit is not necessary. In the future, if the GC is finalizing, it may also have to re-initialize to T.init, if there is an elaborate destructor.
 Obviously, "real" appender is uncopyable
Appender having ref semantics, there would be absolutely no problem "copying" it (read: pass by value).
 and don't give safe access to its
 array until it finished.
I can see the motivation behind this "real" appender. Another (dis)advantage it has is that it wouldn't be initialize-able from an existing array. This means it would have 100% ownership of its internals, but on the other hand, it also means, it also means it can't extend an exiting array :/ So, "at the end of the day", I'm not really sure a "real" appender would pull its weight compared to the standard appender (IMO). -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 29 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138



--- Comment #2 from Denis Shelomovskij <verylonglogin.reg gmail.com> 2013-09-29
19:12:23 MSD ---
(In reply to comment #1)
 (In reply to comment #0)
 3. Needless restriction on element type.
     ...
Appender supports immutable construction (mostly bug free, AFAIK). I don't think a "real" appender would do any better than appender could anyways, so this is a non-argument (IMO).
Mutability have nothing to do with copyability. Struct with disabled postblits are meant here. And current appender design can't accept them.
 2. Performance.
     It will cause calls of postblits and then destructors on collection of copy
 array.
Technically, the destructors don't need to be called, since the GC will leak them anyways :/ See below about "postblits"
 Lets call "real" an appender which will create the one and only array, i.e. it
 will not randomly create a copy of array on appending. Such copy is bad as:
 1. Almost no use.
     In 99% appender is used as "real" appender and the copy is not needed.
I've thought about this before. The *only* case this would really change anything, is on structs that have an elaborate CC. Arguably, these things have no business being in a GC managed array, since it never finalizes its elements (and arguably, if you have a CC, you have a destructor). If we ever get a finalizing GC, then things will be different.
So, you are talking that GC bug must force us to do bad design? I will accept it only for an iron-WONTFIXed one.
 One "workaround" I've though about is that when the type has a CC, Appender
 could carry an extra "ownsArray" boolean. If appender must relocate its array,
 and "owns" the array, then postblit is not necessary. In the future, if the GC
 is finalizing, it may also have to re-initialize to T.init, if there is an
 elaborate destructor.
Such flag will not work as Appender API allows safe access to the array so the array can never be owned.
 Obviously, "real" appender is uncopyable
Appender having ref semantics, there would be absolutely no problem "copying" it (read: pass by value).
Yes. And such ref semantic is not generally needed and just results in needless GC allocation in stuff aimed for performance-critical tasks.
 I can see the motivation behind this "real" appender. Another (dis)advantage it
 has is that it wouldn't be initialize-able from an existing array. This means
 it would have 100% ownership of its internals, but on the other hand, it also
 means, it also means it can't extend an exiting array :/
It of course can't initialize from existing array if element isn't copyable. Otherwise it will create its own copy of array. Where is the problem? Also we talk here about very rare usage of the appender which I'd rather ignore just like the case when its content is needed in the middle of appending as non-general.
 So, "at the end of the day", I'm not really sure a "real" appender would pull
 its weight compared to the standard appender (IMO).
Exactly what I expect for each my proposal. :) So as always I will just create one for my own use. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 29 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138



--- Comment #3 from monarchdodra gmail.com 2013-09-29 09:41:29 PDT ---
(In reply to comment #2)
 (In reply to comment #1)
 (In reply to comment #0)
 3. Needless restriction on element type.
     ...
Appender supports immutable construction (mostly bug free, AFAIK). I don't think a "real" appender would do any better than appender could anyways, so this is a non-argument (IMO).
Mutability have nothing to do with copyability. Struct with disabled postblits are meant here. And current appender design can't accept them.
Right. Sorry. Wait. How do you append something that can't be copied? Is this even relevant?
 I've thought about this before. The *only* case this would really change
 anything, is on structs that have an elaborate CC. Arguably, these things have
 no business being in a GC managed array, since it never finalizes its elements
 (and arguably, if you have a CC, you have a destructor). If we ever get a
 finalizing GC, then things will be different.
So, you are talking that GC bug must force us to do bad design? I will accept it only for an iron-WONTFIXed one.
You have a point.
 One "workaround" I've though about is that when the type has a CC, Appender
 could carry an extra "ownsArray" boolean. If appender must relocate its array,
 and "owns" the array, then postblit is not necessary. In the future, if the GC
 is finalizing, it may also have to re-initialize to T.init, if there is an
 elaborate destructor.
Such flag will not work as Appender API allows safe access to the array so the array can never be owned.
Not really, the appender API is rich enough for appender to "know" if anybody else has yet accessed the array, either through arr, or via construction. If no one else has a handle on the array, and appender is about to relocate, then it can pretty much do anything it wants. Maybe the word "owns" is inadequate. The bool would be called "arrayHasExternalView".
 Obviously, "real" appender is uncopyable
Appender having ref semantics, there would be absolutely no problem "copying" it (read: pass by value).
Yes. And such ref semantic is not generally needed and just results in needless GC allocation in stuff aimed for performance-critical tasks.
I think appender was mostly to offset how ridiculously slow ~= is. I'm not sure it wasn't design with *absolute* performance in mind. BTW, you might want to know about: http://d.puremagic.com/issues/show_bug.cgi?id=10864 Off topic: I'd like to try to make a push for some templatization in druntime. I thought you might be interested in the project. I'll keep you posted.
 I can see the motivation behind this "real" appender. Another (dis)advantage it
 has is that it wouldn't be initialize-able from an existing array. This means
 it would have 100% ownership of its internals, but on the other hand, it also
 means, it also means it can't extend an exiting array :/
It of course can't initialize from existing array if element isn't copyable. Otherwise it will create its own copy of array. Where is the problem?
You are creating a copy. Appender can be used to append to an existing array, whith lots of capacity, without relocating it. This can be very useful for say, processing lines in IO, where you are re-using the same buffer for each line.
 Also we
 talk here about very rare usage of the appender which I'd rather ignore just
 like the case when its content is needed in the middle of appending as
 non-general.
Well very rare remains to be proven. Still, I don't think it's that big of a problem if a "real" appender simply duplicated before allocating: To each their own behavior. You can't get "super speed" if you aren't willing to give appended total ownership of the passed data.
 So, "at the end of the day", I'm not really sure a "real" appender would pull
 its weight compared to the standard appender (IMO).
Exactly what I expect for each my proposal. :) So as always I will just create one for my own use.
-- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Sep 29 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138


Dmitry Olshansky <dmitry.olsh gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh gmail.com


--- Comment #4 from Dmitry Olshansky <dmitry.olsh gmail.com> 2013-10-20
03:26:03 PDT ---
(In reply to comment #1)
 (In reply to comment #0)
 3. Needless restriction on element type.
     One may want to create and array of uncopyable type which non-"real"
 appender can't do by definition.
Appender supports immutable construction (mostly bug free, AFAIK). I don't think a "real" appender would do any better than appender could anyways, so this is a non-argument (IMO).
Construct in place. See also C++11 emplace_back and friends. Even these guys see the benefit.
 2. Performance.
     It will cause calls of postblits and then destructors on collection of copy
 array.
Technically, the destructors don't need to be called, since the GC will leak them anyways :/ See below about "postblits"
Who told you that a long-standing bug allows us to basically do even worse job then we have to ? ;)
 Lets call "real" an appender which will create the one and only array, i.e. it
 will not randomly create a copy of array on appending. Such copy is bad as:
 1. Almost no use.
     In 99% appender is used as "real" appender and the copy is not needed.
I've thought about this before. The *only* case this would really change anything, is on structs that have an elaborate CC. Arguably, these things have no business being in a GC managed array, since it never finalizes its elements (and arguably, if you have a CC, you have a destructor). If we ever get a finalizing GC, then things will be different. One "workaround" I've though about is that when the type has a CC, Appender could carry an extra "ownsArray" boolean. If appender must relocate its array, and "owns" the array, then postblit is not necessary. In the future, if the GC is finalizing, it may also have to re-initialize to T.init, if there is an elaborate destructor.
Sadly Appender leaks refernces to its internal array so it's can own the data at all.
 Obviously, "real" appender is uncopyable
Appender having ref semantics, there would be absolutely no problem "copying" it (read: pass by value).
IMHO it could just be uncopyable type and RefCounted!T should solve the issue of passing it as reference anywhere.
 and don't give safe access to its
 array until it finished.
I can see the motivation behind this "real" appender. Another (dis)advantage it has is that it wouldn't be initialize-able from an existing array. This means it would have 100% ownership of its internals, but on the other hand, it also means, it also means it can't extend an exiting array :/
You miss a big use case. Frankly patching Appender on top of an array is a minor thing that buys us a tiny bit over assumeSafeAppend. Doing a lot of reallocations in the process of building and not owning data is what makes Appender very poor at constructing ... array-like containers.
 So, "at the end of the day", I'm not really sure a "real" appender would pull
 its weight compared to the standard appender (IMO).
Then call it a Builder! Such a builder has a chance at good API design unlike the current abomination. In particular it could accumulate data in chunks and never reallocate until the very end. Secondly while assembling the final array/container it may move data freely both supporting non-copyable stuff _and_ being more efficient. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Oct 20 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138



--- Comment #5 from Dmitry Olshansky <dmitry.olsh gmail.com> 2013-10-20
03:32:42 PDT ---
(In reply to comment #0)
 Lets call "real" an appender which will create the one and only array, i.e. it
 will not randomly create a copy of array on appending. Such copy is bad as:
 1. Almost no use.
     In 99% appender is used as "real" appender and the copy is not needed.
 2. Performance.
     It will cause calls of postblits and then destructors on collection of copy
 array.
 3. Needless restriction on element type.
     One may want to create and array of uncopyable type which non-"real"
 appender can't do by definition.
4. Support building stuff other then built-in array on GC heap. 5. Deterministic memory management. There is no point in keeping discarded memory flaoting around after the final array is produced. 6. (sky is the limit) Use the allocators if such an ephemeral thing ever comes to materialize.
 Obviously, "real" appender is uncopyable and don't give safe access to its
 array until it finished.
Honestly the thing current Appender is used for is a Builder. As in StringBuilder of C#/Java minus the madness of extra methods. With rebranding it as Builder and few extra points I'm all for it :) -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Oct 20 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=11138


Denis Shelomovskij <verylonglogin.reg gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         AssignedTo|nobody puremagic.com        |verylonglogin.reg gmail.com


--- Comment #6 from Denis Shelomovskij <verylonglogin.reg gmail.com> 2013-10-20
15:45:49 MSD ---
(In reply to comment #5)
 With rebranding it as Builder and few extra points I'm all for it :)
And you will get it. Thanks for the positive response. As always I will create it in my Unstandard library first and make an inclusion proposal. But not sure I will do it fast. Everybody if free to reassign the issue to him, I will provide all my related code and ideas. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Oct 20 2013