www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - On Phobos GC hunt

reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
I made a proposal to quantatively measure and tabulate all GC 
allocations in Phobos before coming up with solutions to " nogc 
Phobos".

After approving node from Andrei I've come up with a piece of 
automation to extract this data and post it on wiki.

So here is the exhustive list of everything calling into GC in 
Phobos (-vgc compiler flag):

http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage

Including source links, a wild guess at function's name and the 
compiler's warning message for potential GC call.

As far as data goes this is about as good as we can get, the next 
phase is labeling this stuff with potential solution(s). Again 
doing all by hand is tedious and hardly useful.

Instead we need to observe patterns and label it automatically 
until the non-trivial subset remains. So everybody, please take 
time and identify simple patterns and post back your ideas on 
solution(s).

So far I see the most frequent cases:
- `new SomeException` - switch to RC exceptions
- AA access - ??? (use user-defined AA type as parameter?)
- array concat - ???
- closure - ???



---
Dmitry Olshansky
Oct 07 2014
next sibling parent reply "grm" <gerhard.mueller gmsoft.at> writes:
1.) It may be helpful to reduce the noise in that every match 
after a new is ignored (and probaly multiple 'operator ~' alarms 
within the same statement).

2.) There seems to be a problem with repeated alarms:
When viewing the page source, this link shows up numerous times. 
See
https://github.com/D-Programming-Language//phobos/blob/d4d98124ab6cbef7097025a7cfd1161d1963c87e/std/conv.d#L688

/Gerhard
Oct 07 2014
next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 7 October 2014 at 16:23:19 UTC, grm wrote:
 2.) There seems to be a problem with repeated alarms:
 When viewing the page source, this link shows up numerous 
 times. See
 https://github.com/D-Programming-Language//phobos/blob/d4d98124ab6cbef7097025a7cfd1161d1963c87e/std/conv.d#L688
That's because of multiple template instantiations of the same function. These should probably be filtered for this use case.
Oct 07 2014
prev sibling parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Tuesday, 7 October 2014 at 16:23:19 UTC, grm wrote:
 1.) It may be helpful to reduce the noise in that every match 
 after a new is ignored (and probaly multiple 'operator ~' 
 alarms within the same statement).
The tool currently is quick line-based hack, hence no notion of statement. It's indeed a good idea to merge all messages for one statement and de-duplicate on per statement basis.
 2.) There seems to be a problem with repeated alarms:
 When viewing the page source, this link shows up numerous 
 times. See
 https://github.com/D-Programming-Language//phobos/blob/d4d98124ab6cbef7097025a7cfd1161d1963c87e/std/conv.d#L688
There are lots of toImpl overloads, deduplication is done on module:LOC basis so the all show up. Going to fix in v2 to merge all of them in one row.
 /Gerhard
Oct 08 2014
parent "grm" <gerhard.mueller gmsoft.at> writes:
Was in a slight hurry and forgot to mention that I (quite sure: 
we all) very much appreciate the hands-on mentality your approach 
shows.

looking forward to v2

/Gerhard
Oct 08 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/7/2014 8:57 AM, Dmitry Olshansky wrote:
 I made a proposal to quantatively measure and tabulate all GC allocations in
 Phobos before coming up with solutions to " nogc Phobos".

 After approving node from Andrei I've come up with a piece of automation to
 extract this data and post it on wiki.
Thanks, Dmitri, this is great work. I suggest at a minimum that all of those get notes added to their documentation that they gc allocate.
Oct 07 2014
parent reply "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 7 October 2014 at 19:24:34 UTC, Walter Bright wrote:
 Thanks, Dmitri, this is great work. I suggest at a minimum that 
 all of those get notes added to their documentation that they 
 gc allocate.
Seems like that's something that should just be automated where possible instead of trying to update the documentation of hundreds of functions.
Oct 07 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/7/14, 12:31 PM, Brad Anderson wrote:
 On Tuesday, 7 October 2014 at 19:24:34 UTC, Walter Bright wrote:
 Thanks, Dmitri, this is great work. I suggest at a minimum that all of
 those get notes added to their documentation that they gc allocate.
Seems like that's something that should just be automated where possible instead of trying to update the documentation of hundreds of functions.
Could ddoc do that? -- Andrei
Oct 07 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-10-07 17:57, Dmitry Olshansky wrote:
 I made a proposal to quantatively measure and tabulate all GC
 allocations in Phobos before coming up with solutions to " nogc Phobos".

 After approving node from Andrei I've come up with a piece of automation
 to extract this data and post it on wiki.

 So here is the exhustive list of everything calling into GC in Phobos
 (-vgc compiler flag):

 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage

 Including source links, a wild guess at function's name and the
 compiler's warning message for potential GC call.

 As far as data goes this is about as good as we can get, the next phase
 is labeling this stuff with potential solution(s). Again doing all by
 hand is tedious and hardly useful.

 Instead we need to observe patterns and label it automatically until the
 non-trivial subset remains. So everybody, please take time and identify
 simple patterns and post back your ideas on solution(s).

 So far I see the most frequent cases:
 - `new SomeException` - switch to RC exceptions
 - AA access - ??? (use user-defined AA type as parameter?)
 - array concat - ???
 - closure - ???
I did some processing of the data and this is the results I got: 772 | 'new' causes GC allocation 515 | operator ~= may cause GC allocation 380 | operator ~ may cause GC allocation 113 | array literal may cause GC allocation 90 | setting 'length' may cause GC allocation 77 | indexing an associative array may cause GC allocation 34 | using closure causes GC allocation 16 | 'delete' requires GC 5 | associative array literal may cause GC allocation Total 9 I didn't look at any source code to see what "new" is actually allocating, for example. -- /Jacob Carlborg
Oct 07 2014
parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg wrote:
 I didn't look at any source code to see what "new" is actually 
 allocating, for example.
I did some random sampling, and it's 90% exceptions, with the occasional array allocation. I noticed that a lot of the ~ and ~= complaints are in code that only ever runs at compile time (generating strings for mixin). I wonder if there's any way we can silence these false positives.
Oct 07 2014
next sibling parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Tuesday, 7 October 2014 at 21:59:08 UTC, Peter Alexander wrote:
 On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg 
 wrote:
 I didn't look at any source code to see what "new" is actually 
 allocating, for example.
I did some random sampling, and it's 90% exceptions, with the occasional array allocation.
That's interesting. I suspected around 50%. Well that's even better since if we do ref-counted exceptions we solve 90% of problem ;)
 I noticed that a lot of the ~ and ~= complaints are in code 
 that only ever runs at compile time (generating strings for 
 mixin). I wonder if there's any way we can silence these false 
 positives.
I'm going to use blacklist for these as compiler can't in general know if it is going to be used exclusively at CTFE or not. Okay, I think I should go a bit futher with the second version of the tool. Things on todo list: - make tool general enough to work for any GitHub based project (and hackable for other hostings) - use Brian's D parser to accurately find artifacts - detect "throw new SomeStuff" pattern and automatically populate potential fix line - list all source links in one coulmn for the same function (this needs proper parser) - use blacklist of <module-name>:<artifact name> to filter out CTFE - use current data from wiki for "potential fix" column if present Holy grail is: - plot DOT call-graph of GC-users, with leafs being the ones reported by -vgc. So I start with this list then add functions them, then functions that use these functions and so on.
Oct 08 2014
parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Wednesday, 8 October 2014 at 07:52:37 UTC, Dmitry Olshansky 
wrote:
 On Tuesday, 7 October 2014 at 21:59:08 UTC, Peter Alexander

 Okay,  I think I should go a bit futher with the second version 
 of the tool.

 Things on todo list:
  - make tool general enough to work for any GitHub based 
 project (and hackable for other hostings)
  - use Brian's D parser to accurately find artifacts
  - detect "throw new SomeStuff" pattern and automatically 
 populate potential fix line
  - list all source links in one coulmn for the same function 
 (this needs proper parser)
  - use blacklist of <module-name>:<artifact name> to filter out 
 CTFE
  - use current data from wiki for "potential fix" column if 
 present
The new version is out, it's a bit rough for a proper announcement yet and misses a couple of things from my todo list but the improvement is so radical I decided to share it anyway. With the new pattern-matcher/parser I hacked together in on top of Brain's lexer it's now surgically precise in labeling artifacts. Also I retained as much as possible of original comments (line numbers have changed), and grouped source links per artifact. Updated Wiki: http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage Tool: https://github.com/DmitryOlshansky/gchunt Also it's "universal" as in any github-hosted D project, for example here is an output for druntime: http://wiki.dlang.org/Stuff_in_Druntime_That_Generates_Garbage Still todo: - blacklisting of modules/artifacts - detect usage of (i)dup - label throw new xyz as `EX` - a few bugs to fix in artifact labeling
Oct 14 2014
next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Tuesday, 14 October 2014 at 13:29:33 UTC, Dmitry Olshansky 
wrote:
 On Wednesday, 8 October 2014 at 07:52:37 UTC, Dmitry Olshansky 
 wrote:
 On Tuesday, 7 October 2014 at 21:59:08 UTC, Peter Alexander

 Okay,  I think I should go a bit futher with the second 
 version of the tool.

 Things on todo list:
 - make tool general enough to work for any GitHub based 
 project (and hackable for other hostings)
 - use Brian's D parser to accurately find artifacts
 - detect "throw new SomeStuff" pattern and automatically 
 populate potential fix line
 - list all source links in one coulmn for the same function 
 (this needs proper parser)
 - use blacklist of <module-name>:<artifact name> to filter out 
 CTFE
 - use current data from wiki for "potential fix" column if 
 present
The new version is out, it's a bit rough for a proper announcement yet and misses a couple of things from my todo list but the improvement is so radical I decided to share it anyway. With the new pattern-matcher/parser I hacked together in on top of Brain's lexer it's now surgically precise in labeling artifacts. Also I retained as much as possible of original comments (line numbers have changed), and grouped source links per artifact. Updated Wiki: http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage Tool: https://github.com/DmitryOlshansky/gchunt Also it's "universal" as in any github-hosted D project, for example here is an output for druntime: http://wiki.dlang.org/Stuff_in_Druntime_That_Generates_Garbage Still todo: - blacklisting of modules/artifacts - detect usage of (i)dup - label throw new xyz as `EX` - a few bugs to fix in artifact labeling
Thanks a million! That's very very useful.
Oct 15 2014
parent "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Wednesday, 15 October 2014 at 11:25:58 UTC, Chris wrote:
 On Tuesday, 14 October 2014 at 13:29:33 UTC, Dmitry Olshansky 
 wrote:
 On Wednesday, 8 October 2014 at 07:52:37 UTC, Dmitry Olshansky 
 wrote:
 On Tuesday, 7 October 2014 at 21:59:08 UTC, Peter Alexander

 Okay,  I think I should go a bit futher with the second 
 version of the tool.

 Things on todo list:
 - make tool general enough to work for any GitHub based 
 project (and hackable for other hostings)
 - use Brian's D parser to accurately find artifacts
 - detect "throw new SomeStuff" pattern and automatically 
 populate potential fix line
 - list all source links in one coulmn for the same function 
 (this needs proper parser)
 - use blacklist of <module-name>:<artifact name> to filter 
 out CTFE
 - use current data from wiki for "potential fix" column if 
 present
The new version is out, it's a bit rough for a proper announcement yet and misses a couple of things from my todo list but the improvement is so radical I decided to share it anyway. With the new pattern-matcher/parser I hacked together in on top of Brain's lexer it's now surgically precise in labeling artifacts. Also I retained as much as possible of original comments (line numbers have changed), and grouped source links per artifact. Updated Wiki: http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage Tool: https://github.com/DmitryOlshansky/gchunt Also it's "universal" as in any github-hosted D project, for example here is an output for druntime: http://wiki.dlang.org/Stuff_in_Druntime_That_Generates_Garbage Still todo: - blacklisting of modules/artifacts - detect usage of (i)dup - label throw new xyz as `EX` - a few bugs to fix in artifact labeling
Thanks a million! That's very very useful.
I sure hoped so! :) Sadly I'm going to be incredibly busy this weekend, so the proper release date shifts to sometime afterwards.
Oct 16 2014
prev sibling parent "safety0ff" <safety0ff.dev gmail.com> writes:
On Tuesday, 14 October 2014 at 13:29:33 UTC, Dmitry Olshansky 
wrote:
 Also it's "universal" as in any github-hosted D project, for 
 example here is an output for druntime:

 http://wiki.dlang.org/Stuff_in_Druntime_That_Generates_Garbage

 Still todo:
  - a few bugs to fix in artifact labeling
One artefact labelling bug I noticed was GC.removeRange and GC.removeRoot were placed in the Artefact column where it should have been GC.rangeIter and GC.rootIter.
Oct 18 2014
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 07 Oct 2014 21:59:05 +0000
schrieb "Peter Alexander" <peter.alexander.au gmail.com>:

 On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg wrote:
 I didn't look at any source code to see what "new" is actually 
 allocating, for example.
I did some random sampling, and it's 90% exceptions, with the occasional array allocation. I noticed that a lot of the ~ and ~= complaints are in code that only ever runs at compile time (generating strings for mixin). I wonder if there's any way we can silence these false positives.
Code in if(__ctfe) blocks could be (and should be) allowed: https://github.com/D-Programming-Language/dmd/pull/3572 But if you have got a normal function (string generateMixin()) the compiler can't really know that it's only used at compile time. And if it's not a template the code using the GC will be compiled, even if it's never called. This might be enough to get undefined symbol errors if you don't have an GC, so the error messages are kinda valid.
Oct 08 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/14, 1:13 AM, Johannes Pfau wrote:
 Am Tue, 07 Oct 2014 21:59:05 +0000
 schrieb "Peter Alexander" <peter.alexander.au gmail.com>:

 On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg wrote:
 I didn't look at any source code to see what "new" is actually
 allocating, for example.
I did some random sampling, and it's 90% exceptions, with the occasional array allocation. I noticed that a lot of the ~ and ~= complaints are in code that only ever runs at compile time (generating strings for mixin). I wonder if there's any way we can silence these false positives.
Code in if(__ctfe) blocks could be (and should be) allowed: https://github.com/D-Programming-Language/dmd/pull/3572 But if you have got a normal function (string generateMixin()) the compiler can't really know that it's only used at compile time. And if it's not a template the code using the GC will be compiled, even if it's never called. This might be enough to get undefined symbol errors if you don't have an GC, so the error messages are kinda valid.
That's a bummer. Can we get the compiler to remove the "if (__ctfe)" code after semantic checking? Andrei
Oct 08 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/14, 1:01 PM, Andrei Alexandrescu wrote:
 On 10/8/14, 1:13 AM, Johannes Pfau wrote:
 Am Tue, 07 Oct 2014 21:59:05 +0000
 schrieb "Peter Alexander" <peter.alexander.au gmail.com>:

 On Tuesday, 7 October 2014 at 20:13:32 UTC, Jacob Carlborg wrote:
 I didn't look at any source code to see what "new" is actually
 allocating, for example.
I did some random sampling, and it's 90% exceptions, with the occasional array allocation. I noticed that a lot of the ~ and ~= complaints are in code that only ever runs at compile time (generating strings for mixin). I wonder if there's any way we can silence these false positives.
Code in if(__ctfe) blocks could be (and should be) allowed: https://github.com/D-Programming-Language/dmd/pull/3572 But if you have got a normal function (string generateMixin()) the compiler can't really know that it's only used at compile time. And if it's not a template the code using the GC will be compiled, even if it's never called. This might be enough to get undefined symbol errors if you don't have an GC, so the error messages are kinda valid.
That's a bummer. Can we get the compiler to remove the "if (__ctfe)" code after semantic checking?
Or would "static if (__ctfe)" work? -- Andrei
Oct 08 2014
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 10/8/14 4:10 PM, Andrei Alexandrescu wrote:
 On 10/8/14, 1:01 PM, Andrei Alexandrescu wrote:
 That's a bummer. Can we get the compiler to remove the "if (__ctfe)"
 code after semantic checking?
Or would "static if (__ctfe)" work? -- Andrei
Please don't ask me to explain why, because I still don't know. But _ctfe is a normal runtime variable :) It has been explained to me before, why it has to be a runtime variable. I think Don knows the answer. -Steve
Oct 08 2014
parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Wednesday, 8 October 2014 at 20:15:51 UTC, Steven 
Schveighoffer wrote:
 On 10/8/14 4:10 PM, Andrei Alexandrescu wrote:
 On 10/8/14, 1:01 PM, Andrei Alexandrescu wrote:
 That's a bummer. Can we get the compiler to remove the "if 
 (__ctfe)"
 code after semantic checking?
Or would "static if (__ctfe)" work? -- Andrei
Please don't ask me to explain why, because I still don't know. But _ctfe is a normal runtime variable :) It has been explained to me before, why it has to be a runtime variable. I think Don knows the answer.
Well, the contents of the static if expression have to be evaluated at compile time, so static if (__ctfe) would always be true. Also, if it were to somehow work as imagined then you'd have nonsensical things like this: static if (__ctfe) class Wat {} auto foo() { static if (__ctfe) return new Wat(); return null; } static wat = foo(); wat now has a type at runtime that only exists at compile time.
Oct 08 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Or would "static if (__ctfe)" work? -- Andrei
Currently it doesn't work, because __ctfe is a run-time variable. Walter originally tried and failed to make it a compile-time variable. Bye, bearophile
Oct 08 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 08 Oct 2014 13:10:11 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 Or would "static if (__ctfe)" work? -- Andrei
ha! The Famous Bug! it works, but not as people expected. as "static if" evaluates when function is *compiling*, __ctfe is false there, and so the whole "true" branch will be removed as dead code. i believe that compiler should warn about this, 'cause i'm tend to repeatedly hit this funny thing.
Oct 08 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 8 Oct 2014 23:20:13 +0300
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:

 On Wed, 08 Oct 2014 13:10:11 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:
=20
 Or would "static if (__ctfe)" work? -- Andrei
ha! The Famous Bug! it works, but not as people expected. as "static if" evaluates when function is *compiling*, __ctfe is false there, and so the whole "true" branch will be removed as dead code. =20 i believe that compiler should warn about this, 'cause i'm tend to repeatedly hit this funny thing.
Lol, definitely! I made that mistake myself and Robert Schadek, too in his std.logger. It is now the #1 bug in D code. --=20 Marco
Oct 10 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 10 Oct 2014 09:14:28 +0200
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Am Wed, 8 Oct 2014 23:20:13 +0300
 schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:
=20
 On Wed, 08 Oct 2014 13:10:11 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:
=20
 Or would "static if (__ctfe)" work? -- Andrei
ha! The Famous Bug! it works, but not as people expected. as "static if" evaluates when function is *compiling*, __ctfe is false there, and so the whole "true" branch will be removed as dead code. =20 i believe that compiler should warn about this, 'cause i'm tend to repeatedly hit this funny thing.
=20 Lol, definitely! I made that mistake myself and Robert Schadek, too in his std.logger. It is now the #1 bug in D code.
i made a quick patch that warns on "static if (__ctfe)": https://issues.dlang.org/show_bug.cgi?id=3D13601
Oct 10 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 8 Oct 2014 23:20:13 +0300
ketmar via Digitalmars-d <digitalmars-d puremagic.com> wrote:

p.s. or vice versa: "static if (__ctfe)" is always true, to non-ctfe
code will be removed. sorry, i can't really remember what is true, but
anyway, it works by removeing one of the branches altogether.
Oct 08 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/08/2014 10:25 PM, ketmar via Digitalmars-d wrote:
 On Wed, 8 Oct 2014 23:20:13 +0300
 ketmar via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 p.s. or vice versa: "static if (__ctfe)" is always true, to non-ctfe
 code will be removed. sorry, i can't really remember what is true, but
 anyway, it works by removeing one of the branches altogether.
This is probably a regression somewhere after 2.060, because with 2.060 I get Error: variable __ctfe cannot be read at compile time Error: expression __ctfe is not constant or does not evaluate to a bool as I'd expect.
Oct 08 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 08 Oct 2014 23:40:01 +0200
Timon Gehr via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 This is probably a regression somewhere after 2.060, because with
 2.060 I get
=20
 Error: variable __ctfe cannot be read at compile time
 Error: expression __ctfe is not constant or does not evaluate to a
 bool
=20
 as I'd expect.
i remember now that i was copypasting toHash() from druntime some time ago and changed "if (__ctfe)" to "static if (__ctfe)" in process. it compiles and works fine, and i don't even noticed what i did until i tried to change non-ctfe part of toHash() and found that my changes had no effect at all. and then i discovered that "static". this was 2.066 or 2.067-git. and now i can clearly say that "static if (__ctfe)" leaving only ctfe part. that was somewhat confusing, as i was pretty sure that "if (__ctfe)" *must* be used with "static".
Oct 08 2014
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/08/2014 11:40 PM, Timon Gehr wrote:
 This is probably a regression somewhere after 2.060, because with 2.060
 I get

 Error: variable __ctfe cannot be read at compile time
 Error: expression __ctfe is not constant or does not evaluate to a bool

 as I'd expect.
Marked the bugzilla case as regression. https://issues.dlang.org/show_bug.cgi?id=13601
Oct 18 2014
prev sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 8 Oct 2014 23:25:18 +0300
ketmar via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wed, 8 Oct 2014 23:20:13 +0300
 ketmar via Digitalmars-d <digitalmars-d puremagic.com> wrote:
=20
 p.s. or vice versa: "static if (__ctfe)" is always true, to non-ctfe
 code will be removed. sorry, i can't really remember what is true, but
 anyway, it works by removeing one of the branches altogether.
hm. i need some sleep. or new keyboard. or both.
Oct 08 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 08 Oct 2014 13:01:43 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 10/8/14, 1:13 AM, Johannes Pfau wrote:
 Code in if(__ctfe) blocks could be (and should be) allowed:
 https://github.com/D-Programming-Language/dmd/pull/3572

 But if you have got a normal function (string generateMixin()) the
 compiler can't really know that it's only used at compile time. And
 if it's not a template the code using the GC will be compiled, even
 if it's never called. This might be enough to get undefined symbol
 errors if you don't have an GC, so the error messages are kinda
 valid.
That's a bummer. Can we get the compiler to remove the "if (__ctfe)" code after semantic checking? Andrei
I think you misunderstood, code in if(__ctfe) is already removed, it never gets into the binary. But the nogc/-vgc checks still complain about GC allocations in if(__ctfe). This is easy to fix, but as ctfe is a runtime variable you could also do (if(__ctfe || dice() == 1 )) and the decision about complex cases stopped pull #3572. What I meant is that the compiler can't know that this code is CTFE-only and -vgc must complain: string generateMixin(string a) {return "int " ~ a ~ ";";} mixin(generateMixin()); But there are workarounds: http://dpaste.dzfl.pl/e689585c0a95 (Note that dead-code elimination should be able to remove all functions marked as private)
Oct 09 2014
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/08/2014 10:01 PM, Andrei Alexandrescu wrote:
 That's a bummer. Can we get the compiler to remove the "if (__ctfe)"
 code after semantic checking?

 Andrei
It seems that __ctfe is treated as constant in the backend. At least there is no asm code generated for these examples (even without -O). void main() { import std.stdio; if (__ctfe) writeln("foo"); } int main() { return __ctfe ? 1 : 0; }
Oct 18 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Martin Nowak"  wrote in message news:m1udcn$s35$1 digitalmars.com...

 On 10/08/2014 10:01 PM, Andrei Alexandrescu wrote:
 That's a bummer. Can we get the compiler to remove the "if (__ctfe)"
 code after semantic checking?

 Andrei
It seems that __ctfe is treated as constant in the backend.
 At least there is no asm code generated for these examples (even without
-O). void main() { import std.stdio; if (__ctfe) writeln("foo"); } int main() { return __ctfe ? 1 : 0; }
__ctfe is passed to the backend as 'false', and even with -O the basic optimizations will strip it as dead code. So it is removed after semantic checking, but that doesn't help as semantic checking includes nogc checking.
Oct 20 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/7/14, 8:57 AM, Dmitry Olshansky wrote:
 I made a proposal to quantatively measure and tabulate all GC
 allocations in Phobos before coming up with solutions to " nogc Phobos".

 After approving node from Andrei I've come up with a piece of automation
 to extract this data and post it on wiki.

 So here is the exhustive list of everything calling into GC in Phobos
 (-vgc compiler flag):

 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage
Awesome! I've started adding explanations to the first few entries, let's use that crowdsourcing thing to fill this! -- Andrei
Oct 07 2014
prev sibling next sibling parent "Xinok" <xinok live.com> writes:
On Tuesday, 7 October 2014 at 15:57:59 UTC, Dmitry Olshansky 
wrote:
 So here is the exhustive list of everything calling into GC in 
 Phobos (-vgc compiler flag):

 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage
A correction on TimSortImpl, it does actually generate garbage by calling uninitializedArray to allocate the buffer. The check for __ctfe is unnecessary (it may have been needed sometime ago or was added naively). Timsort is an O(n/2) algorithm and requires a buffer, but there's no reason for it to be GC-allocated. It could simply be malloc'd and free'd before the function returns.
Oct 07 2014
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 07 Oct 2014 15:57:58 +0000
schrieb "Dmitry Olshansky" <dmitry.olsh gmail.com>:

 
 Instead we need to observe patterns and label it automatically 
 until the non-trivial subset remains. So everybody, please take 
 time and identify simple patterns and post back your ideas on 
 solution(s).
 
I just had a look at all closure allocations and identified these patterns: 1) Fixable by manually stack-allocating closure A delegate is passed to some function which stores this delegate and therefore correctly doesn't mark the parameter as scope. However, the lifetime of the stored delegate is still limited to the current function (e.g. it's stored in a struct instance, but on the stack). Can be fixed by creating a static struct{T... members; void doSomething(){access members}} instance on stack and passing &stackvar.doSomething as delegate. 2) Using delegates to add state to ranges ---- return iota(dim). filter!(i => ptr[i])(). map!(i => BitsSet!size_t(ptr[i], i * bitsPerSizeT))(). joiner(); ---- This code adds state to ranges without declaring a new type: the ptr variable is not accessible and needs to be move into a closure. Declaring a custom range type is a solution, but not straightforward: If the ptr field is moved into the range a closure is not necessary. But if the range is copied, it's address changes and the delegate passed to map is now invalid. 3) Functions taking delegates as generic parameters receiveTimeout,receive,formattedWrite accept different types, including delegates. The delegates can all be scope to avoid the allocation but is void foo(T)(scope T) a good idea? The alternative is probably making an overload for delegates with scope attribute. (The result is that all functions calling receiveTimeout,... with a delegate allocate a closure) 4) Solvable with manual memory management Some specific functions can't be easily fixed, but the delegates they create have a well defined lifetime (for example spawn creates a delegate which is only needed at the startup of a new thread, it's never used again). These could be malloc+freed. 5) Design issue These functions generally create a delegate using variables passed in as parameters. There's no way to avoid closures here. Although manual allocation is an possible, the lifetime is undefined and can only be managed by the GC. 6) Other Two cases can be fixed by moving a buffer into a struct or moving a function out of a member function into it's surrounding class. Also notable: 17 out of 35 cases are in std.net.curl. This is because curl heavily uses delegates and wrapper delegates.
Oct 08 2014
parent reply "Dmitry Olshansky" <dmitry.olsh gmail.com> writes:
On Wednesday, 8 October 2014 at 11:25:25 UTC, Johannes Pfau wrote:
 Am Tue, 07 Oct 2014 15:57:58 +0000
 schrieb "Dmitry Olshansky" <dmitry.olsh gmail.com>:

 
 Instead we need to observe patterns and label it automatically 
 until the non-trivial subset remains. So everybody, please 
 take time and identify simple patterns and post back your 
 ideas on solution(s).
 
I just had a look at all closure allocations and identified these patterns:
Awesome! This is exactly the kind of help I wanted.
 1) Fixable by manually stack-allocating closure
    A delegate is passed to some function which stores this 
 delegate and
    therefore correctly doesn't mark the parameter as scope. 
 However,
    the lifetime of the stored delegate is still limited to the 
 current
    function (e.g. it's stored in a struct instance, but on the 
 stack).

    Can be fixed by creating a static struct{T... members; void
    doSomething(){access members}} instance on stack and passing
    &stackvar.doSomething as delegate.
Hm... Probably we can create a template for this.
 2) Using delegates to add state to ranges
    ----
    return iota(dim).
      filter!(i => ptr[i])().
      map!(i => BitsSet!size_t(ptr[i], i * bitsPerSizeT))().
      joiner();
    ----
    This code adds state to ranges without declaring a new type: 
 the ptr
    variable is not accessible and needs to be move into a 
 closure.
    Declaring a custom range type is a solution, but not
    straightforward: If the ptr field is moved into the range a 
 closure
    is not necessary. But if the range is copied, it's address 
 changes
    and the delegate passed to map is now invalid.
Indeed, such code is fine in "user-space" but have no place in the library.
 3) Functions taking delegates as generic parameters
    receiveTimeout,receive,formattedWrite accept different types,
    including delegates. The delegates can all be scope to avoid 
 the
    allocation but is void foo(T)(scope T) a good idea? The 
 alternative
    is probably making an overload for delegates with scope 
 attribute.

    (The result is that all functions calling receiveTimeout,... 
 with a
    delegate allocate a closure)

 4) Solvable with manual memory management
    Some specific functions can't be easily fixed, but the 
 delegates
    they create have a well defined lifetime (for example spawn 
 creates
    a delegate which is only needed at the startup of a new 
 thread, it's
    never used again). These could be malloc+freed.
I think this and (2) can be solved if we come up with solid support for RC-closures.
 5) Design issue
    These functions generally create a delegate using variables 
 passed
    in as parameters. There's no way to avoid closures here. 
 Although
    manual allocation is an possible, the lifetime is undefined 
 and can
    only be managed by the GC.

 6) Other
    Two cases can be fixed by moving a buffer into a struct or 
 moving a
    function out of a member function into it's surrounding 
 class.
Yeah, there are always outliers ;)
 Also notable: 17 out of 35 cases are in std.net.curl. This is 
 because
 curl heavily uses delegates and wrapper delegates.
Interesting... it must be due to cURL callback-based API. All in all, std.net.curl is a constant source of complaints, it may need some work to fix other issues anyway.
Oct 08 2014
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 8 October 2014 at 12:09:00 UTC, Dmitry Olshansky 
wrote:
 I think this and (2) can be solved if we come up with solid 
 support for RC-closures.
Delegates don't obey data sharing type checks though. A long standing language issue.
Oct 08 2014
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Tue, 07 Oct 2014 15:57:58 +0000
schrieb "Dmitry Olshansky" <dmitry.olsh gmail.com>:

 I made a proposal to quantatively measure and tabulate all GC 
 allocations in Phobos before coming up with solutions to " nogc 
 Phobos".
 
 After approving node from Andrei I've come up with a piece of 
 automation to extract this data and post it on wiki.
 
 So here is the exhustive list of everything calling into GC in 
 Phobos (-vgc compiler flag):
 
 http://wiki.dlang.org/Stuff_in_Phobos_That_Generates_Garbage
 
 Including source links, a wild guess at function's name and the 
 compiler's warning message for potential GC call.
 
 As far as data goes this is about as good as we can get, the next 
 phase is labeling this stuff with potential solution(s). Again 
 doing all by hand is tedious and hardly useful.
 
 Instead we need to observe patterns and label it automatically 
 until the non-trivial subset remains. So everybody, please take 
 time and identify simple patterns and post back your ideas on 
 solution(s).
 
 So far I see the most frequent cases:
 - `new SomeException` - switch to RC exceptions
 - AA access - ??? (use user-defined AA type as parameter?)
 - array concat - ???
 - closure - ???
 
 
 
 ---
 Dmitry Olshansky
Another observation: idup/dup are not reported by -vgc (This is correct behavior. nogc detects these as normal functions without nogc attribute and complains. -vgc does not report calls to non- nogc functions). However, idup/dup might be common and it might make sense to grep for them manually?
Oct 09 2014