www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Using core.reflect to rewrite code

reply Stefan Koch <uplink.coder googlemail.com> writes:
Good evening everyone,

before going to bed I have been playing with my new project again.

This time I wanted to generate a D wrapper struct.

core.reflect can be used to transform

```D
extern (C)
{
     struct FwdCType;
     void setText(FwdCType* ctx, const char* text);
     void setInt(FwdCType* c, const int value);

     int unrelated_function();

     ubyte* getBlob(const FwdCType* ctx);
}
```

into

```D
struct FwdCTypeWrapper {
     FwdCType* ctx;

     void setText (const(char*) text) {
         setText(ctx, text);
     }

     void setInt (const(int) value) {
         setInt(ctx, value);
     }

     ubyte* getBlob () {
         getBlob(ctx);
     }

}
```
Note that the wrapping generation was smart enough to omit the 
unrelated function because no pointer to the context struct was 
detected in the signature.

The code that does this is just over 80 lines of code (omitting 
the boilerplate).
and it can be seen here:
https://gist.github.com/UplinkCoder/93cb06e4921ab4c96752c6325e03e42d
Aug 10 2021
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 Good evening everyone,

 before going to bed I have been playing with my new project 
 again.


I've manged to improve the performance of using core.reflect by 
roughly 2x because I realized that class literals created by 
core.reflect don't need "scrubbing".

Scrubbing in this case means walking the expression-tree of an 
expression created by CTFE and removing "ctfe-only-features" from 
it thereby making it a proper ASTNode which can be inserted 
without "damaging" the tree.

To be honest I am surprised that I was able to get a 2x 
performance win for core.reflect without major architectural 
changes.

I guess sometimes one is lucky :)

Now core.reflect is relatively fast it's time to play around with 
it :)

Please let me know what you think about the examples I have 
posted so far.

Cheers,

Stefan
Aug 11 2021
next sibling parent Brian Tiffin <btiffin gnu.org> writes:
On Wednesday, 11 August 2021 at 23:58:21 UTC, Stefan Koch wrote:

 On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 Good evening everyone,

 before going to bed I have been playing with my new project 
 again.


 I've manged to improve the performance of using core.reflect by 
 roughly 2x because I realized that class literals created by 
 core.reflect don't need "scrubbing".
 ...
 Cheers,

 Stefan


I'm too new here, and probably speaking out of turn, not at a 
level to profit from the core reflections you are working on, 
Stefan, but I just want to add a cheering response.  Sounds very 
cool, and I look forward to the day I can write some D that takes 
advantage.

Well done.  *Even though I'm talking out a hole that doesn't know 
anything about the subject matter.*

Have good, make well.
Aug 11 2021
prev sibling parent reply max haughton <maxhaton gmail.com> writes:
On Wednesday, 11 August 2021 at 23:58:21 UTC, Stefan Koch wrote:

 On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 [...]


 I've manged to improve the performance of using core.reflect by 
 roughly 2x because I realized that class literals created by 
 core.reflect don't need "scrubbing".

 [...]


Can you profile it vs. a template metaprogramming solution?
Aug 11 2021
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 12 August 2021 at 02:12:23 UTC, max haughton wrote:

 On Wednesday, 11 August 2021 at 23:58:21 UTC, Stefan Koch wrote:

 On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 [...]


 I've manged to improve the performance of using core.reflect 
 by roughly 2x because I realized that class literals created 
 by core.reflect don't need "scrubbing".

 [...]


 Can you profile it vs. a template metaprogramming solution?


Yes but I am still not quite sure how to write the equivalent 
template ;)
Aug 11 2021
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Thursday, 12 August 2021 at 04:47:52 UTC, Stefan Koch wrote:

 Yes but I am still not quite sure how to write the equivalent 
 template ;)


You'd just loop through the members, see if there's the struct 
pointer as the argument, and if so, generate a forwarder function 
with the &this passed to it.

Though I wouldn't generate an actual wrapper here since UFCS does 
the same thing in practice, I have done similar in my script 
wrapper for ufcs

http://arsd-official.dpldocs.info/source/arsd.jsvar.d.html#L2480
Aug 13 2021
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 13 August 2021 at 12:51:21 UTC, Adam D Ruppe wrote:

 On Thursday, 12 August 2021 at 04:47:52 UTC, Stefan Koch wrote:

 Yes but I am still not quite sure how to write the equivalent 
 template ;)


 You'd just loop through the members, see if there's the struct 
 pointer as the argument, and if so, generate a forwarder 
 function with the &this passed to it.

 Though I wouldn't generate an actual wrapper here since UFCS 
 does the same thing in practice, I have done similar in my 
 script wrapper for ufcs

 http://arsd-official.dpldocs.info/source/arsd.jsvar.d.html#L2480


Thanks for your input.
In the meantime I have been able to reproduce the behavior in a 
template.
`WrapperString` in the following gist 
https://gist.github.com/UplinkCoder/e60584a2c8f46ae4a490117b878ecec1

The performance of the template is roughly 2 times worse when 
compared to the `core.reflect` solution still.
Aug 14 2021
parent reply max haughton <maxhaton gmail.com> writes:
On Saturday, 14 August 2021 at 11:15:50 UTC, Stefan Koch wrote:

 On Friday, 13 August 2021 at 12:51:21 UTC, Adam D Ruppe wrote:

 [...]


 Thanks for your input.
 In the meantime I have been able to reproduce the behavior in a 
 template.
 `WrapperString` in the following gist 
 https://gist.github.com/UplinkCoder/e60584a2c8f46ae4a490117b878ecec1

 The performance of the template is roughly 2 times worse when 
 compared to the `core.reflect` solution still.


That's very promising, is this currently relying on your Task 
work or could it work in the compiler as is?
Aug 14 2021
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 14 August 2021 at 14:32:08 UTC, max haughton wrote:

 On Saturday, 14 August 2021 at 11:15:50 UTC, Stefan Koch wrote:

 On Friday, 13 August 2021 at 12:51:21 UTC, Adam D Ruppe wrote:

 [...]


 Thanks for your input.
 In the meantime I have been able to reproduce the behavior in 
 a template.
 `WrapperString` in the following gist 
 https://gist.github.com/UplinkCoder/e60584a2c8f46ae4a490117b878ecec1

 The performance of the template is roughly 2 times worse when 
 compared to the `core.reflect` solution still.


 That's very promising, is this currently relying on your Task 
 work or could it work in the compiler as is?


It can be used as is.

The task system is not required.

Since unlike typefunctions is does not relay on partially 
resolved nodes.
Aug 14 2021
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 12 August 2021 at 02:12:23 UTC, max haughton wrote:

 On Wednesday, 11 August 2021 at 23:58:21 UTC, Stefan Koch wrote:

 On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 [...]


 I've manged to improve the performance of using core.reflect 
 by roughly 2x because I realized that class literals created 
 by core.reflect don't need "scrubbing".

 [...]


 Can you profile it vs. a template metaprogramming solution?


TL;DR
The template solution [1] takes 60% more time and 80% more memory
compared to the core.reflect solution.
While being more difficult to write and therefore being less 
useful.

I have written a template solution, while it does not offer the 
same functionality.
It should provide a reasonable lower bound for one.
The template only collect structs and functions.
Writing the correct filtering and constructing the string is a 
task for another day :)




Here are the results:
```
uplink uplink-black:~/d/dmd(core_reflect)$ hyperfine 
"generated/linux/release/64/dmd -c testTemplate.d" 
"generated/linux/release/64/dmd -c testCollector.d"

   Time (mean ± σ):      28.5 ms ±   2.3 ms    [User: 21.9 ms, 
System: 6.8 ms]
   Range (min … max):    21.1 ms …  33.2 ms    95 runs


   Time (mean ± σ):      21.7 ms ±   2.7 ms    [User: 16.5 ms, 
System: 5.4 ms]
   Range (min … max):    13.8 ms …  27.6 ms    147 runs

Summary
   'generated/linux/release/64/dmd -c testCollector.d' ran
     1.31 ± 0.19 times faster than 'generated/linux/release/64/dmd 
-c testTemplate.d'
```

and a little proxy for memory use as well as the output

```
uplink uplink-black:~/d/dmd(core_reflect)$ /usr/bin/time 
generated/linux/release/64/dmd -c testTemplate.d
tuple(getName, getOrdinal, setName, unrelated)
(Ctx)
0.02user 0.00system 0:00.03elapsed 96%CPU (0avgtext+0avgdata 
25824maxresident)k
0inputs+8outputs (0major+5191minor)pagefaults 0swaps
```

```
uplink uplink-black:~/d/dmd(core_reflect)$ /usr/bin/time 
generated/linux/release/64/dmd -c testCollector.d
class CtxWrapper {
     const(char*) getName () {
         getName(ctx);
     }

     uint getOrdinal () {
         getOrdinal(ctx);
     }

     void setName (const(char*) name) {
         setName(ctx, name);
     }

}

0.02user 0.00system 0:00.02elapsed 104%CPU (0avgtext+0avgdata 
14364maxresident)k
0inputs+40outputs (0major+2245minor)pagefaults 0swaps
```

template solution benchmarked:
[1] 
https://gist.github.com/UplinkCoder/c2838252c55c9fdf4fc526e2a8c5ce7e

core.reflect solution benchmarked:
[2] 
https://gist.github.com/UplinkCoder/93cb06e4921ab4c96752c6325e03e42d
Aug 12 2021
prev sibling parent reply surlymoor <surlymoor cock.li> writes:
On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 [...]


I enjoy your work, but what is the probability; assuming this 
functionality's consummation; of core.reflect's addition to 
druntime? I remember type functions weren't received well by 
Walter (and Andrei?).
Aug 11 2021
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 12 August 2021 at 05:53:58 UTC, surlymoor wrote:

 On Tuesday, 10 August 2021 at 23:51:29 UTC, Stefan Koch wrote:

 [...]


 I enjoy your work, but what is the probability; assuming this 
 functionality's consummation; of core.reflect's addition to 
 druntime? I remember type functions weren't received well by 
 Walter (and Andrei?).


I do not know. I have not spoken to them yet.

core.reflect is conceptually and in regards of implementation 
much simpler than type-functions were; so I would hope that they 
would be accepted.

Regards,
Stefan
Aug 12 2021