www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Runtime code reloading in D, part 1

reply Benjamin Thaut <code benjamin-thaut.de> writes:
An article about runtime code reloading in the context of game 
developement. A topic I'm currently working on in my spare time. I hope 
it holds some valuable information for everyone working with D.

http://3d.benjamin-thaut.de/?p=25

Kind Regards
Benjamin Thaut
Dec 30 2012
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:

 http://3d.benjamin-thaut.de/?p=25
The pointer to the RTTI information for each member is a 
workaround for a strange bug I encountered. For POD structs the 
rtInfo template will be instancated, but for some reason you can 
not query the results at runtime. It is possible that the linker 
stripps the information away but I could not yet investigate 
this issue further. If I can not query the RTTI information at 
runtime I fallback to the RTTI info pointer stored in the “next” 
member of thMemberInfo.<
This seems one of the situations where knowing "why" is almost more important than "doing": understanding the cause, and maybe even fixing it, will help future D programmers. Bye, bearophile
Dec 30 2012
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.12.2012 14:27, schrieb bearophile:
 This seems one of the situations where knowing "why" is almost more
 important than "doing": understanding the cause, and maybe even fixing
 it, will help future D programmers.

 Bye,
 bearophile
The problem only happens if you have a POD struct in a library file (.lib). If the POD struct is directly in the main executable everything works just fine. The only hint Walter gave me when I opened a thread in the D newsgroup was "look at the difference of the object files". I don't really know how that should be done atm. Kind Regards Benjamin Thaut
Dec 30 2012
prev sibling next sibling parent "F i L" <witte2008 gmail.com> writes:
On Sunday, 30 December 2012 at 12:32:00 UTC, Benjamin Thaut wrote:
 An article about runtime code reloading in the context of game 
 developement. A topic I'm currently working on in my spare 
 time. I hope it holds some valuable information for everyone 
 working with D.

 http://3d.benjamin-thaut.de/?p=25

 Kind Regards
 Benjamin Thaut
Awesome, thanks for the post. I've been playing around with this idea for awhile, and that article is immensely helpful. Proper .so/.dll support is very needed for this, and I hope something is happening on that front.
Dec 30 2012
prev sibling next sibling parent reply r_m_r <r_m_r mailinator.com> writes:
On 12/30/2012 06:02 PM, Benjamin Thaut wrote:
 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.
Thanks for the article. It was very informative. Waiting for the next part :-) regards, r_m_r
Dec 30 2012
parent reply "DypthroposTheImposter" <mcbracket gmail.com> writes:
   Do you find that D without GC is more effective than C++? Seems 
like you would be stuck using structs which seems somewhat 
limiting, even compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they 
use C++ and rely more on the ability to serialize everything
Dec 31 2012
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 31.12.2012 15:02, schrieb DypthroposTheImposter:
    Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

   UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything
Why should I be stuck with structs? Its the exact same as in C++. I did build my own new and delete operators (as templates). It's not that hard and works pretty well, only the syntax is not ideal in some cases. I can use everything you can use with D+GC. Sometimes you have to be carefull with implict allocations (e.g. closures, array literals) but I have a leak detector that points me to these directly and usually its easy to free these manually. And I'm quite a bit more productive then in C++. Module constructors with a defined order instead of random static initalizers, code generation isnstead of huge amounts of boilerplate code and many other features are the cause of this. Kind Regards Benjamin Thaut
Dec 31 2012
parent reply "ixid" <nuaccount gmail.com> writes:
On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:
 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:
   Do you find that D without GC is more effective than C++? 
 Seems like
 you would be stuck using structs which seems somewhat 
 limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though 
 they use
 C++ and rely more on the ability to serialize everything
Why should I be stuck with structs? Its the exact same as in C++. I did build my own new and delete operators (as templates). It's not that hard and works pretty well, only the syntax is not ideal in some cases. I can use everything you can use with D+GC. Sometimes you have to be carefull with implict allocations (e.g. closures, array literals) but I have a leak detector that points me to these directly and usually its easy to free these manually. And I'm quite a bit more productive then in C++. Module constructors with a defined order instead of random static initalizers, code generation isnstead of huge amounts of boilerplate code and many other features are the cause of this. Kind Regards Benjamin Thaut
Is D moving away from your sort of use? Games and bioinformatics would seem to be the areas the language should be trying to get people to start using it in. The features you're using seem very much like they should be a part and mode of using the language.
Jan 04 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 05.01.2013 05:24, schrieb ixid:
 On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:
 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:
   Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything
Why should I be stuck with structs? Its the exact same as in C++. I did build my own new and delete operators (as templates). It's not that hard and works pretty well, only the syntax is not ideal in some cases. I can use everything you can use with D+GC. Sometimes you have to be carefull with implict allocations (e.g. closures, array literals) but I have a leak detector that points me to these directly and usually its easy to free these manually. And I'm quite a bit more productive then in C++. Module constructors with a defined order instead of random static initalizers, code generation isnstead of huge amounts of boilerplate code and many other features are the cause of this. Kind Regards Benjamin Thaut
Is D moving away from your sort of use? Games and bioinformatics would seem to be the areas the language should be trying to get people to start using it in. The features you're using seem very much like they should be a part and mode of using the language.
I wouldn't say its moving away from it. Some recent changes to druntime have made it significantly less leaking. But on the other hand a API design like toString() which pretty much does leak in almost all cases don't exactly help a GC free D. In Summary it feels to me that GC free D is not important to the community or the active contributors. I also see D's biggest chances in becoming popular in the performance critical fields of programming. Which would be systems programming, Gaming and others. For programming fields in which a GC is applicable people tend to use languages like C# or Java, because they are truly safe (see the recent "ref is unsafe" discussion in the newsgroup), they have nice productivity features like runtime code changing through the VM and have a way better GC because the language was designed from the beginning to support a advanced GC. Kind Regards Benjamin Thaut
Jan 06 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:

 In Summary it feels to me that GC free D is not important
 to the community or the active contributors.
I think it will become more important for them, in future. At the moment the work is mostly on finishing immutability, purity, shared, and other parts of the core language, and working on process characteristics (like GIT workflows) that are needed to make everything else work efficiently.
 they have nice productivity features like runtime code changing 
 through the VM
Some of those productivity features are important, and despite D doesn't run on a virtual machine I think some of them can be added to D, once enough attention and work is given on them. Bye, bearophile
Jan 06 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 11:37, Benjamin Thaut wrote:

 I wouldn't say its moving away from it. Some recent changes to druntime
 have made it significantly less leaking. But on the other hand a API
 design like toString() which pretty much does leak in almost all cases
 don't exactly help a GC free D. In Summary it feels to me that GC free D
 is not important to the community or the active contributors.
The design of toString() has been up for debate a couple of times. Many are not happy with the design. -- /Jacob Carlborg
Jan 06 2013
parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 12:10:38 +0100
schrieb Jacob Carlborg <doob me.com>:

 On 2013-01-06 11:37, Benjamin Thaut wrote:
 
 I wouldn't say its moving away from it. Some recent changes to
 druntime have made it significantly less leaking. But on the other
 hand a API design like toString() which pretty much does leak in
 almost all cases don't exactly help a GC free D. In Summary it
 feels to me that GC free D is not important to the community or the
 active contributors.
The design of toString() has been up for debate a couple of times. Many are not happy with the design.
some modules already provide the (scope void delegate(const(char)[]) sink overload, it's also supported by to, format and writefln, afaik. https://github.com/D-Programming-Language/phobos/blob/master/std/uuid.d#L806 I guess when we have custom allocators we can also provide toString methods templated on allocators.
Jan 06 2013
parent reply David <d dav1d.de> writes:
 I guess when we have custom allocators we can also provide toString
 methods templated on allocators.
 
Custom allocators released together with HL3?
Jan 06 2013
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 13:14:39 +0100
schrieb David <d dav1d.de>:

 I guess when we have custom allocators we can also provide toString
 methods templated on allocators.
 
Custom allocators released together with HL3?
A lot of things are blocked by custom allocators and this is indeed an issue. It wouldn't make much sense to redefine toString without custom allocators though. BTW: I guess the Duke Nukem Forever joke doesn't work anyore ;-)
Jan 06 2013
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Jan-2013 14:37, Benjamin Thaut пишет:
 Am 05.01.2013 05:24, schrieb ixid:
 On Monday, 31 December 2012 at 14:40:48 UTC, Benjamin Thaut wrote:
 Am 31.12.2012 15:02, schrieb DypthroposTheImposter:
   Do you find that D without GC is more effective than C++? Seems like
 you would be stuck using structs which seems somewhat limiting, even
 compared to C++...

  UE4 has similar reloading capabilities(using DLLs), though they use
 C++ and rely more on the ability to serialize everything
Why should I be stuck with structs? Its the exact same as in C++. I did build my own new and delete operators (as templates). It's not that hard and works pretty well, only the syntax is not ideal in some cases. I can use everything you can use with D+GC. Sometimes you have to be carefull with implict allocations (e.g. closures, array literals) but I have a leak detector that points me to these directly and usually its easy to free these manually. And I'm quite a bit more productive then in C++. Module constructors with a defined order instead of random static initalizers, code generation isnstead of huge amounts of boilerplate code and many other features are the cause of this. Kind Regards Benjamin Thaut
Is D moving away from your sort of use? Games and bioinformatics would seem to be the areas the language should be trying to get people to start using it in. The features you're using seem very much like they should be a part and mode of using the language.
I wouldn't say its moving away from it. Some recent changes to druntime have made it significantly less leaking. But on the other hand a API design like toString() which pretty much does leak in almost all cases don't exactly help a GC free D. In Summary it feels to me that GC free D is not important to the community or the active contributors.
Regarding toString there is a better signature that avoids useless allocations: void toString(scope void delegate(const (char)[]) sink); It takes a delegate to output string representation directly to the destination via 'sink' delegate (that may e.g. write chars to file). Plus the 'scope' part of declaration avoids allocating the said delegate on the heap. I'd even say that string toString(); is an artifact of the past, instead to!string should be used (if allocating a string is fine).
 I also see D's biggest chances in becoming popular in the performance
 critical fields of programming. Which would be systems programming,
 Gaming and others. For programming fields in which a GC is applicable
 people tend to use languages like C# or Java, because they are truly
 safe (see the recent "ref is unsafe" discussion in the newsgroup), they
 have nice productivity features like runtime code changing through the
 VM and have a way better GC because the language was designed from the
 beginning to support a advanced GC.

 Kind Regards
 Benjamin Thaut
-- Dmitry Olshansky
Jan 06 2013
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com>wrote:

 Regarding toString there is a better signature that avoids useless
 allocations:

 void toString(scope void delegate(const (char)[]) sink);

 It takes a delegate to output string representation directly to the
 destination via 'sink' delegate (that may e.g. write chars to file). Plus
 the 'scope' part of declaration avoids allocating the said delegate on the
 heap.

 I'd even say that string toString(); is an artifact of the past, instead
 to!string should be used (if allocating a string is fine).
I know you explained that already, but see, that again slipped my mind. This toString(sink) thingy seems cool, but is there a documentation on it somewhere? Without a related doc, I fear no one will know this exists.
Jan 06 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Jan-2013 23:55, Philippe Sigaud пишет:
 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.
Guess I need to find the original on toString and bit-blit it over with this new sink thingy ;) -- Dmitry Olshansky
Jan 06 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
07-Jan-2013 00:02, Dmitry Olshansky пишет:
 06-Jan-2013 23:55, Philippe Sigaud пишет:
 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.
Guess I need to find the original on toString and bit-blit it over with this new sink thingy ;)
There is a short note in object.d: ... string toString(); Convert Object to a human readable string. ... That is any class may override it. However the real relevant golden bit of information is buried after a row of formatValue templates (that hardly makes any sense to newcomers) at the bottom of std.format page: (http://dlang.org/phobos/std_format.html) Aggregates (struct, union, class, and interface) are basically formatted by calling toString. toString should have one of the following signatures: const void toString(scope void delegate(const(char)[]) sink, FormatSpec fmt); const void toString(scope void delegate(const(char)[]) sink, string fmt); const void toString(scope void delegate(const(char)[]) sink); const string toString(); So it's quite flexible. Even better then I've been advertising it. Note that classes are covered. I bet it can call virtual functions through base reference if base defines the new variation of toString. The problem is if it's going to be extended to wchar/dchar... At least wchar is useful on Windows. But the real fun is that I fail to find any real description of toString in the docs for d-p-l website! The only files in d-p-l site where it is found in are: - errors.dd: ... class Error has a pure virtual a function called toString() which produces a char[] with a human readable description of the error.) Nothing useful and it's out of date! It returns a string (immutable) these days. - lazy-evaluation.dd uses as an example of expensive computation and has the form of toString(i) where i is an int (wtf?!). Probably horribly out of date as it should use either format or to!string. - tempalte.dd uses std.string.toString as an example (same as in lazy?) and it's again an out of date piece of crap. And a couple more files call it here and there (e.g. windows.dd to display exception message box). -- Dmitry Olshansky
Jan 06 2013
prev sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 20:55, schrieb Philippe Sigaud:
 On Sun, Jan 6, 2013 at 6:49 PM, Dmitry Olshansky <dmitry.olsh gmail.com
 <mailto:dmitry.olsh gmail.com>> wrote:

     Regarding toString there is a better signature that avoids useless
     allocations:

     void toString(scope void delegate(const (char)[]) sink);

     It takes a delegate to output string representation directly to the
     destination via 'sink' delegate (that may e.g. write chars to file).
     Plus the 'scope' part of declaration avoids allocating the said
     delegate on the heap.

     I'd even say that string toString(); is an artifact of the past,
     instead to!string should be used (if allocating a string is fine).


 I know you explained that already, but see, that again slipped my mind.
 This toString(sink) thingy seems cool, but is there a documentation on
 it somewhere? Without a related doc, I fear no one will know this exists.
There was a language change request for it in the wiki, but I can't find it anymore. Kind Regards Benjamin Thaut
Jan 06 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-12-30 13:32, Benjamin Thaut wrote:
 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.

 http://3d.benjamin-thaut.de/?p=25
This looks very cool. Question, are you manually triggering the code for generating the RTTI? BTW, have you seen this old project implementing runtime reflection: http://flectioned.kuehne.cn/ -- /Jacob Carlborg
Jan 05 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 05.01.2013 12:15, schrieb Jacob Carlborg:
 On 2012-12-30 13:32, Benjamin Thaut wrote:
 An article about runtime code reloading in the context of game
 developement. A topic I'm currently working on in my spare time. I hope
 it holds some valuable information for everyone working with D.

 http://3d.benjamin-thaut.de/?p=25
This looks very cool. Question, are you manually triggering the code for generating the RTTI? BTW, have you seen this old project implementing runtime reflection: http://flectioned.kuehne.cn/
Yes I know flectioned. But I did not require that much RTTI information. No I'M not manually triggering the code for generating the RTTI. As mentioned in the article it is done via the RTInfo template inside object_.d / object.di which is automatically instanciated for each type used during compilation. Kind Regards Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 11:30, Benjamin Thaut wrote:

 Yes I know flectioned. But I did not require that much RTTI information.
 No I'M not manually triggering the code for generating the RTTI. As
 mentioned in the article it is done via the RTInfo template inside
 object_.d / object.di which is automatically instanciated for each type
 used during compilation.
Ah, so you modified the existing RTInfo? I didn't know that existed. -- /Jacob Carlborg
Jan 06 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 12:17, schrieb Jacob Carlborg:
 On 2013-01-06 11:30, Benjamin Thaut wrote:

 Yes I know flectioned. But I did not require that much RTTI information.
 No I'M not manually triggering the code for generating the RTTI. As
 mentioned in the article it is done via the RTInfo template inside
 object_.d / object.di which is automatically instanciated for each type
 used during compilation.
Ah, so you modified the existing RTInfo? I didn't know that existed.
It does not exist. The current RTInfo template just outputs a null pointer for every type. It is planned that the RTInfo template will be used for a percise GC in the future. (Did you read the article? ^^) Kind Regards Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 12:28, Benjamin Thaut wrote:

 It does not exist. The current RTInfo template just outputs a null
 pointer for every type.
Yes, but it still exists.
 It is planned that the RTInfo template will be
 used for a percise GC in the future. (Did you read the article? ^^)
Yes, I've read the article. How do you think I otherwise could have come up with these questions :) I can see now that you wrote: "This template is instanciated for every type that is used during compilation and thus is ideal to generate RTTI information" I get it now. -- /Jacob Carlborg
Jan 06 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 12:50, schrieb Jacob Carlborg:
 On 2013-01-06 12:28, Benjamin Thaut wrote:

 It does not exist. The current RTInfo template just outputs a null
 pointer for every type.
Yes, but it still exists.
 It is planned that the RTInfo template will be
 used for a percise GC in the future. (Did you read the article? ^^)
Yes, I've read the article. How do you think I otherwise could have come up with these questions :) I can see now that you wrote: "This template is instanciated for every type that is used during compilation and thus is ideal to generate RTTI information" I get it now.
Here is the full implementation if you are interested: https://github.com/Ingrater/druntime/blob/master/src/rtti.d Kind Regards Benjamin Thaut
Jan 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-06 13:19, Benjamin Thaut wrote:

 Here is the full implementation if you are interested:
 https://github.com/Ingrater/druntime/blob/master/src/rtti.d
Thanks. Is that possible to do without having to modify the runtime? -- /Jacob Carlborg
Jan 06 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.01.2013 13:23, schrieb Jacob Carlborg:
 On 2013-01-06 13:19, Benjamin Thaut wrote:

 Here is the full implementation if you are interested:
 https://github.com/Ingrater/druntime/blob/master/src/rtti.d
Thanks. Is that possible to do without having to modify the runtime?
No. The template has to be known inside of object_.d/object.di so it is not possible without modifing the runtime. Also you usually want to have RTTI info for druntime/phobos too.
Jan 06 2013