www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Binary data-structure serialization

reply Robert <robert.muench robertmuench.de> writes:
Hi, are there are any projectes / snippets / prototypes where someone 
tried to serializa D run-time data-structures (like an associative 
array) into a file?

--
Robert M. Münch
http://www.robertmuench.de
May 30 2010
next sibling parent reply Robert <robert.muench robertmuench.de> writes:
On 2010-05-30 12:38:10 +0200, Robert <robert.muench robertmuench.de> said:

 Hi, are there are any projectes / snippets / prototypes where someone 
 tried to serializa D run-time data-structures (like an associative 
 array) into a file?

Thinking further about this: How about a way to change the underlying memory manager from RAM to a MMF? If I could specify which memory allocator to use when creating a class, The class could be created in a MMF (which is mapped to main memory and hence there should be no difference in speed) and all references would go to the MMF as well. With such a low-level switch persistent data-structures would be for free. The memory allocator could be optimized to yield minimal MMF. Any comments? -- Robert M. Münch http://www.robertmuench.de
May 30 2010
next sibling parent reply Trass3r <un known.com> writes:
 If I could specify which memory allocator to use when creating a class,  
 The class could be created in a MMF (which is mapped to main memory and  
 hence there should be no difference in speed) and all references would  
 go to the MMF as well.

Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you want?
May 31 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-05-31 17:01:16 +0200, Trass3r said:

 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you want?

Ah, that's cool. Didn't saw it. This looks very good. Will those clases be GCed? -- Robert M. Münch www.robertmuench.de
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 06:41 PM, Masahiro Nakagawa wrote:
 On Wed, 02 Jun 2010 06:29:12 +0900, Robert M. Münch
 <robert.muench robertmuench.de> wrote:

 On 2010-05-31 17:01:16 +0200, Trass3r said:

 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you
 want?

Ah, that's cool. Didn't saw it. This looks very good. Will those clases be GCed?

Default no. Documentation is here: http://digitalmars.com/d/2.0/memory.html#newdelete NOTE: This info is a bit outdated. std.gc was moved to core.memory and std.outofmemory was moved to onOutOfMemoryError in core.exception.

Sorry, class allocators are gone. Andrei
Jun 01 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 Sorry, class allocators are gone.

I must have missed the memo :o) Can you explain a bit? Bye, bearophile
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 07:35 PM, bearophile wrote:
 Andrei Alexandrescu:
 Sorry, class allocators are gone.

I must have missed the memo :o) Can you explain a bit?

It was an ill-defined misfeature that didn't deserve to stay in the language. Walter and I agreed to let it go, but somehow forgot to announce it. Andrei
Jun 01 2010
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 06/01/2010 08:17 PM, Andrei Alexandrescu wrote:
 On 06/01/2010 07:35 PM, bearophile wrote:
 Andrei Alexandrescu:
 Sorry, class allocators are gone.

I must have missed the memo :o) Can you explain a bit?

It was an ill-defined misfeature that didn't deserve to stay in the language. Walter and I agreed to let it go, but somehow forgot to announce it. Andrei

replacement?
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 08:29 PM, Ellery Newcomer wrote:
 On 06/01/2010 08:17 PM, Andrei Alexandrescu wrote:
 On 06/01/2010 07:35 PM, bearophile wrote:
 Andrei Alexandrescu:
 Sorry, class allocators are gone.

I must have missed the memo :o) Can you explain a bit?

It was an ill-defined misfeature that didn't deserve to stay in the language. Walter and I agreed to let it go, but somehow forgot to announce it. Andrei

replacement?

TightArray. Andrei
Jun 01 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 03:58:57 +0200, Andrei Alexandrescu said:

 TightArray.

What's that? Any link / reference, I haven't found it in the docs. I was quite happy to see a class allocator because it gives flexibility in how & where class objects are allocated. That would have matched my requirements perfectly. I need to somehow get control of the allocation procedure but would like to let the GC take care of the returned memory. -- Robert M. Münch www.robertmuench.de
Jun 02 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Lars T. Kyllingstad:
 In particular, see
 http://erdani.com/d/phobos/std_container.html#TightArray

The TightArray seems a reference-counted array of items, where items can be references of objects allocated on the heap. This doesn't solve the problem of avoiding heap allocations when you want to use objects. How can you allocate objects themselves in an arena, a stack, or a pool? Bye, bearophile
Jun 02 2010
prev sibling parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 11:37:18 +0200, Lars T. Kyllingstad said:

 It's part of his proposal for a container library, which he has presented
 in some other threads:
 
 http://erdani.com/d/phobos/std_container.html
 http://erdani.com/d/phobos/container.d
 
 In particular, see
 
 http://erdani.com/d/phobos/std_container.html#TightArray

Thanks for the link. Is this stuff a phobos fork? An extension to phobos? Is this related to D2? I'm a bit confused how to relate this work to the runtime lib. -- Robert M. Münch www.robertmuench.de
Jun 02 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Robert M. Münch:
 Is this stuff a phobos fork? An extension to phobos? Is this related to 
 D2? I'm a bit confused how to relate this work to the runtime lib.

It's just a module that will be added to Phobos. Bye, bearophile
Jun 02 2010
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to 
 announce it.

Are stack-allocated (scoped) classes too gone? Bye, bearophile
Jun 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes. Andrei
Jun 01 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 De facto no, de jure yes.

Walter is listening you (see comment 6): http://d.puremagic.com/issues/show_bug.cgi?id=2070 I have used a good number of "scope" in classes for my small programs that I am slowly porting from D1 to D2. A higher level language, that omits manual low-level optimizations requires a more powerful compiler, able to optimize away some abstractions by itself. So I hope D will soon use a good escape analysis to stack-allocate classes (and dynamic arrays), as recent versions of HotSpot do (but even with an escape analysis, the objects inside objects get allocated elsewhere anyway. Maybe this requires another kind of escape analysis?). (I have said "use" because I think DMD is already able to perform some escape analysis). http://java.sun.com/javase/6/webnotes/6u14.html
Optimization Using Escape Analysis: The -XX:+DoEscapeAnalysis option directs
HotSpot to look for objects that are created and referenced by a single thread
within the scope of a method compilation. Allocation is omitted for such
non-escaping objects, and their fields are treated as local variables, often
residing in machine registers. Synchronization on non-escaping objects is also
elided.<

EA, from page 52: http://webdocs.cs.ualberta.ca/~amaral/IBM-Stoodley-talks/UofAKASWorkshop.pdf To focus it's better to start with a very simple example. So at first dmd can remove the heap allocation in this basic case: import std.stdio: writeln; final class Rectangle { int x, y; this(int xx, int yy) { this.x = xx; this.y = yy; } int area() { return this.x * this.y; } } void main () { enum N = 100; ulong tot = 0; foreach (x; 0 .. N) foreach (y; 0 .. N) { Rectangle rectangle = new Rectangle(x, y); tot += rectangle.area(); } writeln(tot); } So the optimization phase can turn that into something equivalent to: import std.stdio: writeln; void main () { enum N = 100; ulong tot = 0; foreach (x; 0 .. N) foreach (y; 0 .. N) tot += x * y; writeln(tot); } That was a naive/toy example, but you can't start from very complex things. Later more complex cases can be added, like dynamic arrays, synchronized methods, etc. Bye, bearophile
Jun 02 2010
prev sibling next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 De facto no, de jure yes.

Maybe the "Typesafe Variadic Functions For class objects" can be removed from D2. Bye, bearophile
Jun 02 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it. Andrei
Jun 02 2010
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 I am implementing stack allocation for classes in library code;
 I'm just undecided on where to put it.

A module that contains code to manage memory, create memory pools, allocate a whole 2D dynamic array out of a single chunck of memory, and so on. Bye, bearophile
Jun 02 2010
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it.

I think it belongs in druntime, since it's an allocation feature. How easy is it to use?

The signature for non-class types is: T* emplace(T, Args...)(void * address, Args args); The function creates an object of type T at location address and initializes it with args. Returns a pointer to the just-created value. I needed this function for TightArray. I haven't yet defined the signature for class types, but probably it would look like this: T emplace(T, Args...)(void * address, Args args); To obtain stack allocation, you may therefore write something like: auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff); Although it uses the obscure classInstanceSize, I think it's near the desired balance between terseness and making the programmer sweat for a dangerous achievement. Andrei
Jun 02 2010
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 02:49 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it.

I think it belongs in druntime, since it's an allocation feature. How easy is it to use?

The signature for non-class types is: T* emplace(T, Args...)(void * address, Args args); The function creates an object of type T at location address and initializes it with args. Returns a pointer to the just-created value. I needed this function for TightArray. I haven't yet defined the signature for class types, but probably it would look like this: T emplace(T, Args...)(void * address, Args args); To obtain stack allocation, you may therefore write something like: auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff); Although it uses the obscure classInstanceSize, I think it's near the desired balance between terseness and making the programmer sweat for a dangerous achievement.

This makes it better than scope classes for one reason -- it's an expression instead of a declaration. That means I can use it to create a temporary. But I have some qualms here -- when is the object's destructor called? I know it probably cannot be done any other way unless we had something like macros, but it's not easy for humans to parse this mess. Sure, it should be somewhat difficult for the developer to write, but what about the reader? I suppose there just isn't an easy way to do this. Also, wouldn't it have to be something like this: auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff); And finally, is there any possible way to write an emplace that uses an allocator alias instead of the actual memory location? i.e. T emplace2(T, alias alloc, Args...)(Args args) { return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args); } Anything to avoid the double-specification of T would be nice. This of course would be impossible to use with alloca without macros. -Steve

Good points. To ensure the destructor is called, I guess we need this: system struct StackAllocated(T) { private ubyte buffer[__traits(classInstanceSize, T)] = void; property T payload() { return cast(T) buffer.ptr; } alias payload this; disable this(); // meh disable this(this); this(Args...)(auto ref Args args) { emplace!T(buffer.ptr, buffer.length); } ~this() { clear(payload); } } Andrei
Jun 02 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 To ensure the destructor is called, I guess we need this:
 
  system struct StackAllocated(T)
 {
      private ubyte buffer[__traits(classInstanceSize, T)] = void;
       property T payload() { return cast(T) buffer.ptr; }
      alias payload this;
       disable this(); // meh
       disable this(this);
      this(Args...)(auto ref Args args)
      {
          emplace!T(buffer.ptr, buffer.length);
      }
      ~this()
      {
          clear(payload);
      }
 }

Is this useful? static if (__traits(hasMember, T, "__dtor")) { ~this() { clear(payload); } } Bye, bearophile
Jun 02 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2010 03:34 PM, bearophile wrote:
 Andrei Alexandrescu:
 To ensure the destructor is called, I guess we need this:

  system struct StackAllocated(T)
 {
       private ubyte buffer[__traits(classInstanceSize, T)] = void;
        property T payload() { return cast(T) buffer.ptr; }
       alias payload this;
        disable this(); // meh
        disable this(this);
       this(Args...)(auto ref Args args)
       {
           emplace!T(buffer.ptr, buffer.length);
       }
       ~this()
       {
           clear(payload);
       }
 }

Is this useful? static if (__traits(hasMember, T, "__dtor")) { ~this() { clear(payload); } } Bye, bearophile

Absolutely! Andrei
Jun 02 2010
prev sibling parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-02 21:34:32 +0200, Andrei Alexandrescu said:

 The signature for non-class types is:
 
 T* emplace(T, Args...)(void * address, Args args);
 
 The function creates an object of type T at location address and 
 initializes it with args. Returns a pointer to the just-created value. 
 I needed this function for TightArray.

Hi, does this already work? This would make it possible to allocate T within a memory-mapped file. I would like to give it a try. Not sure how to deal with re-allocation problem as pointers within the MMF need to be MMF relative and not absolut. IMO via operator overloading this should be possible to handle in the background. -- Robert M. Münch www.robertmuench.de
Jun 03 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/03/2010 02:31 AM, Robert M. Münch wrote:
 On 2010-06-02 21:34:32 +0200, Andrei Alexandrescu said:

 The signature for non-class types is:

 T* emplace(T, Args...)(void * address, Args args);

 The function creates an object of type T at location address and
 initializes it with args. Returns a pointer to the just-created value.
 I needed this function for TightArray.

Hi, does this already work? This would make it possible to allocate T within a memory-mapped file. I would like to give it a try.

I didn't have much time to work on it lately, but I'll check it in soon.
 Not sure how to deal with re-allocation problem as pointers within the
 MMF need to be MMF relative and not absolut. IMO via operator
 overloading this should be possible to handle in the background.

Once created, a file mapping is at a constant address. If you remap a file (an unsafe operation), you can memcpy the data around. All D objects are assumed to be relocatable. Andrei
Jun 03 2010
parent Robert <robert.muench robertmuench.de> writes:
On 2010-06-03 15:45:11 +0200, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Once created, a file mapping is at a constant address.

Hi, sorry but don't get this. This is true for one session. But the next time you run the prog the mapping can be on an other address.
 If you remap a file (an unsafe operation),

What do you mean by "remaping a file"?
  you can memcpy the data around. All D objects are assumed to be relocatable.

Still don't get it. If a MMF is mapped into VM at 0x1000 and you create objects and pointers, the pointers within the MMF need to get 0x1000 substracted. Only in this case the pointer within the MMF references an address = the index within the file. -- Robert M. Münch http://www.robertmuench.de
Jun 03 2010
prev sibling next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Wed, 02 Jun 2010 06:29:12 +0900, Robert M. Münch
<robert.muench robertmuench.de> wrote:

 On 2010-05-31 17:01:16 +0200, Trass3r said:

 Isn't http://digitalmars.com/d/2.0/class.html#ClassAllocator what you  
 want?

Ah, that's cool. Didn't saw it. This looks very good. Will those clases be GCed?

Default no. Documentation is here: http://digitalmars.com/d/2.0/memory.html#newdelete NOTE: This info is a bit outdated. std.gc was moved to core.memory and std.outofmemory was moved to onOutOfMemoryError in core.exception.
Jun 01 2010
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 02 Jun 2010 10:47:37 +0200, Robert M. Münch wrote:

 On 2010-06-02 03:58:57 +0200, Andrei Alexandrescu said:
 
 TightArray.

What's that? Any link / reference, I haven't found it in the docs.

It's part of his proposal for a container library, which he has presented in some other threads: http://erdani.com/d/phobos/std_container.html http://erdani.com/d/phobos/container.d In particular, see http://erdani.com/d/phobos/std_container.html#TightArray -Lars
Jun 02 2010
prev sibling next sibling parent "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
On Wed, 02 Jun 2010 15:15:25 +0200, Robert M. Münch wrote:

 On 2010-06-02 11:37:18 +0200, Lars T. Kyllingstad said:
 
 It's part of his proposal for a container library, which he has
 presented in some other threads:
 
 http://erdani.com/d/phobos/std_container.html
 http://erdani.com/d/phobos/container.d
 
 In particular, see
 
 http://erdani.com/d/phobos/std_container.html#TightArray

Thanks for the link. Is this stuff a phobos fork? An extension to phobos? Is this related to D2? I'm a bit confused how to relate this work to the runtime lib.

It's a work in progress, supposed to be included in Phobos for D2 at some point. It was discussed in these threads: http://www.digitalmars.com/d/archives/digitalmars/D/ container_stuff_110423.html http://www.digitalmars.com/d/archives/digitalmars/D/ std.container_update_110509.html http://www.digitalmars.com/d/archives/digitalmars/D/std.container_update_- _now_Array_is_in_110574.html -Lars
Jun 02 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option. -Steve
Jun 02 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it.

I think it belongs in druntime, since it's an allocation feature. How easy is it to use? -Steve
Jun 02 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it.

I think it belongs in druntime, since it's an allocation feature. How easy is it to use?

The signature for non-class types is: T* emplace(T, Args...)(void * address, Args args); The function creates an object of type T at location address and initializes it with args. Returns a pointer to the just-created value. I needed this function for TightArray. I haven't yet defined the signature for class types, but probably it would look like this: T emplace(T, Args...)(void * address, Args args); To obtain stack allocation, you may therefore write something like: auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff); Although it uses the obscure classInstanceSize, I think it's near the desired balance between terseness and making the programmer sweat for a dangerous achievement.

This makes it better than scope classes for one reason -- it's an expression instead of a declaration. That means I can use it to create a temporary. But I have some qualms here -- when is the object's destructor called? I know it probably cannot be done any other way unless we had something like macros, but it's not easy for humans to parse this mess. Sure, it should be somewhat difficult for the developer to write, but what about the reader? I suppose there just isn't an easy way to do this. Also, wouldn't it have to be something like this: auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff); And finally, is there any possible way to write an emplace that uses an allocator alias instead of the actual memory location? i.e. T emplace2(T, alias alloc, Args...)(Args args) { return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args); } Anything to avoid the double-specification of T would be nice. This of course would be impossible to use with alloca without macros. -Steve
Jun 02 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:57:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


       disable this(); // meh

Is this wishful thinking or going to be a reality? ;) -Steve
Jun 02 2010
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 02 Jun 2010 15:57:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:49 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:34:32 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:27 PM, Steven Schveighoffer wrote:
 On Wed, 02 Jun 2010 15:16:53 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/02/2010 02:04 PM, Steven Schveighoffer wrote:
 On Tue, 01 Jun 2010 21:59:20 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 06/01/2010 08:53 PM, bearophile wrote:
 Andrei Alexandrescu Wrote:
 Walter and I agreed to let it go, but somehow forgot to
 announce it.

Are stack-allocated (scoped) classes too gone?

De facto no, de jure yes.

I don't really understand this. Are they going to be gone or not? In any case, can we at least keep them for unsafe D? Stack allocation is not easy to do via placement new, the compiler is much better suited for it, and stack allocating classes can be a huge performance gain. Until we get full escape analysis, scope classes are the only option.

There's no need for a language feature. I am implementing stack allocation for classes in library code; I'm just undecided on where to put it.

I think it belongs in druntime, since it's an allocation feature. How easy is it to use?

The signature for non-class types is: T* emplace(T, Args...)(void * address, Args args); The function creates an object of type T at location address and initializes it with args. Returns a pointer to the just-created value. I needed this function for TightArray. I haven't yet defined the signature for class types, but probably it would look like this: T emplace(T, Args...)(void * address, Args args); To obtain stack allocation, you may therefore write something like: auto obj = emplace(alloca(_traits(classInstanceSize, T)), stuff); Although it uses the obscure classInstanceSize, I think it's near the desired balance between terseness and making the programmer sweat for a dangerous achievement.

This makes it better than scope classes for one reason -- it's an expression instead of a declaration. That means I can use it to create a temporary. But I have some qualms here -- when is the object's destructor called? I know it probably cannot be done any other way unless we had something like macros, but it's not easy for humans to parse this mess. Sure, it should be somewhat difficult for the developer to write, but what about the reader? I suppose there just isn't an easy way to do this. Also, wouldn't it have to be something like this: auto obj = emplace!T(alloca(_traits(classInstanceSize, T)), stuff); And finally, is there any possible way to write an emplace that uses an allocator alias instead of the actual memory location? i.e. T emplace2(T, alias alloc, Args...)(Args args) { return emplace!(T, Args)(alloc(_traits(classInstanceSize, T)), args); } Anything to avoid the double-specification of T would be nice. This of course would be impossible to use with alloca without macros. -Steve

Good points. To ensure the destructor is called, I guess we need this: system struct StackAllocated(T) { private ubyte buffer[__traits(classInstanceSize, T)] = void; property T payload() { return cast(T) buffer.ptr; } alias payload this; disable this(); // meh disable this(this); this(Args...)(auto ref Args args) { emplace!T(buffer.ptr, buffer.length); } ~this() { clear(payload); } }

Hm... is this valid usage then? foo(StackAllocated!MyClass(a, b, c)); :) -Steve
Jun 02 2010
prev sibling next sibling parent reply "Masahiro Nakagawa" <repeatedly gmail.com> writes:
Hi Robert,

On Sun, 30 May 2010 19:38:10 +0900, Robert <robert.muench robertmuench.de>  
wrote:

 Hi, are there are any projectes / snippets / prototypes where someone  
 tried to serializa D run-time data-structures (like an associative  
 array) into a file?

I may not be right about your question, but I am writing msgpack4d. http://www.bitbucket.org/repeatedly/msgpack4d This library will be available in Phobos(see d.announce). Now, I fix bugs and improve some features. Hope this helps. Masahiro
May 31 2010
parent Robert <robert.muench robertmuench.de> writes:
On 2010-05-31 11:09:54 +0200, "Masahiro Nakagawa" <repeatedly gmail.com> said:

 I may not be right about your question, but I am writing msgpack4d.
 
 http://www.bitbucket.org/repeatedly/msgpack4d
 
 This library will be available in Phobos(see d.announce).
 Now, I fix bugs and improve some features.

Hi, does it support data-structures like trees, lists etc. or is the focus to transfer plain types? I'm looking for a way, where I can save and load complex data-structures like graphs, b-trees etc. to a file. -- Robert M. Münch http://www.robertmuench.de
May 31 2010
prev sibling next sibling parent reply Eric Poggel <dnewsgroup yage3d.net> writes:
On 5/30/2010 6:38 AM, Robert wrote:
 Hi, are there are any projectes / snippets / prototypes where someone
 tried to serializa D run-time data-structures (like an associative
 array) into a file?

 --
 Robert M. Münch
 http://www.robertmuench.de

After having difficulty getting ddbg to work, I decided to write a dump function so I could easily see my data structures at runtime. The biggest part of it is a json encoder which can handle most D structures. If I recall, it only has trouble with enum's and pointers (since there's no length info). It's available in Yage's source here (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d) Yage is licensed LGPL but I grant permission to use this under the Boost 1.0 license if you or anyone else needs it. However, for now it's only one-way and there's no deserialization part to it.
May 31 2010
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-01 02:13:18 +0200, Eric Poggel said:

 After having difficulty getting ddbg to work, I decided to write a dump 
 function so I could easily see my data structures at runtime.  The 
 biggest part of it is a json encoder which can handle most D 
 structures.   If I recall, it only has trouble with enum's and pointers 
 (since there's no length info).
 
 It's available in Yage's source here 
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d) 
 Yage is licensed LGPL but I grant permission to use this under the 
 Boost 1.0 license if you or anyone else needs it.

Thanks for this. I will take a look.
 However, for now it's only one-way and there's no deserialization part to it.

Well, this part is left for the reader as an exercise ;-). -- Robert M. Münch www.robertmuench.de
Jun 01 2010
parent reply Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/1/2010 5:31 PM, Robert M. Münch wrote:
 On 2010-06-01 02:13:18 +0200, Eric Poggel said:

 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.

Thanks for this. I will take a look.
 However, for now it's only one-way and there's no deserialization part
 to it.

Well, this part is left for the reader as an exercise ;-).

It seems it also doesn't serialize members of parent classes, which can be a major caveat. I took a look at it yesterday but unfortunately wasn't able to figure it out.
Jun 06 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-06-06 21:32, Eric Poggel wrote:
 On 6/1/2010 5:31 PM, Robert M. Münch wrote:
 On 2010-06-01 02:13:18 +0200, Eric Poggel said:

 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.

Thanks for this. I will take a look.
 However, for now it's only one-way and there's no deserialization part
 to it.

Well, this part is left for the reader as an exercise ;-).

It seems it also doesn't serialize members of parent classes, which can be a major caveat. I took a look at it yesterday but unfortunately wasn't able to figure it out.

You can have a look at my serialization library: http://dsource.org/projects/orange/ it could use some testing. * It handles both serializing and deserializing * It automatically serializes the base classes * It supports events (before and after (de)serializing) * It supports non-serialized fields (you can say that some fields in a class should not be serialized) * It's licensed under the Boost license * It's fairly std/runtime library independent (hopefully it's only the XMLArchive that is dependent on the runtime library, I've only tested it with Tango) * You can create new archive types and use them with the existing serializer * It currently only supports XML as the archive type but the library is built so you can create new archive types and use them with the existing serializer * If you want to serialize objects through base class references you need to register a serialize function, everything else should be handled automatically You can have a look at an example of usage at the project site. -- /Jacob Carlborg
Jun 07 2010
parent Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/7/2010 7:37 AM, Jacob Carlborg wrote:
 On 2010-06-06 21:32, Eric Poggel wrote:
 On 6/1/2010 5:31 PM, Robert M. Münch wrote:
 On 2010-06-01 02:13:18 +0200, Eric Poggel said:

 After having difficulty getting ddbg to work, I decided to write a
 dump function so I could easily see my data structures at runtime. The
 biggest part of it is a json encoder which can handle most D
 structures. If I recall, it only has trouble with enum's and pointers
 (since there's no length info).

 It's available in Yage's source here
 (http://dsource.org/projects/yage/browser/trunk/src/yage/core/json.d)
 Yage is licensed LGPL but I grant permission to use this under the
 Boost 1.0 license if you or anyone else needs it.

Thanks for this. I will take a look.
 However, for now it's only one-way and there's no deserialization part
 to it.

Well, this part is left for the reader as an exercise ;-).

It seems it also doesn't serialize members of parent classes, which can be a major caveat. I took a look at it yesterday but unfortunately wasn't able to figure it out.

You can have a look at my serialization library: http://dsource.org/projects/orange/ it could use some testing. * It handles both serializing and deserializing * It automatically serializes the base classes * It supports events (before and after (de)serializing) * It supports non-serialized fields (you can say that some fields in a class should not be serialized) * It's licensed under the Boost license * It's fairly std/runtime library independent (hopefully it's only the XMLArchive that is dependent on the runtime library, I've only tested it with Tango) * You can create new archive types and use them with the existing serializer * It currently only supports XML as the archive type but the library is built so you can create new archive types and use them with the existing serializer * If you want to serialize objects through base class references you need to register a serialize function, everything else should be handled automatically You can have a look at an example of usage at the project site.

Thanks. This looks pretty nice!
Jun 07 2010
prev sibling next sibling parent "Masahiro Nakagawa" <repeatedly gmail.com> writes:
On Mon, 31 May 2010 22:36:13 +0900, Robert <robert.muench robertmuench.de>  
wrote:

 On 2010-05-31 11:09:54 +0200, "Masahiro Nakagawa" <repeatedly gmail.com>  
 said:

 I may not be right about your question, but I am writing msgpack4d.
  http://www.bitbucket.org/repeatedly/msgpack4d
  This library will be available in Phobos(see d.announce).
 Now, I fix bugs and improve some features.

Hi, does it support data-structures like trees, lists etc. or is the focus to transfer plain types? I'm looking for a way, where I can save and load complex data-structures like graphs, b-trees etc. to a file.

You can define mp_pack / mp_unpack if need(Complex data-structure requires specified routines). I added simple b-tree example to example/custom.d. If you want more efficient serialization library for complex data-structures, you will need to create like existing DBMs. Masahiro
Jun 01 2010
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 30 May 2010 06:38:10 -0400, Robert <robert.muench robertmuench.de>  
wrote:

 Hi, are there are any projectes / snippets / prototypes where someone  
 tried to serializa D run-time data-structures (like an associative  
 array) into a file?

 --
 Robert M. Münch
 http://www.robertmuench.de

I've written a JSON serializer/de-serializer which uses compile time type reflection. Doesn't handle cycles or polymorphism yet, but I'd gladly share it if you (or others) are interested.
Jun 03 2010
parent =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench robertmuench.de> writes:
On 2010-06-04 02:27:56 +0200, Robert Jacques said:

 I've written a JSON serializer/de-serializer which uses compile time 
 type reflection. Doesn't handle cycles or polymorphism yet, but I'd 
 gladly share it if you (or others) are interested.

Thanks for the offer. AFAIK JSON is focused for wire-transfer, what I'm looking for is a way to efficiently write such structures into a file in binary form and re-create the run-time structure fast. What I need is directed to a database / transactional system where millions of in/out operations happen. -- Robert M. Münch http://www.robertmuench.de
Jun 04 2010
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 04 Jun 2010 08:43:11 -0400, Robert M. Münch  
<robert.muench robertmuench.de> wrote:

 On 2010-06-04 02:27:56 +0200, Robert Jacques said:

 I've written a JSON serializer/de-serializer which uses compile time  
 type reflection. Doesn't handle cycles or polymorphism yet, but I'd  
 gladly share it if you (or others) are interested.

Thanks for the offer. AFAIK JSON is focused for wire-transfer, what I'm looking for is a way to efficiently write such structures into a file in binary form and re-create the run-time structure fast. What I need is directed to a database / transactional system where millions of in/out operations happen.

Pretty much all serialization methods have some amount of verification overhead, which it sounds like you neither want nor need. I'd recommend directly using .tupleof for structs/classes. Here's some simplified code snippets from my JSON library: // From toJson(T)(T value) static if(is(T == struct) || is(T == class)) { foreach(i,v;value.tupleof) { result[getVariableName!(T,i)] = toJson(v); } } // From fromJson(T)(ref T dst) static if(is(T == struct) || is(T == class)) { static if(__traits(compiles, dst = new T() )) if(dst is null) dst = new T(); static if(is(T == class)) enforce(dst !is null); foreach(i,v;dst.tupleof) { // Warning V is read-only because ref v doesn't compile auto pJson = getVariableName!(T,i) in obj; if(!pJson) continue; static if( is(typeof(v)==ulong) ) { dst.tupleof[i] = roundTo!(long)(pJson.number); // Hack around bug 3418 } else { pJson.fromJson( dst.tupleof[i] ); } } }
Jun 04 2010