www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - perpetual: new module for mapping file into memory

reply "Sergei Degtiarev" <sdegtiarev yahoo.com> writes:
I suggests a new module for mapping  files into memory. Instead 
of viewing the file as array of bytes, it directly maps the 
object onto file allowing to handle it as regular in-memory 
object. The data remain persistent after the program exits and 
may be re-opened and used again or shared between processes.
Dynamic arrays are also supported with some limitations.
The code is at
https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual
Is anybody interested?
Aug 17 2015
next sibling parent reply "Liam McSherry" <mcsherry.liam gmail.com> writes:
On Monday, 17 August 2015 at 18:27:48 UTC, Sergei Degtiarev wrote:

 [...]


There is already this in Phobos: 
http://dlang.org/phobos/std_mmfile.html

Is there any difference between `std.mmfile` and your solution?
Aug 17 2015
parent "Sergei Degtiarev" <sdegtiarev yahoo.com> writes:
On Monday, 17 August 2015 at 19:00:37 UTC, Liam McSherry wrote:

 Is there any difference between `std.mmfile` and your solution?


Implementation at system level is similar, the approach is 
different. std.mmfile returns uninitialized chunk of memory, 
std.perpetual returns an object which can be viewed and 
manipulated like regular object of the underlying type. This is 
kind of OOP approach to shared memory.
Aug 17 2015
prev sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 18/08/2015 6:27 a.m., Sergei Degtiarev wrote:

 I suggests a new module for mapping  files into memory. Instead of
 viewing the file as array of bytes, it directly maps the object onto
 file allowing to handle it as regular in-memory object. The data remain
 persistent after the program exits and may be re-opened and used again
 or shared between processes.
 Dynamic arrays are also supported with some limitations.
 The code is at
 https://github.com/sdegtiarev/phobos/tree/master/std/experimental/perpetual
 Is anybody interested?


What I like about this approach is:
- No memory allocation, yes yuppie!
- Uses what ever your type is you want!
- The type you want is initialized, yuppie

What I don't like:
- Does not allow for calling the constructor on the type should it have one
- Does not map to std.mmfile in any way
- No way to say an offset in the file to start working from (also maybe 
an upper limit for arrays)
- You implement basically everything in std.mmfile all over again

Fix up the things above and maybe with a bit more cleaning up it can go 
into std.mmfile. It's basically a poor mans (de/)serializer which is 
quite a nice thing to have.
Aug 17 2015
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 18-Aug-2015 07:45, Rikki Cattermole wrote:

 What I don't like:
 - Does not allow for calling the constructor on the type should it have one


Hm it depends - one thing is that after mmaping a file the type may 
already be constructed, if it wasn't then emplacement should be the 
right thing to do. That is there should be a default/string value.


 - Does not map to std.mmfile in any way


Which is a good thing. std.mmfile is crappy 8 year old ported crap.


 - No way to say an offset in the file to


That may be useful.


 - You implement basically everything in std.mmfile all over again


Low-level API might be in order but it's not std.mmfile which is not 
low-level enough. Separating out a set of portable mmaping primitives 
would be awesome.

What I don't like is perpetual being a class - too bad as it won't honor 
scoping and destructor may or may not be called. Just make it a struct.

Also type should be Perputal, then perpetual is helper function that 
creates it then by means of IFTI type may be deduced. I suggest the 
following API:

// all parameters after file are either T itself
auto p = perpetual("file-name", 42);// would deduce T=int

// or args to T:
struct RGB{ int r,g,b; }
// if not in file use RGB(1,2,3)
auto p = perpetual!RGB("file-name", 1, 2, 3);

-- 
Dmitry Olshansky
Aug 17 2015
parent reply "Sergei Degtiarev" <sdegtiarev yahoo.com> writes:
On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky 
wrote:

 What I don't like is perpetual being a class - too bad as it 
 won't honor scoping and destructor may or may not be called. 
 Just make it a struct.


Why do you think the destructor may not be called? It is, and it 
syncs memory and unmaps the file. Making perpetual a struct would 
require implementing reference counting or disabling copying, and 
all such staff. In my opinion, perpetual should definitely have 
reference semantics, this is exactly a point where D shines.



 Also type should be Perputal, then perpetual is helper function 
 that creates it then by means of IFTI type may be deduced. I 
 suggest the following API:

 // all parameters after file are either T itself
 auto p = perpetual("file-name", 42);// would deduce T=int

 // or args to T:
 struct RGB{ int r,g,b; }
 // if not in file use RGB(1,2,3)
 auto p = perpetual!RGB("file-name", 1, 2, 3);


Right, just forgot. Will do, thanks.
Aug 18 2015
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 18-Aug-2015 17:32, Sergei Degtiarev wrote:

 On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky wrote:

 What I don't like is perpetual being a class - too bad as it won't
 honor scoping and destructor may or may not be called. Just make it a
 struct.


 Why do you think the destructor may not be called? It is, and it syncs
 memory and unmaps the file. Making perpetual a struct would require
 implementing reference counting or disabling copying, and all such
 staff. In my opinion, perpetual should definitely have reference
 semantics, this is exactly a point where D shines.


Class is allocated on GC heap dtor is called on collection in that sense 
it is a finalizer. Just use struct if you don't need inheritance and/or 
do resource management.


-- 
Dmitry Olshansky
Aug 18 2015
prev sibling parent "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Tuesday, 18 August 2015 at 14:32:18 UTC, Sergei Degtiarev 
wrote:

 On Tuesday, 18 August 2015 at 06:52:47 UTC, Dmitry Olshansky 
 wrote:

 What I don't like is perpetual being a class - too bad as it 
 won't honor scoping and destructor may or may not be called. 
 Just make it a struct.


 Why do you think the destructor may not be called? It is, and 
 it syncs memory and unmaps the file. Making perpetual a struct 
 would require implementing reference counting or disabling 
 copying, and all such staff. In my opinion, perpetual should 
 definitely have reference semantics, this is exactly a point 
 where D shines.


It might be run, but there is no guarantee a class destructor 
will ever be called.  A struct destructor will be when it goes 
out of scope.  Hence Scoped!

http://forum.dlang.org/thread/ikuomwcxhydluojkhjqy forum.dlang.org
Aug 19 2015
prev sibling parent reply "Sergei Degtiarev" <sdegtiarev yahoo.com> writes:
On Tuesday, 18 August 2015 at 04:46:10 UTC, Rikki Cattermole 
wrote:

 - Does not allow for calling the constructor on the type should 
 it have one


I plan to implement arguments forwarding to constructor, just 
decided that the concept would be more clear with minimal code. 
Technically, the object is initialized with .init and is in valid 
state.


 - Does not map to std.mmfile in any way
 - You implement basically everything in std.mmfile all over 
 again


Does not need to, there are only two system calls involved and it 
makes no sense to reuse such amount of code.


 - No way to say an offset in the file to start working from 
 (also maybe an upper limit for arrays)


This might be useful in some cases, but in general from my 
experience (I used this utility for several years in C++), it 
downgrades the interface, makes you regarding the data as set of 
bytes rather than an object. It is always possible to wrap up 
everything in a struct, so that working portion is at right 
offset.


 it can go into std.mmfile.


The concept is very different from plain shared memory, 
std.mmfile can be improved with a few lines of code, but it 
wouldn't change the way it's used.


 It's basically a poor mans (de/)serializer.


Not at all, it allows fast access to the data, so it might be 
used as effectively as regular values, and insurance from data 
loss at crash or stop.
Aug 18 2015
parent "Sergei Degtiarev" <sdegtiarev yahoo.com> writes:
On Tuesday, 18 August 2015 at 14:16:30 UTC, Sergei Degtiarev 
wrote:

 Right, just forgot. Will do, thanks.


I added helper functions with argument forwarding for 
initialization.
Aug 18 2015