www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - D styled data format, Json failed

reply "Saaa" <empty needmail.com> writes:
I looked at the JSON format and it seems very inefficient at loading arrays 
as it isn't limited to one type per array.
This is nice when you want to save a small array with different typed 
elements but for my purposes this is kind of a performance problem.

This is why I will try and get suggestions again about the D-styled format I 
tried to suggest a few threads ago :)

Let me suggest a simple example:
(Please tell me when something isn't obvious :)

---file.dat
//comment
int number = 10;

float [3][2]  simpleArray = [
[0.1, 0.2, 0.3],
[0.4, 0.5, 0.6]];
--

---main.d
static import ddata;
void main()
{
char[][] file = read (`file.dat`);
int i;
char c;

ddata.get(file,'number',i);
ddata.get(file,'number',c); //type fail, thus returns -1 or throws an 
exception

float [3][2] a;

ddata.get(file,'simpleArray',a);
ddata.write(file,'simpleArrayCopy', a);

a[0][0] =3.0;

ddata.write(file,'simpleArray', a);

write('file.dat', file);
}
--

resulting data file:

---file.dat
//comment
int number = 10;

float [3][2]  simpleArray = [
[3.0, 0.2, 0.3],
[0.4, 0.5, 0.6]];

float [3][2]  simpleArrayCopy = [
[0.1, 0.2, 0.3],
[0.4, 0.5, 0.6]];
--
May 03 2009
next sibling parent "nobody" <somebody somewhere.com> writes:
"Saaa" <empty needmail.com> wrote in message 
news:gtlrs3$1b9s$1 digitalmars.com...

 I looked at the JSON format and it seems very inefficient at loading 
 arrays as it isn't limited to one type per array.
 This is nice when you want to save a small array with different typed 
 elements but for my purposes this is kind of a performance problem.

 This is why I will try and get suggestions again about the D-styled format 
 I tried to suggest a few threads ago :)

 Let me suggest a simple example:
 (Please tell me when something isn't obvious :)

 ---file.dat
 //comment
 int number = 10;

 float [3][2]  simpleArray = [
 [0.1, 0.2, 0.3],
 [0.4, 0.5, 0.6]];
 --

 ---main.d
 static import ddata;
 void main()
 {
 char[][] file = read (`file.dat`);
 int i;
 char c;

 ddata.get(file,'number',i);
 ddata.get(file,'number',c); //type fail, thus returns -1 or throws an 
 exception

 float [3][2] a;

 ddata.get(file,'simpleArray',a);
 ddata.write(file,'simpleArrayCopy', a);

 a[0][0] =3.0;

 ddata.write(file,'simpleArray', a);

 write('file.dat', file);
 }
 --

 resulting data file:

 ---file.dat
 //comment
 int number = 10;

 float [3][2]  simpleArray = [
 [3.0, 0.2, 0.3],
 [0.4, 0.5, 0.6]];

 float [3][2]  simpleArrayCopy = [
 [0.1, 0.2, 0.3],
 [0.4, 0.5, 0.6]];
 -- 



I've run into similar problems when I wanted to save lots of stuff to a 
file, and be able to load them again.
Huge multi-dimensional arrays seem to be an issue with JSON and similar..

Such a file writer as you propose would be very handy to have.
May 04 2009
prev sibling next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Saaa wrote:


 
 I looked at the JSON format and it seems very inefficient at loading 




 as it isn't limited to one type per array.
 This is nice when you want to save a small array with different typed 
 elements but for my purposes this is kind of a performance problem.
 
 This is why I will try and get suggestions again about the D-styled format 




 tried to suggest a few threads ago :)
 
 Let me suggest a simple example:
 (Please tell me when something isn't obvious :)
 



I guess you will have to write this one yourself, it will be to D what JSON 
is to javascript ;)

I wonder how much of an performance improvement you will get though when 
loading data at runtime. You still have to parse and check everything since 
it remains a text based format. 

I know of one other solution but that's again not available for D: google 
protobuf: http://code.google.com/p/protobuf/
This uses a text-based format for describing the structure of the data, but 
the actual data can be in an optimized binary format.
May 04 2009
parent "Saaa" <empty needmail.com> writes:
 I guess you will have to write this one yourself, it will be to D what 
 JSON
 is to javascript ;)





 I wonder how much of an performance improvement you will get though when
 loading data at runtime.



we may never know :D


 You still have to parse and check everything since
 it remains a text based format.



mention
otherwise it might be useless for the large arrays I wish to get/write



 I know of one other solution but that's again not available for D: google
 protobuf: http://code.google.com/p/protobuf/
 This uses a text-based format for describing the structure of the data, 
 but
 the actual data can be in an optimized binary format.



You may write the binary extension if you like :)
May 05 2009
prev sibling parent reply "Saaa" <empty needmail.com> writes:
My first stab at the get function.
As you might see, I need help :D
Thanks!

How do I make the function take a variadic argument and get its type?

//I guess the void pointer isn't the correct way, but I don't know any other 
way
void get(in char[][] file, in char[] indentifier, void* var)
{

TypeInfo type = typeof(var);
int row;

// Is it possible to check it is an array, no matter the depth?
if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
{
  throw new exception;
  //which kind of exception should I throw?
}

if (file.length == 0 || file.length >int.max)
{
  throw new exception;
  //ditto
}

//gets the row number of the first correct occurence for which the following 
is true
//( file[row][0..len] ==  type.stringof ~ ' ' ~ identifier )
row = getRow(file, type, identifier);
if (row < 0 || row >=line.length)
{
  throw new exception;
  //same question :)
}
//parses the file from row
//what is the prototype of a function which has a variadic return type?
var = parse(file, row, type, identifier);
//can all types hold null?
if (var == null)
{
  throw new exception;
  //again
}

return;
}
May 05 2009
parent reply Christopher Wright <dhasenan gmail.com> writes:
Saaa wrote:

 My first stab at the get function.
 As you might see, I need help :D
 Thanks!
 
 How do I make the function take a variadic argument and get its type?



void get(in char[][] file, in char[] identifier, ...)
{
	TypeInfo type = _arguments[0];
	void* var = _argptr;
	// etc
}


 void get(in char[][] file, in char[] indentifier, void* var)
 {
 
 TypeInfo type = typeof(var);
 int row;
 
 // Is it possible to check it is an array, no matter the depth?



If you use tango:

import tango.core.RuntimeTraits;
if (isArray(type)) {}

Otherwise, just copy the code from RuntimeTraits. The functions you 
would need are: realType, isStaticArray, isDynamicArray, isArray


 if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
 {
   throw new exception;
   //which kind of exception should I throw?



Define your own exception type.


 //what is the prototype of a function which has a variadic return type?



"Variadic" means that the function can take any number of arguments.

Use Variant, from std.variant or tango.core.Variant. It's typesafe. You 
could also return a void*, but that will require allocation.


 var = parse(file, row, type, identifier);
 //can all types hold null?



If you use Variant, you can check isEmpty / hasValue or some such.
May 06 2009
parent "Saaa" <empty needmail.com> writes:
Thanks!
With this I can go on for a bit :)

"Christopher Wright" <dhasenan gmail.com> wrote in message 
news:gtrpo7$2h7j$1 digitalmars.com...

 Saaa wrote:

 My first stab at the get function.
 As you might see, I need help :D
 Thanks!

 How do I make the function take a variadic argument and get its type?



 void get(in char[][] file, in char[] identifier, ...)
 {
 TypeInfo type = _arguments[0];
 void* var = _argptr;
 // etc
 }


 void get(in char[][] file, in char[] indentifier, void* var)
 {

 TypeInfo type = typeof(var);
 int row;

 // Is it possible to check it is an array, no matter the depth?



 If you use tango:

 import tango.core.RuntimeTraits;
 if (isArray(type)) {}

 Otherwise, just copy the code from RuntimeTraits. The functions you would 
 need are: realType, isStaticArray, isDynamicArray, isArray


 if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
 {
   throw new exception;
   //which kind of exception should I throw?



 Define your own exception type.


 //what is the prototype of a function which has a variadic return type?



 "Variadic" means that the function can take any number of arguments.

 Use Variant, from std.variant or tango.core.Variant. It's typesafe. You 
 could also return a void*, but that will require allocation.


 var = parse(file, row, type, identifier);
 //can all types hold null?



 If you use Variant, you can check isEmpty / hasValue or some such.
May 06 2009