www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - The app hanging after reach 1750MB of RAM

reply Suliman <evermind live.ru> writes:
I am writing app that extract data from DB to array of structures.

	void getSingleTrackInfo()
	{
		
		foreach(item; getTablesGPSSensorList)
		{
			ResultRange result = mysqlconnection.query(sqlquery);
			auto MySQLPointsLonLat = result.array;
			
			carGPSPoint cargpspoint; // create struct
			carGPSPoint [] cargpspoints; // create array of structures

			foreach(i, point;MySQLPointsLonLat)
			{
				cargpspoint.id = point[0].coerce!ulong;
				cargpspoint.recordDate = point[1].coerce!string;
				cargpspoint.velocity = point[2].coerce!double;
				cargpspoint.lat = point[3].coerce!string;
				cargpspoint.lon = point[4].coerce!string;
				cargpspoints ~=cargpspoint;
					
			}

		}
	}


I expected that on every step:
`foreach(item; getTablesGPSSensorList)` structures `carGPSPoint` 
will be recreated and memory will be free. But App after starting 
begin eat memory, and do it till reach 1750 of RAM. After it it's 
not crush, but simply stop other processing.

1. By the code on every iterate `carGPSPoint` should be recreated 
and memory should be free, why this does not happen?
2. How to free my memory?
Apr 18
next sibling parent reply Suliman <evermind live.ru> writes:
Also I can't understand why app take so much memory? I checked 
array of structures size with this code:

auto mymem = cargpspoints.length * 
typeof(cargpspoints[0]).sizeof;			
writeln(mymem);

And it's print: 16963440
it's about 16MB...

What is takes all other memory?
Apr 18
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 18 April 2017 at 13:28:57 UTC, Suliman wrote:
 Also I can't understand why app take so much memory? I checked 
 array of structures size with this code:

 auto mymem = cargpspoints.length * 
 typeof(cargpspoints[0]).sizeof;			
 writeln(mymem);

 And it's print: 16963440
 it's about 16MB...

 What is takes all other memory?
1. You're measuring it wrong. Array length is already measured in terms of type size. But remember that every call result.array will allocate, and very call coerce!string will also allocate. 2. Since you're iterating over every result once, there's no point in converting it to an array first. 3. Consider using the Row.toStruct() method for conversion. 4. Consider using std.array.appender instead of ~=. void getSingleTrackInfo() { foreach(item; getTablesGPSSensorList) { ResultRange result = mysqlconnection.query(sqlquery); carGPSPoint cargpspoint; // create struct auto arr = appender!(carGPSPoint[]); // create array of structures foreach(row; result) { arr ~= row.toStruct(cargpspoint); } // arr.data should hold the array of structures } }
Apr 18
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov wrote:

         foreach(row; result)
         {
             arr ~= row.toStruct(cargpspoint);
         }
Sorry, this should be foreach(row; result) { row.toStruct(cargpspoint); arr ~= cargpspoint; }
Apr 18
parent Suliman <evermind live.ru> writes:
On Tuesday, 18 April 2017 at 14:15:59 UTC, Stanislav Blinov wrote:
 On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov 
 wrote:

         foreach(row; result)
         {
             arr ~= row.toStruct(cargpspoint);
         }
Sorry, this should be foreach(row; result) { row.toStruct(cargpspoint); arr ~= cargpspoint; }
Thanks I will try! But why in my code memory do not released? Do you have any idea?
Apr 18
prev sibling parent reply Suliman <evermind live.ru> writes:
 auto mymem = cargpspoints.length * 
 typeof(cargpspoints[0]).sizeof;			
 writeln(mymem);

 And it's print: 16963440
 it's about 16MB...

 What is takes all other memory?
1. You're measuring it wrong. Array length is already measured in terms of type size.
So should I do: cargpspoints.length * cargpspoints[0].sizeof ? Btw, `cargpspoints.length * typeof(cargpspoints[0]).sizeof` and `cargpspoints.length * cargpspoints[0].sizeof` show same result.
Apr 19
next sibling parent Suliman <evermind live.ru> writes:
I have added GC.stat 
https://dlang.org/library/core/memory/gc.stats.html here the 
result:

freeSize: 49698640  | usedSize: 170502320
freeSize: 41174592  | usedSize: 217823680
freeSize: 53868576  | usedSize: 247072736
freeSize: 86494800  | usedSize: 307769776
freeSize: 58176640  | usedSize: 499665792
freeSize: 148233232 | usedSize: 534389744
freeSize: 148141376 | usedSize: 534481600
freeSize: 108467312 | usedSize: 641264528
freeSize: 55118432  | usedSize: 694613408
freeSize: 80579472  | usedSize: 803370096
freeSize: 78444480  | usedSize: 1006831680
freeSize: 291629360 | usedSize: 860755664
freeSize: 242912736 | usedSize: 976581152
freeSize: 241673232 | usedSize: 977820656
freeSize: 168092160 | usedSize: 1118510592
freeSize: 128405616 | usedSize: 1426632592
freeSize: 73146272  | usedSize: 1616109664
freeSize: 17962320  | usedSize: 1671293616
freeSize: 20342912  | usedSize: 1736021888

on the last value app is hanging.
Apr 19
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 19 April 2017 at 07:28:32 UTC, Suliman wrote:

 1. You're measuring it wrong. Array length is already measured 
 in terms of type size.
So should I do: cargpspoints.length * cargpspoints[0].sizeof ?
No. .sizeof is the statically known size of a type, it can't take into account dynamically allocated memory. cargpspoint.length * cargpspoints[0].sizeof will tell you the estimate size of the array, in bytes. But each of the elements also has strings - dynamic arrays that are allocated elsewhere, their length is not included in that calculation. So you could iterate over the array and sum up lengths of all strings to get an estimate. Even then, that's just that: an estimate. Actual amount of memory allocated for a dynamic array T[] *may* be greater than length * T.sizeof. The only way to know that is to query the allocator used (in this case, GC).
Apr 19
parent Suliman <evermind live.ru> writes:
The problem is solved. See for more detail 
https://github.com/mysql-d/mysql-native/issues/104
Apr 24
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:
 I am writing app that extract data from DB to array of 
 structures.

 	void getSingleTrackInfo()
 	{
 		
 		foreach(item; getTablesGPSSensorList)
 		{
 			ResultRange result = mysqlconnection.query(sqlquery);
 			auto MySQLPointsLonLat = result.array;
Is ResultRange closing query when exhausted? Did you try to call .close() on it after work?
Apr 19
parent Suliman <evermind live.ru> writes:
On Wednesday, 19 April 2017 at 15:18:32 UTC, crimaniak wrote:
 On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:
 I am writing app that extract data from DB to array of 
 structures.

 	void getSingleTrackInfo()
 	{
 		
 		foreach(item; getTablesGPSSensorList)
 		{
 			ResultRange result = mysqlconnection.query(sqlquery);
 			auto MySQLPointsLonLat = result.array;
Is ResultRange closing query when exhausted? Did you try to call .close() on it after work?
Yes I tried: ResultRange result = mysqlconnection.query(sqlquery); auto MySQLPointsLonLat = result.array; result.close(); Did not help :(
Apr 19