digitalmars.D.learn - The app hanging after reach 1750MB of RAM
- Suliman (31/31) Apr 18 2017 I am writing app that extract data from DB to array of structures.
- Suliman (8/8) Apr 18 2017 Also I can't understand why app take so much memory? I checked
- Stanislav Blinov (23/31) Apr 18 2017 1. You're measuring it wrong. Array length is already measured in
- Stanislav Blinov (7/11) Apr 18 2017 Sorry, this should be
- Suliman (3/15) Apr 18 2017 Thanks I will try! But why in my code memory do not released? Do
- Suliman (4/14) Apr 19 2017 So should I do:
- Suliman (23/23) Apr 19 2017 I have added GC.stat
- Stanislav Blinov (14/18) Apr 19 2017 No. .sizeof is the statically known size of a type, it can't take
- Suliman (2/2) Apr 24 2017 The problem is solved. See for more detail
- crimaniak (3/12) Apr 19 2017 Is ResultRange closing query when exhausted? Did you try to call
- Suliman (6/19) Apr 19 2017 Yes I tried:
I am writing app that extract data from DB to array of structures.
void getSingleTrackInfo()
{
foreach(item; getTablesGPSSensorList)
{
ResultRange result = mysqlconnection.query(sqlquery);
auto MySQLPointsLonLat = result.array;
carGPSPoint cargpspoint; // create struct
carGPSPoint [] cargpspoints; // create array of structures
foreach(i, point;MySQLPointsLonLat)
{
cargpspoint.id = point[0].coerce!ulong;
cargpspoint.recordDate = point[1].coerce!string;
cargpspoint.velocity = point[2].coerce!double;
cargpspoint.lat = point[3].coerce!string;
cargpspoint.lon = point[4].coerce!string;
cargpspoints ~=cargpspoint;
}
}
}
I expected that on every step:
`foreach(item; getTablesGPSSensorList)` structures `carGPSPoint`
will be recreated and memory will be free. But App after starting
begin eat memory, and do it till reach 1750 of RAM. After it it's
not crush, but simply stop other processing.
1. By the code on every iterate `carGPSPoint` should be recreated
and memory should be free, why this does not happen?
2. How to free my memory?
Apr 18 2017
Also I can't understand why app take so much memory? I checked array of structures size with this code: auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof; writeln(mymem); And it's print: 16963440 it's about 16MB... What is takes all other memory?
Apr 18 2017
On Tuesday, 18 April 2017 at 13:28:57 UTC, Suliman wrote:Also I can't understand why app take so much memory? I checked array of structures size with this code: auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof; writeln(mymem); And it's print: 16963440 it's about 16MB... What is takes all other memory?1. You're measuring it wrong. Array length is already measured in terms of type size. But remember that every call result.array will allocate, and very call coerce!string will also allocate. 2. Since you're iterating over every result once, there's no point in converting it to an array first. 3. Consider using the Row.toStruct() method for conversion. 4. Consider using std.array.appender instead of ~=. void getSingleTrackInfo() { foreach(item; getTablesGPSSensorList) { ResultRange result = mysqlconnection.query(sqlquery); carGPSPoint cargpspoint; // create struct auto arr = appender!(carGPSPoint[]); // create array of structures foreach(row; result) { arr ~= row.toStruct(cargpspoint); } // arr.data should hold the array of structures } }
Apr 18 2017
On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov wrote:
foreach(row; result)
{
arr ~= row.toStruct(cargpspoint);
}
Sorry, this should be
foreach(row; result)
{
row.toStruct(cargpspoint);
arr ~= cargpspoint;
}
Apr 18 2017
Suliman <evermind live.ru> On Tuesday, 18 April 2017 at 14:15:59 UTC, Stanislav Blinov wrote:On Tuesday, 18 April 2017 at 14:09:28 UTC, Stanislav Blinov wrote:Thanks I will try! But why in my code memory do not released? Do you have any idea?foreach(row; result) { arr ~= row.toStruct(cargpspoint); }Sorry, this should be foreach(row; result) { row.toStruct(cargpspoint); arr ~= cargpspoint; }
Apr 18 2017
So should I do: cargpspoints.length * cargpspoints[0].sizeof ? Btw, `cargpspoints.length * typeof(cargpspoints[0]).sizeof` and `cargpspoints.length * cargpspoints[0].sizeof` show same result.auto mymem = cargpspoints.length * typeof(cargpspoints[0]).sizeof; writeln(mymem); And it's print: 16963440 it's about 16MB... What is takes all other memory?1. You're measuring it wrong. Array length is already measured in terms of type size.
Apr 19 2017
I have added GC.stat https://dlang.org/library/core/memory/gc.stats.html here the result: freeSize: 49698640 | usedSize: 170502320 freeSize: 41174592 | usedSize: 217823680 freeSize: 53868576 | usedSize: 247072736 freeSize: 86494800 | usedSize: 307769776 freeSize: 58176640 | usedSize: 499665792 freeSize: 148233232 | usedSize: 534389744 freeSize: 148141376 | usedSize: 534481600 freeSize: 108467312 | usedSize: 641264528 freeSize: 55118432 | usedSize: 694613408 freeSize: 80579472 | usedSize: 803370096 freeSize: 78444480 | usedSize: 1006831680 freeSize: 291629360 | usedSize: 860755664 freeSize: 242912736 | usedSize: 976581152 freeSize: 241673232 | usedSize: 977820656 freeSize: 168092160 | usedSize: 1118510592 freeSize: 128405616 | usedSize: 1426632592 freeSize: 73146272 | usedSize: 1616109664 freeSize: 17962320 | usedSize: 1671293616 freeSize: 20342912 | usedSize: 1736021888 on the last value app is hanging.
Apr 19 2017
On Wednesday, 19 April 2017 at 07:28:32 UTC, Suliman wrote:No. .sizeof is the statically known size of a type, it can't take into account dynamically allocated memory. cargpspoint.length * cargpspoints[0].sizeof will tell you the estimate size of the array, in bytes. But each of the elements also has strings - dynamic arrays that are allocated elsewhere, their length is not included in that calculation. So you could iterate over the array and sum up lengths of all strings to get an estimate. Even then, that's just that: an estimate. Actual amount of memory allocated for a dynamic array T[] *may* be greater than length * T.sizeof. The only way to know that is to query the allocator used (in this case, GC).1. You're measuring it wrong. Array length is already measured in terms of type size.So should I do: cargpspoints.length * cargpspoints[0].sizeof ?
Apr 19 2017
The problem is solved. See for more detail https://github.com/mysql-d/mysql-native/issues/104
Apr 24 2017
On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:
I am writing app that extract data from DB to array of
structures.
void getSingleTrackInfo()
{
foreach(item; getTablesGPSSensorList)
{
ResultRange result = mysqlconnection.query(sqlquery);
auto MySQLPointsLonLat = result.array;
Is ResultRange closing query when exhausted? Did you try to call
.close() on it after work?
Apr 19 2017
On Wednesday, 19 April 2017 at 15:18:32 UTC, crimaniak wrote:On Tuesday, 18 April 2017 at 11:43:24 UTC, Suliman wrote:Yes I tried: ResultRange result = mysqlconnection.query(sqlquery); auto MySQLPointsLonLat = result.array; result.close(); Did not help :(I am writing app that extract data from DB to array of structures. void getSingleTrackInfo() { foreach(item; getTablesGPSSensorList) { ResultRange result = mysqlconnection.query(sqlquery); auto MySQLPointsLonLat = result.array;Is ResultRange closing query when exhausted? Did you try to call .close() on it after work?
Apr 19 2017