digitalmars.D.learn - Using an async buffer
- bioinfornatics (52/52) Nov 07 2019 Dear,
- =?UTF-8?Q?Ali_=c3=87ehreli?= (25/28) Nov 07 2019 I admit I don't fully understand the lifetime issues but removing the
- bioinfornatics (27/56) Nov 08 2019 Thanks a lot Ali, the error message was understandable to me, and
- bioinfornatics (1/2) Nov 08 2019 ... the error message was not understandable to me ...
- bioinfornatics (35/37) Nov 08 2019 I do not have found yet why the line counter is false.
- bioinfornatics (4/14) Nov 08 2019 Updating the code with: (ref ubyte[] buffer){ buffer =
Dear, I try to use the async buffer describe into std.parallelism documentation but my test code core dump! documentation: https://dlang.org/phobos/std_parallelism.html#.TaskPool.asyncBuf.2 snipptet: https://paste.fedoraproject.org/paste/K5W1O1dLzZ0sPV8NeGztUg Trace: $ gdb test (gdb) r Starting program: /env/export/v_home/q_unix/jmercier/Projets/wc/test [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". nb octets: 10485760 nb iterations: 10240 Done [New Thread 0x7ffff681f700 (LWP 34340)] [New Thread 0x7ffff601e700 (LWP 34341)] [New Thread 0x7ffff581d700 (LWP 34342)] [New Thread 0x7ffff501c700 (LWP 34343)] [New Thread 0x7ffff481b700 (LWP 34344)] [New Thread 0x7fffdffff700 (LWP 34345)] [New Thread 0x7fffdf7fe700 (LWP 34346)] [New Thread 0x7fffdeffd700 (LWP 34347)] Program received signal SIGUSR1, User defined signal 1. [Switching to Thread 0x7fffdeffd700 (LWP 34347)] 0x00007ffff72119f5 in pthread_cond_wait GLIBC_2.3.2 () from /lib64/libpthread.so.0 Missing separate debuginfos, use: debuginfo-install glibc-2.17-292.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 (gdb) bt /lib64/libpthread.so.0 from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88 from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88 std.parallelism.TaskPool.startWorkLoop() () from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libphobos2-ldc-shared.so.88 from /env/ig/soft/ig/ldc2-1.18.0/el7-x86_64-generic/lib64/libdruntime-ldc-shared.so.88 /lib64/libpthread.so.0 Thanks for your help Best regards
Nov 07 2019
On 11/07/2019 07:07 AM, bioinfornatics wrote:Dear, I try to use the async buffer describe into std.parallelism documentation but my test code core dump!I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes: // (1) Comment out the module-level variable // File file; // ... system void next(File file, ref ubyte[] buf) { // (2.a) Use the parameter 'file' // (2.b) Adjust the length of the buffer // (rawRead may read less than the requested size) buf = file.rawRead(buf); } // ... // (3) Define a local variable auto file = File(filePath, "rb"); // ... // (4) Use "callables" that use the local 'file': auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf), () => file.eof, bufferSize); Ali
Nov 07 2019
On Friday, 8 November 2019 at 01:12:37 UTC, Ali Çehreli wrote:On 11/07/2019 07:07 AM, bioinfornatics wrote:Thanks a lot Ali, the error message was understandable to me, and the way you fixed differ from the documentation. Does that means they are a bug ? Moreover, this snippet is used to evaluate how to parse efficiently versus the wc -l command. fixed snippet: https://paste.fedoraproject.org/paste/B~PjobIlVXIaJPfMjCcWSA And I have 2 problems 1/ the executable is at least twice a time slower than wc -l I try to increase a) the buffer with a multiple of page size (parameter -n) b) the number of thread (parameter -t) 2/ the result is not exactly the same as wc -l give wich imply a bug while counting \n inside the buffer. $ time ./build/wc_File_by_chunks_async_buffer -n 4 -t 8 -i test_100M 1638656 test_100M real 0m0.106s user 0m0.101s sys 0m0.051s $ time wc -l test_100M 1638400 test_100M real 0m0.067s user 0m0.030s sys 0m0.037s Thanks again Ali without you It was impossible to meDear, I try to use the async buffer describe into std.parallelism documentation but my test code core dump!I admit I don't fully understand the lifetime issues but removing the "code smell" of the modul-level File object solved the issue for me, which requires four changes: // (1) Comment out the module-level variable // File file; // ... system void next(File file, ref ubyte[] buf) { // (2.a) Use the parameter 'file' // (2.b) Adjust the length of the buffer // (rawRead may read less than the requested size) buf = file.rawRead(buf); } // ... // (3) Define a local variable auto file = File(filePath, "rb"); // ... // (4) Use "callables" that use the local 'file': auto asyncReader = taskPool.asyncBuf((ref ubyte[] buf) => next(file, buf), () => file.eof, bufferSize); Ali
Nov 08 2019
the error message was understandable to me,... the error message was not understandable to me ...
Nov 08 2019
On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote:I do not have found yet why the line counter is false. I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation: https://dlang.org/phobos/std_stdio.html#.File.rawRead as example I change the code in order to shuffle line content to get the ability to locate the bug Real last lines are: 1 190121746114132251381321230342516302196252336238211523943272873744285119323293314107 316322221221132661353262123081115418570291330356278322215013742329426 714213310231111593822146521912312869120169289362332157427352432313112226373403123825 6812101511112462691294263101232 90182312212511430133514352114282271133753782360462351124233948222161956731321 11822481725231121323330910521376234322119392811262411335432102108273 463633112212153255811679207 while the code give: 1 190121746114132251381321230342516302196252336238211523943272873744285119323293314107 316322221221132661353262123081115418570291330356278322215013742329426 714213310231111593822146521912312869120169289362332157427352432313112226373403123825 6812101511112462691294263101232 90182312212511430133514352114282271133753782360462351124233948222161956731321 11822481725231121323330910521376234322119392811262411335432102108273 4636331122121532558116792071512 1553203330299234738282167126033321272154232912111312416461868182323242 111932223509162312223104310231321116573254736811479599513441112318312221230321154 11363210249282132717260536372386522748224512596323581311 121932131303243861212327470532121636029110222323531121763 2499315312114672213683218207122451351984311032612832096363463812 .... I think we see the buffer content which is reused. so A big part of last read come from the previous chunkthe error message was understandable to me,... the error message was not understandable to me ...
Nov 08 2019
bioinfornatics <bioinfornatics fedoraproject.org> On Friday, 8 November 2019 at 14:32:25 UTC, bioinfornatics wrote:On Friday, 8 November 2019 at 08:58:36 UTC, bioinfornatics wrote:Updating the code with: (ref ubyte[] buffer){ buffer = file.rawRead(buffer); } fix the problem.[...]I do not have found yet why the line counter is false. I can tell if the amount to read imply that the last read is not not strictly equal to the buffer then the result is false as if it left less thing to read the result is not shorter than requested buffer size as it is told into the documentation: https://dlang.org/phobos/std_stdio.html#.File.rawRead [...]
Nov 08 2019