www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 11791] New: std.file.write failed to write huge files

reply d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791

           Summary: std.file.write failed to write huge files
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: Phobos
        AssignedTo: nobody puremagic.com
        ReportedBy: ilyayaroshenko gmail.com


--- Comment #0 from Илья Ярошенко <ilyayaroshenko gmail.com>
2013-12-21 10:49:34 PST ---
std.file.write failed to write files over 4GB (in my case 3.3GB).

But std.stdio works well.

-- 
Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 21 2013
next sibling parent d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791


Peter Alexander <peter.alexander.au gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |peter.alexander.au gmail.co
                   |                            |m


--- Comment #1 from Peter Alexander <peter.alexander.au gmail.com> 2014-01-27
14:03:25 PST ---
Would it be possible to provide more information?  In what way does it fail
(exception, seg fault, just stops writing at a certain point, etc.)

-- 
Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 27 2014
prev sibling next sibling parent d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791



--- Comment #2 from Илья Ярошенко <ilyayaroshenko gmail.com>
2014-01-28 00:42:59 PST ---
import std.file;

void main() {
    auto data = new byte[2L^^31];
    std.file.write("data", data);
}

std.file.FileException std/file.d(391): data: Success
----------------
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
std.file.writeImpl(const(char[]), const(void[]), const(uint))+0xe1) [0x451fe1]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
std.file.write(const(char[]), const(void[]))+0x38) [0x451ef8]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(_Dmain+0x52)
[0x44f242]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll().void __lambda1()+0x18) [0x44f990]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x44f8ea]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).runAll()+0x30) [0x44f950]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void
rt.dmain2._d_run_main(int, char**, extern (C) int
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x44f8ea]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(_d_run_main+0x1a3)
[0x44f86b]
/tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(main+0x17)
[0x44f25f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f73bff53de5]
----------------
[Finished in 18.5s with exit code 1]
[cmd: ['rdmd', '/home/ilya/Desktop/test.d']]
[dir: /home/ilya/Desktop]
[path:
/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/ilya/ilyayaroshenko gmail.com/system/ldc/bin/]

-- 
Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 28 2014
prev sibling next sibling parent d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791



--- Comment #3 from Илья Ярошенко <ilyayaroshenko gmail.com>
2014-01-28 00:45:15 PST ---
(In reply to comment #2)
 import std.file;
 
 void main() {
     auto data = new byte[2L^^31];
     std.file.write("data", data);
 }
 
 std.file.FileException std/file.d(391): data: Success

works fine with 2L^^30 -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Jan 28 2014
prev sibling next sibling parent d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791



--- Comment #4 from Peter Alexander <peter.alexander.au gmail.com> 2014-01-28
14:17:52 PST ---
(In reply to comment #3)
 (In reply to comment #2)
 import std.file;
 
 void main() {
     auto data = new byte[2L^^31];
     std.file.write("data", data);
 }
 
 std.file.FileException std/file.d(391): data: Success

works fine with 2L^^30

Ok, so that line indicates that the call to "close" failed (returned non-zero), but didn't set errno, which is weird. One possibility is that you are running a 32-bit kernel, and write will only be able to write INT_MAX bytes, which would explain the error (although I would have hoped for it to set errno). Are you running a 32-bit kernel? I suppose an alternate question is: can you write a file like this using plain C functions, or is it a D specific issue? Phobos literally just forwards the call onto a few C functions (open file, write, close), so I suspect it is not an issue in Phobos. -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Jan 28 2014
prev sibling parent d-bugmail puremagic.com writes:
https://d.puremagic.com/issues/show_bug.cgi?id=11791



--- Comment #5 from Илья Ярошенко <ilyayaroshenko gmail.com>
2014-01-29 00:37:44 PST ---
(In reply to comment #4)
 (In reply to comment #3)
 (In reply to comment #2)
 import std.file;
 
 void main() {
     auto data = new byte[2L^^31];
     std.file.write("data", data);
 }
 
 std.file.FileException std/file.d(391): data: Success

works fine with 2L^^30

Ok, so that line indicates that the call to "close" failed (returned non-zero), but didn't set errno, which is weird. One possibility is that you are running a 32-bit kernel, and write will only be able to write INT_MAX bytes, which would explain the error (although I would have hoped for it to set errno). Are you running a 32-bit kernel? I suppose an alternate question is: can you write a file like this using plain C functions, or is it a D specific issue? Phobos literally just forwards the call onto a few C functions (open file, write, close), so I suspect it is not an issue in Phobos.

kubuntu 13.04 $ uname -a Linux Y 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux D programs also are x86_64, but it is possible to use x86_32 C library in ubuntu. With std.stdio.File I can write an 6 GB files. -- Configure issuemail: https://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Jan 29 2014