www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - any update on large file support for linux?

reply Lars Holowko <lars.holowko gmail.com> writes:
Hi everybody,

Are there any status updates on the large file support for dmd on Linux?
I found this bug:=A0http://d.puremagic.com/issues/show_bug.cgi?id=3D3409
which has not been commented on for quite a while.
I was trying to dig around in druntime and phobos - a lot seems
already in place so I was hoping that I could get it to work with
minor hacking:
- =A0enable=A0__USE_LARGEFILE64 for 32 bit dmd in
druntime/src/core/sys/posix/config.d (no idea why this is determined
by the processor's native pointer size)
- change std.stdio.File.seek to call=A0fseeko instead of fseek

=A0=A0 =A0void seek(long offset, int origin =3D SEEK_SET)
=A0=A0 =A0{
=A0=A0 =A0 =A0 =A0enforce(p && p.handle,
=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0"Attempting to seek() in an unopened file=
");
=A0=A0 =A0 =A0 =A0//     Dubious: why is fseek in std.c.stdio taking an int=
???
=A0=A0 =A0 =A0 =A0errnoEnforce(core.sys.posix.stdio.fseeko(
// =A0 =A0 =A0 =A0errnoEnforce(core.stdc.stdio.fseek(
// =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0p.handle, to!int(offset), origin)=
 =3D=3D 0,
=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0p.handle, offset, origin) =3D=3D =
0,
=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0"Could not seek in file `"~p.name~"'");
=A0=A0 =A0}


But when I run the slightly modified std.stdio sample:

import std.stdio;
import core.sys.posix.sys.types;
void main(string args[])
{
=A0=A0 =A0writefln("Typeof(off_t) =3D %s", typeid(off_t));
=A0=A0 =A0auto f =3D File("test.txt", "w"); // open for writing
=A0=A0 =A0f.write("Hello");
=A0=A0 =A0f.seek(1024 * 1024 * 1024 * 6, SEEK_SET);
=A0=A0 =A0if (args.length > 1)
=A0=A0 =A0{
=A0=A0 =A0 =A0 =A0auto g =3D f; // now g and f write to the same file
=A0=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0// internal reference count is 2
=A0=A0 =A0 =A0 =A0g.write(", ", args[1]);
=A0=A0 =A0 =A0 =A0// g exits scope, reference count decreases to 1
=A0=A0 =A0}
=A0=A0 =A0f.writeln("!");
}

I get something like this:

Typeof(off_t) =3D long
std.exception.ErrnoException std/stdio.d(526): Could not seek in file
`test.txt' (Invalid argument)
----------------
./io_test() [0x8055833]
./io_test() [0x8054d40]
./io_test() [0x8049859]
./io_test() [0x804f6b6]
./io_test() [0x804f610]
./io_test() [0x804f6fa]
./io_test() [0x804f610]
./io_test() [0x804f5b6]
/lib32/libc.so.6(__libc_start_main+0xe6) [0xf763bbd6]
./io_test() [0x8049741]



nm io_test | grep -e fopen -e fseek
080552f0 T _D3std5stdio5fopenFxAaxAaZPOS4core4stdc5stdio6_iobuf
=A0=A0 =A0 =A0 =A0 U fopen64  GLIBC_2.1
=A0=A0 =A0 =A0 =A0 U fseeko64  GLIBC_2.1

The 64 bit fopen and fseek calls seem to be linked, off_t is correctly
aliased to long. Does anyone have an idea what else I might be
missing?
Thanks a lot,

Lars
Sep 03 2010
parent reply "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
The SVN version of std.stdio supports large files on Linux and OSX.  The 
next release will be a nice one, I think. :)

-Lars
Sep 07 2010
parent reply Lars Holowko <lars.holowko gmail.com> writes:
On Tue, Sep 7, 2010 at 1:08 AM, Lars T. Kyllingstad
<public kyllingen.nospamnet> wrote:

 The SVN version of std.stdio supports large files on Linux and OSX. =A0Th=


e

 next release will be a nice one, I think. :)

 -Lars


Thanks Lars,

For the hint to the svn versions. Things seem to work there. What
really surprised me is that dmd compiles

 f.seek(1024 * 1024 * 1024 * 6, SEEK_SET);

but fails with

std.exception.ErrnoException std/stdio.d(538): Could not seek in file
`test.txt' (Invalid argument)

whereas

 f.seek(1024 * 1024 * 1024 * 6L, SEEK_SET);

works fine.

I did not realize that 1024 * 1024 * 1024 * 6 turns negative and then
gets converted to a negative long (without even a warning). Overseeing
that had killed my own efforts to hack 64-bit support into phobos ;-)

Thanks again,

(another ;-)) Lars
Sep 07 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/7/10 11:05 CDT, Lars Holowko wrote:

 On Tue, Sep 7, 2010 at 1:08 AM, Lars T. Kyllingstad
 <public kyllingen.nospamnet>  wrote:

 The SVN version of std.stdio supports large files on Linux and OSX.  The
 next release will be a nice one, I think. :)

 -Lars


 Thanks Lars,

 For the hint to the svn versions. Things seem to work there. What
 really surprised me is that dmd compiles

   f.seek(1024 * 1024 * 1024 * 6, SEEK_SET);

 but fails with

 std.exception.ErrnoException std/stdio.d(538): Could not seek in file
 `test.txt' (Invalid argument)

 whereas

   f.seek(1024 * 1024 * 1024 * 6L, SEEK_SET);

 works fine.

 I did not realize that 1024 * 1024 * 1024 * 6 turns negative and then
 gets converted to a negative long (without even a warning). Overseeing
 that had killed my own efforts to hack 64-bit support into phobos ;-)

 Thanks again,

 (another ;-)) Lars


Hmmm... the compiler could and should warn about integer overflow in 
computed constant. I suggest you file this as an improvement in bugzilla.

Glad to hear large files are working for you.


Andrei
Sep 07 2010
parent Lars Holowko <lars.holowko gmail.com> writes:
On Tue, Sep 7, 2010 at 9:34 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 I did not realize that 1024 * 1024 * 1024 * 6 turns negative and then
 gets converted to a negative long (without even a warning). Overseeing
 that had killed my own efforts to hack 64-bit support into phobos ;-)


 Hmmm... the compiler could and should warn about integer overflow in
 computed constant. I suggest you file this as an improvement in bugzilla.

 Glad to hear large files are working for you.


 Andrei


I filed an enhancement request
http://d.puremagic.com/issues/show_bug.cgi?id=4835 as Andrei had
recommended.

Lars
Sep 07 2010