www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 2842] New: std.file.listdir on OSX produces invalid UTF-8 sequence

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842

           Summary: std.file.listdir on OSX produces invalid UTF-8 sequence
           Product: D
           Version: 1.043
          Platform: PC
        OS/Version: Mac OS X
            Status: NEW
          Severity: minor
          Priority: P2
         Component: Phobos
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: af liquidstate.eu


std.file.listdir throws an "invalid UTF-8 sequence" exception.
please see the source file and the produced output below.

thanks.

------------------------------
import std.stdio, std.file;

int main(char[][] args)
{
        foreach (file; listdir("."))
                writefln("%s", file);   
        return 0;
}
------------------------------
blah:misc user$ dmd listdir.d
blah:misc user$ ./listdir
..

some_directory
Error: 1invalid UTF-8 sequence
blah:misc user$


-- 
Apr 16 2009
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Mike Kinney <mike.kinney gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mike.kinney gmail.com





PDT ---
What does the "ls -al" show for that directory? 

Is it happening on all files? 

It works ok on v2.031 when run from the samples dir.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jul 16 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Justin Whear <mrjnewt hotmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mrjnewt hotmail.com



---
I can confirm this bug using DMD 1.050 on OS X 10.5.
Running program by OP:
emsi-testings-mac-mini:ratchet emsitesting$ ./listdir 
.
.
ogout
.
e
istory
.
s
.
_central.d
t
re
sion
.
.
ac.sh
inux.sh
_central.o
_central
.
.d
.o
.


The function seems to be trimming the names arbitrarily. Here's ls -al in same
directory:
emsi-testings-mac-mini:ratchet emsitesting$ ls -al
total 1324
drwx------   0 emsitesting  staff   16384 Nov  9 13:10 .
drwxrwxrwt   5 root         admin     170 Nov  5 11:35 ..
-rw--w----   1 emsitesting  staff    6148 Nov  5 11:27 .DS_Store
-rw-------   1 emsitesting  staff    7467 Nov  9 12:36 .bash_history
-rw-r--r--   1 emsitesting  staff     220 Nov  4 11:11 .bash_logout
-rw-r--r--   1 emsitesting  staff    3211 Nov  4 13:32 .bashrc
drwxr-xr-x   2 emsitesting  staff   16384 Nov  4 13:27 .cache
-rw-r--r--   1 emsitesting  staff     675 Nov  4 11:11 .profile
-rw-r--r--   1 emsitesting  staff      81 Nov  6 12:20 .ratchet
drwx------   2 emsitesting  staff   16384 Nov  4 13:53 .ssh
drwxr-xr-x   3 emsitesting  staff   16384 Nov  6 10:50 .subversion
-rwxrwxr-x   1 emsitesting  staff     228 Nov  6 13:41 build_linux.sh
-rwxr-xr-x   1 emsitesting  staff     187 Nov  6 14:17 build_mac.sh
lrwxrwxrwx   1 emsitesting  staff       6 Nov  6 10:50 ccb -> d/ccb/
lrwxrwxrwx   1 emsitesting  staff      16 Nov  6 10:54 chains ->
d/chains/chains/
drwxr-xr-x  26 emsitesting  staff   16384 Nov  9 12:02 d
-rwxr-xr-x   1 emsitesting  staff  261192 Nov  9 13:10 listdir
-rw-rw-r--   1 emsitesting  staff     138 Nov  9 13:10 listdir.d
-rw-rw-r--   1 emsitesting  staff    2748 Nov  9 13:10 listdir.o
drwxr-xr-x   6 emsitesting  staff   16384 Nov  9 12:34 projects
-rwxr-xr-x   1 emsitesting  staff  260223 Nov  6 13:37 ratchet_central
-rw-r--r--   1 emsitesting  staff    2331 Nov  6 13:37 ratchet_central.d
-rw-r--r--   1 emsitesting  staff    9456 Nov  6 13:37 ratchet_central.o

This one's rather critical as I have code that I'm trying to get running on OS
X.



 What does the "ls -al" show for that directory? 
 
 Is it happening on all files? 
 
 It works ok on v2.031 when run from the samples dir.
-- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 09 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Justin Whear <mrjnewt hotmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|minor                       |major



---
This bug still exists in 1.063.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 12 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Jakob Bornecrantz <wallbraker gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |wallbraker gmail.com



14:57:16 PDT ---
*** Issue 6011 has been marked as a duplicate of this issue. ***

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 15 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842




15:05:16 PDT ---
Still exists in 1.064, 1.065 and 1.068.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 15 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Jakob Bornecrantz <wallbraker gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Priority|P2                          |P1
            Version|1.043                       |D1
           Severity|major                       |critical



15:19:47 PDT ---
Looks like the cause of this is due to std.c.linux.linux is being used in
std.file where dirent is defined to this:

    struct dirent
    {
        uint d_ino;             // this is int on some linuxes
        off_t d_off;
        ushort d_reclen;
        ubyte d_type;           // this field isn't there on some linuxes
        char[256] d_name;
    }

While in /usr/include/sys/dirent.h it is defined to this:

struct dirent {
        ino_t d_ino;                    /* file number of entry */
        __uint16_t d_reclen;            /* length of this record */
        __uint8_t  d_type;              /* file type, see below */
        __uint8_t  d_namlen;            /* length of string in d_name */
        char d_name[__DARWIN_MAXNAMLEN + 1];    /* name must be no longer than
this */
};

Cheers Jakob.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 15 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842




15:20:31 PDT ---
Created an attachment (id=976)
Patch that should fix the problem, untested.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 15 2011
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842




15:22:58 PDT ---
Created an attachment (id=977)
Test program from other bug.

Here is the test program from issue 6011.

Cheers Jakob.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 15 2011
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2842


Walter Bright <bugzilla digitalmars.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |bugzilla digitalmars.com
         Resolution|                            |FIXED



21:04:16 PDT ---
https://github.com/D-Programming-Language/phobos/commit/149bab20505637a2a77a13081c2b0fb93b27f959

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
May 27 2011