www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 9045] New: Feature rest for std.asscii => function isNewline

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045

           Summary: Feature rest for std.asscii => function isNewline
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Phobos
        AssignedTo: nobody puremagic.com
        ReportedBy: bioinfornatics gmail.com


--- Comment #0 from bioinfornatics <bioinfornatics gmail.com> 2012-11-18
16:48:50 PST ---
hi,

std.ascii provide some function to check ascii char, to get newline char from
your OS but not a function to said if a char is a newline as does this
function:


bool isNewline(dchar c)  safe pure nothrow {
    return ( c == 0x0A || c == 0x0D )? true : false;
}

is Possible to add it ?

regards

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045


bearophile_hugs eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs eml.cc


--- Comment #1 from bearophile_hugs eml.cc 2012-11-18 16:59:06 PST ---
See representation on various systems:

http://en.wikipedia.org/wiki/Newline

In particular:
On Unix, and Mac OS X: LF (1 char)
On Windows: CR+LF (2 chars)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #2 from bioinfornatics <bioinfornatics gmail.com> 2012-11-18
17:08:08 PST ---
(In reply to comment #1)

 See representation on various systems:
 
 http://en.wikipedia.org/wiki/Newline
 
 In particular:
 On Unix, and Mac OS X: LF (1 char)
 On Windows: CR+LF (2 chars)


yes not easy or into string

bool isNewline(in dchar[] c)  safe pure nothrow {
    bool result = false;
    if( c[0] == 0x0A)
        result = true;
    else if( c.length >= 2 && c[0] == 0x0D && c[1] == 0x0A)
        result = true;
    ...
    ...
    return result;
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045


Jonathan M Davis <jmdavisProg gmx.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jmdavisProg gmx.com


--- Comment #3 from Jonathan M Davis <jmdavisProg gmx.com> 2012-11-18 17:15:06
PST ---
Just check whether it equals std.ascii.newline.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #4 from bioinfornatics <bioinfornatics gmail.com> 2012-11-18
23:19:27 PST ---
(In reply to comment #3)

 Just check whether it equals std.ascii.newline.


yes but the problem is when you need to write a parser and detect end of line.
You can get the given file from various operating system so you can"t use
std.ascii.newline as is system specific.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #5 from Jonathan M Davis <jmdavisProg gmx.com> 2012-11-18 23:28:15
PST ---
Technically speaking, if you don't know which type of line endings a file uses,
you can't possibly correctly determine when you've reached the end of a line.
Best case, you have to assume that '\r\n' and '\n' both designate the end of
the line, whereas it's perfectly legal to have a '\r' be the last character on
a line (i.e. the one before the characters indicating the end of the line) in
Linux, and it's perfectly valid to have '\n' be in the middle of line on
Windows. So, parsing with the assumption that both '\r\n' and '\n' indicate the
end of the line is actually incorrect no matter what OS you're on.

That doesn't mean that it's not entirely unreasonable to have such a function,
but it does mean that it can't possibly be 100% correct.

On an unrelated note, I'd point out that

return ( c == 0x0A || c == 0x0D )? true : false;

is redundant. The ternary operator is completely unnecessary. It would be
better if it were

return c == 0x0A || c == 0x0D;

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 18 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045


Dmitry Olshansky <dmitry.olsh gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |dmitry.olsh gmail.com


--- Comment #6 from Dmitry Olshansky <dmitry.olsh gmail.com> 2012-11-19
10:49:39 PST ---
Somewhat related.

Unicode new line sequence is defined as:
\u{A} | \u{B} | \u{C} | \u{D} | \u{85} | \u{2028} | \u{2029} | \u{D A}

Note that sequence '\r\n' counts as one line end.

See the note here for example:
http://www.unicode.org/reports/tr18/#Line_Boundaries

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 19 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045


monarchdodra gmail.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |monarchdodra gmail.com


--- Comment #7 from monarchdodra gmail.com 2012-11-20 05:42:25 PST ---
(In reply to comment #1)

 See representation on various systems:
 
 http://en.wikipedia.org/wiki/Newline
 
 In particular:
 On Unix, and Mac OS X: LF (1 char)
 On Windows: CR+LF (2 chars)


(In reply to comment #5)

 Technically speaking, if you don't know which type of line endings a file uses
 
 [SNIP]


Isn't the "line ending" a *file* totally irrelevant here? In the sense that it
is a nothing more than the system's *storage* format?

On my windows machine, the *strings* I manipulate don't have "\r\n" as a
newline, they have '\n'. That's the entire reason there is a "rb" and "r"
option when reading a file.

If you *do* have an "\r\n" in your stream, then either:
* You have an actual a '\r' in your stream, which is then followed by a new
line.
* You are actually erroneously manipulating a binary payload, which should be
of type ubyte[], and should not be using the std.ascii functions with it.

Under these circumstance, and following the unicode definition, I'd say:

return 0x0A <= c && c <= 0x0D;

Is not only correct (for ascii), but any attempt to parse more than 1 character
for this info would be incorrect...

PS: WTF is \u{D A}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 20 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #8 from Dmitry Olshansky <dmitry.olsh gmail.com> 2012-11-20
12:13:48 PST ---
(In reply to comment #7)

 (In reply to comment #1)

 See representation on various systems:
 
 http://en.wikipedia.org/wiki/Newline
 
 In particular:
 On Unix, and Mac OS X: LF (1 char)
 On Windows: CR+LF (2 chars)


 
 (In reply to comment #5)

 Technically speaking, if you don't know which type of line endings a file uses
 
 [SNIP]


 
 Isn't the "line ending" a *file* totally irrelevant here? In the sense that it
 is a nothing more than the system's *storage* format?
 


There is no system's encoding. It died and buried in the same toomb as FTP
ASCII mode long time ago. After all files are transfered in many different ways
expecting someone to transcode line-endings everywhere is plain impossible (you
don't always know the target system). So by the end of day reasonable programs
just deal with all the zoo of them.


 On my windows machine, the *strings* I manipulate don't have "\r\n" as a
 newline, they have '\n'. That's the entire reason there is a "rb" and "r"
 option when reading a file.


And I'd say rb option is a woefully broken thing. In fact putting \n does in
fact store \r\n in this mode. You are far safer with binary mode at least it's
WYSIWG.


 If you *do* have an "\r\n" in your stream, then either:
 * You have an actual a '\r' in your stream, which is then followed by a new
 line.



 Under these circumstance, and following the unicode definition, I'd say:
 
 return 0x0A <= c && c <= 0x0D;
 
 Is not only correct (for ascii), but any attempt to parse more than 1 character
 for this info would be incorrect...


No, no and no. It's the fact of life (or rather the standard) that \r\n is a
single entity. And it can't be parsed other the by looking at two characters
(or rather codepoints).


 
 PS: WTF is \u{D A}


It's 2 dchars : \r\n. That is 0x0D and 0x0A. They are being cute and use
flexible width syntax not the old ones: \uXXXX and \UYYYYYYYY.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 20 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #9 from Jonathan M Davis <jmdavisProg gmx.com> 2012-11-20 12:29:09
PST ---
Any code that only cares about \n or \r\n isn't going to work with a function 
which returns true for both. And any code that doesn't care is going to be
ill-served by a such a function, because the reality is that you need to watch
for \r and \n as individual characters and properly handle the cases when
they're separate as well as when they're apart. And the fact that one of them
is multiple characters generally screws with trying to check for them with a
single function in the middle of iterating. Rather, you have to watch for \r
and \n individually and then figure out whether you're dealing with them singly
or together and act appropriately. A function which returns true for a string
is generally useless for the kind of situations where you'd be checking for
newlines, so I'm highly skeptical that such a function is of any real value.
Instead, you end up with stuff like

for(auto range = str.save; !range.empty; range.popFront())
{
    switch(range.front)
    {
        case '\r':
            auto temp = range.save;
            temp.popFront();
            if(temp.front == '\n')
            {
                range.popFront();
                goto case '\n';
            }
            goto default;
        case '\n':
            //do whatever you do for end of line
            break;
        default:
            //do whatever you do for individual characters
            break;
    }
}

And if you all you want to know is whether a particular string starts or ends
with a newline, then it's easy enough to just do str.startsWith("\n", "\r\n")
or str.endsWith("\n", "\r\n"). That gets uglier when you need to deal with
unicode rather than just \n an \r\n, but then all I believe that you really
need is to add [paraSep], and [lineSep] to the list.

I'm not sure that a function telling you whether a string designates the end of
a line is completely useless, but in pretty much every case that I can see code
caring, such a function wouldn't work very well.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 20 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045


Nick Sabalausky <cbkbbejeap mailinator.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |cbkbbejeap mailinator.com


--- Comment #10 from Nick Sabalausky <cbkbbejeap mailinator.com> 2013-03-26
10:04:56 PDT ---
While an 'isNewline(dchar)' func wouldn't work for reasons already discussed,
what *would* work and be very helpful IMO, is something similar to the
'std.conv.parse(...)' functions. Ie, just like this, but properly templated,
range-ified and UTF8/16-ified:

/// If 'str' starts with a Unix, Windows, Unicode or Mac9 newline, it is
/// removed from 'str' and returned. Otherwise, null is returned.
dstring parseNewline(ref dstring str)
{
    if(str.empty)
        return null;

    dstring ret;

    // Newlines are as defined in:
    // http://www.unicode.org/reports/tr18/#Line_Boundaries
    switch(str[0])
    {
    case '\r':
        if(str.length > 1 && str[1] == '\n')
        {
            ret = str[0..2];
            str = str[2..$];
            break;
        }
        goto case;

    case '\n':
    case '\f':
    case '\v':
    case '\x85':
    case paraSep:
    case lineSep:
        ret = str[0..1];
        str = str[1..$];
        break;

    default:
        ret = null;
        break;
    }

    return ret;
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 26 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9045



--- Comment #11 from Nick Sabalausky <cbkbbejeap mailinator.com> 2013-03-28
17:35:23 PDT ---
*** Issue 8880 has been marked as a duplicate of this issue. ***

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 28 2013