www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 9131] New: Invalid UTF-8 when using std.algorithm.equal with dstring and string

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9131

           Summary: Invalid UTF-8 when using std.algorithm.equal with
                    dstring and string
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: Mac OS X
            Status: NEW
          Severity: normal
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: hsteoh quickfur.ath.cx


--- Comment #0 from hsteoh quickfur.ath.cx 2012-12-09 21:57:11 PST ---
This is found by the autotester with
https://github.com/D-Programming-Language/phobos/pull/987. As I have no access
to OSX/64, I have no way to reduce the failing case:

auto joiner(RoR)(RoR r)
if (isInputRange!RoR && isInputRange!(ElementType!RoR))
{
    static struct Result
    {
    private:
        RoR _items;
        ElementType!RoR _current;
        void prepare()
        {
            // Skip over empty subranges.
            if (_items.empty) return;
            while (_items.front.empty)
            {
                _items.popFront();
                if (_items.empty) return;
            }
            _current = _items.front;
        }
    public:
        this(RoR r)
        {
            _items = r;
            prepare();
        }
        static if (isInfinite!RoR)
        {
            enum bool empty = false;
        }
        else
        {
             property auto empty()
            {
                return _items.empty;
            }
        }
         property auto ref front()
        {
            assert(!empty);
            return _current.front;
        }
        void popFront()
        {
            assert(!_current.empty);
            _current.popFront();
            if (_current.empty)
            {
                assert(!_items.empty);
                _items.popFront();
                prepare();
            }
        }
        static if (isForwardRange!RoR && isForwardRange!(ElementType!RoR))
        {
             property auto save()
            {
                Result copy;
                copy._items = _items.save;
                copy._current = _current.save;
                return copy;
            }
        }
    }
    return Result(r);
}

unittest
{
    struct TransientRange
    {
        dchar[128] _buf;
        dstring[] _values;
        this(dstring[] values)
        {
            _values = values;
        }
         property bool empty()
        {
            return _values.length == 0;
        }
         property auto front()
        {
            foreach (i; 0 .. _values.front.length)
            {
                _buf[i] = _values[0][i];
            }
            return _buf[0 .. _values.front.length];
        }
        void popFront()
        {
            _values = _values[1 .. $];
        }
    }

    auto rr = TransientRange(["abc"d, "12"d, "def"d, "34"d]);

    // Can't use array() or equal() directly because they fail with transient
    // .front.
    dchar[] result;
    foreach (c; rr.joiner()) {
        result ~= c;
    }

    assert(equal(result, "abc12def34"),
        "Unexpected result: '%s'".format(result));
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 09 2012
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9131



--- Comment #1 from hsteoh quickfur.ath.cx 2012-12-09 21:58:31 PST ---
P.S. The autotester error is:

Invalid UTF sequence: 8ac37b8 - Encoding an invalid code point in UTF-8

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 09 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9131



--- Comment #2 from github-bugzilla puremagic.com 2012-12-17 09:08:56 PST ---
Commit pushed to master at https://github.com/D-Programming-Language/phobos

https://github.com/D-Programming-Language/phobos/commit/3a2377ccd3318ad42999c2657589c0bbd21c58ff
Replace joiner unittest with one that doesn't suffer from issue 9131.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 17 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9131



--- Comment #3 from hsteoh quickfur.ath.cx 2012-12-17 12:54:46 PST ---
Note that the commit does NOT fix this issue, it's for the pull request that
was affected by this bug.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 17 2012
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=9131


hsteoh quickfur.ath.cx changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WONTFIX


--- Comment #4 from hsteoh quickfur.ath.cx 2012-12-20 14:48:48 PST ---
Found the cause of this bug. It's a stack overflow caused by the transient
range struct using a large static array as buffer: it just takes a few copies
of this struct (since structs are passed by value) to overflow the stack, upon
which memory corruption starts happening.

The solution is to make TransientRange._buf a dynamic array whose length is
initialized in this(), instead of a static array.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Dec 20 2012