www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 8583] New: AA ushort[dchar] byValue range is corrupted on x86_64

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583

           Summary: AA ushort[dchar] byValue range is corrupted on x86_64
           Product: D
           Version: D2
          Platform: x86_64
        OS/Version: All
            Status: NEW
          Severity: critical
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: dmitry.olsh gmail.com


--- Comment #0 from Dmitry Olshansky <dmitry.olsh gmail.com> 2012-08-24
14:00:21 PDT ---
The following sample outlines the problem:

void main(){
    ushort[dchar] simpleIndices = ['A':437];
    assert(simpleIndices['A'] == 437);
    assert(simpleIndices.byKey.front == 'A');
    assert(simpleIndices.byValue.front == 437); // this fails on x64
    //assert(simpleIndices.byValue.front == 0); //and this passes on x64 WTF??!
}

compiled with -m32 it passes
with -m64 it fails and instead the value is 0. 
Looking through my corrupted data in the wild I assume it may as well be
something else then 0 too. 

Tested both with latest git and vanila 2.060 on OS Linux x64 & Win32.

There is a workaround of using .values which does allocate an array of values.
Still I consider it critical as corrupted value bug is usually hard to spot and
narrow down.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 24 2012
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583


edmccard verizon.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |edmccard verizon.net


--- Comment #1 from edmccard verizon.net 2012-08-24 19:40:31 PDT ---
From my tests, the problem occurs whenever keytype.sizeof <= 8, for any value
type. For example, ushort[string], ushort[uint[3]] and ushort[Foo] are ok,
where

  struct Foo { int x; int y; int z; }

but ushort[uint], ushort[uint[2]] and ushort[Bar] fail, where

  struct Bar { int x; int y; }

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Aug 24 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #2 from edmccard verizon.net 2012-08-24 20:05:06 PDT ---
(In reply to comment #1)
 From my tests, the problem occurs whenever keytype.sizeof <= 8, for any value
 type.

Spoke too soon; the problem is not that simple, or does not depend solely on the size of the key type. For example, string[uint[3]] works but ushort[uint[3]] fails, ushort[uint[4]] works, but both string[uint[5]] and ushort[uint[5]] fail. (replace uint[x] with a struct of the same size, and the same things happen). Sorry for the noise. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Aug 24 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Don <clugdbug yahoo.com.au> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |exetoc gmail.com


--- Comment #3 from Don <clugdbug yahoo.com.au> 2012-11-14 02:46:21 PST ---
*** Issue 7632 has been marked as a duplicate of this issue. ***

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 14 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Simon Harris <pearfalse googlemail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pearfalse googlemail.com


--- Comment #4 from Simon Harris <pearfalse googlemail.com> 2013-01-16 07:37:10
PST ---
Something interesting I found in rt/aaA.d, line 106: aligntsize() on 64-bit
compilation returns a value aligned to 16 bytes. _aaGetX uses this function on
key sizes when creating new elements. So for AAs whose key is <= 8 bytes, the
value is still 16 bytes further on in memory.

You can check this by repeating the same alignment trick on the addresses of
byValue.front:

void testAAType(K, V)(V[K] aa)
{
    writefln("testAAType with key %s, value %s", typeid(K), typeid(V));

    for (auto v = aa.byValue; !v.empty; v.popFront()) { // emulated foreach (v
; aa.byValue)
        V* fp = &v.front();
        writef("Unaligned: <%s>", *fp);

        V* fpaligned = cast(V*) (( (cast(ptrdiff_t) fp) + 15) & ~15); // copy
what aligntsize does on x64

        version(D_LP64) {
            writefln("; Aligned: <%s>", *fp.alignto16());
        }
        else {
            writefln(" (Not testing aligned value on a 32-bit executable)");
        }
    }
}

Looking at the key sizes seems to match the pass/fail results in comment #2,
with one exception: ushort[uint[3]]. Not sure why the alignment trick is needed
here, given that uint[3].sizeof > 8. Maybe DMD aligns
AssociativeArray!(uint[3], ushort).Slot to 4 bytes to save space, and since
_aaGetX uses a completely different type set, alignment info gets lost
somewhere.

Tested with DMD 2.060 and 2.061 on OS X 10.8.2.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 16 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #5 from github-bugzilla puremagic.com 2013-01-21 10:28:09 PST ---
Commits pushed to master at https://github.com/D-Programming-Language/druntime

https://github.com/D-Programming-Language/druntime/commit/e9445dcbbfd7dec9275e78fba6633b689ca06c09
Unify AA value alignment to size_t.sizeof (fixes bug 8583)

Sets the alignment of values in both AA declarations (object and aaA)
to size_t.sizeof bytes, which fixes byValue range iteration on x64.

https://github.com/D-Programming-Language/druntime/commit/9eb3f28e299cb0ce0b2fb22798f4086f008a8a87
Add unit tests for bug 8583; restore 16-byte aligment on x64

This adds unit tests for the alignment fix to Slot, ensuring the two
declarations of an AA slot stay in sync. The 16-byte alignment rule for
x64 has been restored.

https://github.com/D-Programming-Language/druntime/commit/105e12f5b933797a4cae84295a3ebe445e8ae909
Merge pull request #386 from pearfalse/master

Explicitly set AA value alignment in slot to size_t.sizeof bytes (fixes bug
8583)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 21 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583


Alex Rønne Petersen <alex lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |alex lycus.org
         Resolution|                            |FIXED


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 21 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8583



--- Comment #6 from github-bugzilla puremagic.com 2013-02-11 23:12:16 PST ---
Commits pushed to https://github.com/D-Programming-Language/druntime

https://github.com/D-Programming-Language/druntime/commit/e9445dcbbfd7dec9275e78fba6633b689ca06c09
Unify AA value alignment to size_t.sizeof (fixes bug 8583)

https://github.com/D-Programming-Language/druntime/commit/9eb3f28e299cb0ce0b2fb22798f4086f008a8a87
Add unit tests for bug 8583; restore 16-byte aligment on x64

https://github.com/D-Programming-Language/druntime/commit/105e12f5b933797a4cae84295a3ebe445e8ae909
Merge pull request #386 from pearfalse/master

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 11 2013