digitalmars.D.bugs - [Issue 10108] New: Thread local slice to array literal references the same data
- d-bugmail puremagic.com (32/32) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (14/14) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (10/10) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- Sean Kelly (8/24) May 18 2013 This is expected because the global is __gshared and there's therefore n...
- Simen Kjaeraas (17/22) May 18 2013 [snip]
- d-bugmail puremagic.com (17/17) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (7/14) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (15/15) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (17/18) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (18/18) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (82/94) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (27/27) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (13/43) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (16/19) May 17 2013 not what FAILS.
- d-bugmail puremagic.com (7/7) May 17 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (8/18) May 18 2013 I dont know anything about relocation magic. But, as I understand you, w...
- d-bugmail puremagic.com (32/32) May 21 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
- d-bugmail puremagic.com (8/20) May 21 2013 http://d.puremagic.com/issues/show_bug.cgi?id=10108
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Summary: Thread local slice to array literal references the same data Product: D Version: unspecified Platform: All OS/Version: All Status: NEW Severity: normal Priority: P2 Component: DMD AssignedTo: nobody puremagic.com ReportedBy: code dawg.eu cat > bug.d << CODE import core.thread; int[] arr = [1,2,3]; __gshared int* thrPtr; void main() { auto thr = new Thread({thrPtr = arr.ptr;}); thr.start(); thr.join(); assert(arr.ptr !is thrPtr); } CODE ---- Only the `arr` variable is store in TLS, but `arr.ptr` references the literal which is in the shared data segment. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Alex Rønne Petersen <alex lycus.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |alex lycus.org CEST --- I'm not sure I understand why this is a bug. Can you elaborate? As far as I know, being able to access TLS data from one thread in another thread through pointers passed via globals is a feature (it's even one I rely on). But maybe I'm misunderstanding this. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 When a thread local variable is a reference type to modifiable data, we must make sure that it is initialized uniquely. This is what the current implementation does which results in hidden sharing. __gshared int[] gArr = [1,2,3]; int[] arr = gArr; -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
This is expected because the global is __gshared and there's therefore no ty= pe protection from doing this. If you want safe sharing, make the global sha= red.=20 On May 17, 2013, at 9:38 AM, d-bugmail puremagic.com wrote:http://d.puremagic.com/issues/show_bug.cgi?id=3D10108 =20 =20 =20-When a thread local variable is a reference type to modifiable data, we mu=stmake sure that it is initialized uniquely. =20 This is what the current implementation does which results in hidden shari=ng.=20 __gshared int[] gArr =3D [1,2,3]; int[] arr =3D gArr; =20 --=20 Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=3Dema=il------- You are receiving this mail because: -------
May 18 2013
On Sat, 18 May 2013 16:59:53 +0200, Sean Kelly <sean invisibleduck.org> wrote:This is expected because the global is __gshared and there's therefore no type protection from doing this. If you want safe sharing, make the global shared.[snip]Uhm, you are aware this is the Bugzilla newsgroup, right? Please reply in Bugzilla. Also, that's exactly what we're saying. This code has problems: import core.thread; int[] arr = [1,2,3]; void main() { new Thread({arr[0] = 3;}); assert(arr[0] == 1); } And they are caused by the data inside arr being shared. This is unsafe, unexpected (due to safety of other globals) and, we feel, goes against the D ethos. -- Simen__gshared int[] gArr = [1,2,3]; int[] arr = gArr;
May 18 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Steven Schveighoffer <schveiguy yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |schveiguy yahoo.com 10:47:13 PDT --- Be careful with your code snippets, unless you are more explicit about the error reporting. Asserts are generally included to show what currently PASSES, not what FAILS. I was about to mark this as invalid, because the code SHOULD pass, but you are correct in that it fails. Your code should say: assert(arr.ptr !is thrPtr); // FAILS!!! or assert(arr.ptr is thrPtr); -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 CEST ---When a thread local variable is a reference type to modifiable data, we must make sure that it is initialized uniquely. This is what the current implementation does which results in hidden sharing. __gshared int[] gArr = [1,2,3]; int[] arr = gArr;Ah, I understand. Carry on. :) -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Walter Bright <bugzilla digitalmars.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED CC| |bugzilla digitalmars.com Resolution| |INVALID 11:25:36 PDT --- The __gshared storage class is an end run around the type system. That's why it has the __ prefix. So, yes, you can use it to get multithreaded "local" access to global data without a compiler error. This is as designed - it's a bug in the code example. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Steven Schveighoffer <schveiguy yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|INVALID | 11:29:07 PDT ---This is as designed - it's a bug in the code example.No, it's not. It's a bug in the code generation. The __gshared variable just demonstrates the bug. What is happening is that each thread-local instance of arr is getting a pointer to the SAME data. The assert should have been written differently. as written, it fails, but this is not noted. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Simen Kjaeraas <simen.kjaras gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |simen.kjaras gmail.com PDT --- Simplified example without __gshared: import core.thread; int[] arr = [1,2,3]; void main( ) { int* p = arr.ptr; auto thr = new Thread({assert(arr.ptr == p);}); // Should have failed. thr.start(); thr.join(); } -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Igor Stepanov <wazar.leollone yahoo.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wazar.leollone yahoo.com 13:57:50 PDT ---Simplified example without __gshared: import core.thread; int[] arr = [1,2,3]; void main( ) { int* p = arr.ptr; auto thr = new Thread({assert(arr.ptr == p);}); // Should have failed. thr.start(); thr.join(); }In other words implicit "thread local" modifier is not transitive. Yes, all threads have a local copy of "arr symbol" and &arr differs in different threads. But all of this "local" symbols points to single array object. The problem lies deeper than it seems. struct Foo { int[] arr; } Foo[] getFooArr() { Foo[] ret; foreach(i; 1 .. 10) { Foo cur; foreach(j; 1 .. 5) { cur.arr ~= j; } ret ~= cur; } return ret; } Foo[] arr = getFooArr(); Now foo points to the arr, each members of it points to array literal. And if we want to create Foo[] is true TLS object, we must to set, that 1. arr must point to tls object (arr.ptr must be thread_tls_start+arr_tls_offset) 2. for each i: arr[i].arr must point to tls object. In other words If compiler see ptr dereference expression (with * or []) it must know, is this ptr is TLS. If it is TLS compiler must add to it value thread_tls_start, otherwise - use it value as is. This functional can be provided, if we declare transitive threadlocal storage class (like shared) and implement special reference behaviour. (e.g. dereference, casting and other.) for example: theradlocal int[][] tls = [[1,2],[3,4],[5,6]]; int* getNthMthElemPointer(theradlocal int[][] a, int n, int m) { return &ptr[n][m]; //implicit cast to non-tls pointer. returned value points to elem it function caller thread } void main() { int* p1 = getNthMthElemPointer(tls, 1, 1); theradlocal int* p2 = &tls[1][1]; void threadFunc(int num)() { writeln(num, " shared ptr: ", p1); writeln(num, " thread local ptr: ", cast(int*)p2); } auto thr1 = new Thread(&threadFunc!1); thr1.start(); auto thr2 = new Thread(&threadFunc!2); thr2.start(); thr1.join(); thr2.join(); } therads will print same "shared ptr" value but different "thread local ptr" However this future is disharmonious with language design I think. Other way: disallow all of tls static initialized values. shared int[] a = [1,2,3]; //OK _gshared int[] b = [1,2,3]; //OK const int[] c = [1,2,3]; //OK int[] d = [1,2,3]; //Disallowed int[] e;//OK static this() { e = [1,2,3]; //If e value will be allocated in heap this code doesn't break type system. } The same applies to classes, pointers and associative arrays in future. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 PDT --- I see. Once again, simplified: import core.thread; struct Foo { int[] arr; } Foo[] arr = [Foo([1,2,3])]; // Should have failed? (1) void main( ) { int* p = arr[0].arr.ptr; auto thr = new Thread({assert(arr[0].arr.ptr == p);}); // Should have failed. (2) thr.start(); thr.join(); } In this case, for the assert to fail, we'd have to deep-dup the array (COW might make that unnecessary, but that's beside the point). This is in a way related to the issue of array literals being mutable, in that it is an example of the compiler erroneously assuming some state may be shared when in fact it shouldn't. I contend that (1) above should simply not compile. It should be required to be placed in a module constructor instead. A case can be made that the compiler should automagically place it in a module constructor for you, but I am not of that opinion. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 16:25:34 PDT ---I see. Once again, simplified: import core.thread; struct Foo { int[] arr; } Foo[] arr = [Foo([1,2,3])]; // Should have failed? (1) void main( ) { int* p = arr[0].arr.ptr; auto thr = new Thread({assert(arr[0].arr.ptr == p);}); // Should have failed. (2) thr.start(); thr.join(); } In this case, for the assert to fail, we'd have to deep-dup the array (COW might make that unnecessary, but that's beside the point). This is in a way related to the issue of array literals being mutable, in that it is an example of the compiler erroneously assuming some state may be shared when in fact it shouldn't. I contend that (1) above should simply not compile. It should be required to be placed in a module constructor instead.Yep int[] x = [1,2,3]; should not be compiled, but shared int[] x = [1,2,3]; //OK const int[] x = [1,2,3]; //OK, because const is global scope == immutable immutable int[] x = [1,2,3]; //OK __gshared int[] x = [1,2,3]; //SameA case can be made that the compiler should automagically place it in a module constructor for you, but I am not of that opinion.I agree, this is not D way, I think. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108Asserts are generally included to show what currently PASSES,not what FAILS. OK, I always write unittests that should pass but I'll be more explicit.In other words implicit "thread local" modifier is not transitive.It's not intended to be transitive, it is a storage class, not a type qualifier. Variables with thread local storage may reference any other data (__gshared, shared, stack, heap) and vice versa.int[] x = [1,2,3]; // should not be compiledIt would be trivial to fix. As the initializer for static data must be a compile time constant we'd just need to store this constant in TLS instead of the data segment. The problem is that ELF has no TLS relocations for data, i.e. we'd need a dynamic initalizer that sets arr.ptr to the TLS data. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 The simple fix is to only allow value types to have TLS initalizers and require static this() for everything else. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 17 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 00:46:39 PDT ---I dont know anything about relocation magic. But, as I understand you, we cannot to use it;int[] x = [1,2,3]; // should not be compiledIt would be trivial to fix. As the initializer for static data must be a compile time constant we'd just need to store this constant in TLS instead of the data segment. The problem is that ELF has no TLS relocations for data, i.e. we'd need a dynamic initalizer that sets arr.ptr to the TLS data.The simple fix is to only allow value types to have TLS initalizers and require static this() for everything else.Yes. This is all we need I think. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 18 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 Sean Kelly <sean invisibleduck.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sean invisibleduck.org PDT --- So I thought I understood this: import core.thread; int[] arr = [1,2,3].dup; void main() { auto t = new Thread({arr[0] = 3;}); t.start(); t.join(); assert(arr[0] == 1); } It looks like we have a thread-local reference "arr" to a __gshared array of int, so I would expect the assert to fail. Except: import core.thread; int[] arr = [1,2,3].dup; void main() { auto t = new Thread({arr[0] = 3;}); t.start(); t.join(); assert(arr[0] == 1); } The .dup should fix the problem, as now each thread gets its own copy of the array, right? But the assert still fails. I suppose I should check the ASM, but the codegen seems kind of broken here. Is a __gshared label being inferred for arr because it's statically slicing __gshared data? -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 21 2013
http://d.puremagic.com/issues/show_bug.cgi?id=10108 12:41:56 PDT ---import core.thread; int[] arr = [1,2,3].dup; void main() { auto t = new Thread({arr[0] = 3;}); t.start(); t.join(); assert(arr[0] == 1); } The .dup should fix the problem, as now each thread gets its own copy of the array, right?It's still being done at compile time, so no. It basically creates a copy at compile time, then stores that in the data segment instead of the original. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
May 21 2013