www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Does anyone understand how to use "shared" types with concurrency

reply Arek <arychlinski gmail.com> writes:
I have the folowing problem:
I like to envelope the class object in struct to control the 
destruction moment and then send this object to another 
thread/fiber (or task, cause I use vibe-d).

I can't find any method to make it working. Any ideas?

dmd (version 075) gives so stupid results, I belive it's broken 
(I've created the issue 
https://issues.dlang.org/show_bug.cgi?id=17749 )

ldc2 returns some errors:

.../import/std/variant.d(610,27): Error: function 
core.stdc.string.memcpy (void* s1, const(void*) s2, ulong n) is 
not callable using argument types (ubyte[32]*, 
shared(Env!(shared(A)))*, ulong)
.../import/std/conv.d(4186,9): Error: static assert  "Cannot 
emplace a Env!(shared(A)) because Env!(shared(A)).this(this) is 
annotated with  disable."
.../import/std/conv.d(4198,24):        instantiated from here: 
emplaceRef!(Env!(shared(A)), Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(301,35):        instantiated from here: 
emplaceRef!(Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(630,21):        instantiated from here: 
handler!(shared(Env!(shared(A))))
.../import/std/variant.d(544,17):        ... (5 instantiations, 
-v to show) ...
../../../.dub/packages/vibe-core-1.1.1/vibe-core/source/vibe/core/con
urrency.d(1223,64):        instantiated from here:
send!(shared(Env!(shared(A))))
app.d(74,7):        instantiated from here: 
send!(shared(Env!(shared(A))))



The code may be complied like this: dub --single --compiler=ldc2 
./app.d   (assuming it's saved in app.d file).


/+
dub.sdl:
name "simple"
dependency "vibe-core" version="~>1.1.0"
+/
import std.stdio;
import std.format;

import vibe.core.core;
import vibe.core.task;
import vibe.core.concurrency;

// simple class with destructor
shared class A
{
	int i;
	this(int i)
	{
		this.i = i;
	}

	~this()
	{
		writeln("destruct ", i);
	}

}

shared struct Env(T) if (is(T == class) && is(T == shared))
{
	T obj;

	this(T o)
	{
		obj = obj;
	}

	this(this)
	{
		// some magic here
	}

	auto opAssign(shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	auto opAssign(ref shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	~this()
	{
		if (obj !is null)
			destroy(obj);
		obj = null;
	}
}

void main()
{
	auto consumer = runTask({
		auto got = receiveOnly!(shared Env!(shared A))();
		writeln(typeof(got).stringof);
	});

	auto producer = runTask({
		auto a = new shared A(1);
		shared b = shared Env!(shared A)(a);
		writeln(typeof(b).stringof);
		send(consumer, b);
	});
	runApplication();
}
Aug 12
next sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the 
 destruction moment and then send this object to another 
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them.
Aug 12
parent reply Arek <arychlinski gmail.com> writes:
On Sunday, 13 August 2017 at 02:50:13 UTC, crimaniak wrote:
 On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the 
 destruction moment and then send this object to another 
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
I tried it too some time ago. Then I read Alexandrescu book and realized that the authors of the language do not want anyone to do this. Long story short, just plan your application so that each complex object is monitored by only one thread/task, and pass not objects, but messages (immutable structs) about what to do with them.
Yeah, I've read this. But conurrency.send cannot pass immutable object. The same story with Unique. What means, that if I have complex objects which I want to transfer between threads I must keep them somewhere in global memory and share among threads. I can use send/receive only to signal availability of their existance. What I need is kind of "move" operation beetwen threads. More over, "shared" looks rather like unfinished concept. It is really difficult to create proper struct (with postblit and destructor) working as shared object. I even have no clue what is exact semantics of "shared" types. Language specification is a little laconic. Anyway, _gshared looks very promising, so I will try to work out any approach. Thanks for answer. Arek
Aug 13
parent reply crimaniak <crimaniak gmail.com> writes:
On Sunday, 13 August 2017 at 11:35:05 UTC, Arek wrote:
 Yeah, I've read this. But conurrency.send cannot pass immutable 
 object. The same story with Unique.
Sorry, read this as 'efficient immutable'.
 More over, "shared" looks rather like unfinished concept.
Yes, exactly.
 Anyway, _gshared looks very promising, so I will try to work 
 out any approach.
More of this, I think, you can't avoid __gshared for any complex work. Even mutexes from Phobos doesn't support shared, so I had to 'cowboy with __gshared' when implementing my site engine.
Aug 13
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Sunday, August 13, 2017 16:40:03 crimaniak via Digitalmars-d-learn wrote:
 More of this, I think, you can't avoid __gshared for any complex
 work. Even mutexes from Phobos doesn't support shared, so I had
 to 'cowboy with __gshared' when implementing my site engine.
The way to handle shared is to protect the section of code that's using the shared object with either a mutex or synchronized block, and then you cast away shared from the object within that section and operate on it as thread-local. When you're done, you make sure that you don't have any thread-local references to the data, and you release the mutex or exit the synchronized block. e.g. something like shared T sharedObj = getSharedObj(); synchronized(mutex) { T nonSharedObj = cast(T)sharedObject // do stuff... // make sure that no references to nonSharedObj have escaped } // now, there's just the shared version of the object And no, this isn't ideal, but the only semi-decent solution that's been proposed that safely casts away shared for you is synchronized classes, which Andrei describes in TDPL but have never been implemented. And because they can only safely strip off the outermost layer of shared, they're of questionable usefulness anyway. Ultimately, even with synchronized classes, in many situations, the programmer is going to have to carefully cast away shared to operate on the object within a protected context. Now, the fact that the mutex objects don't handle shared correctly is another issue entirely. Having to cast away shared from mutexes is dumb, because you're obviously not going to be protecting them with a mutex, and their operations have to be atomic anyway for them to do what they do. So, that definitely needs to be fixed. However, I believe that it _has_ been fixed in master, and it might have made it into a release now, but I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it should be. In general though, the idea is that you simply don't operate on shared objects except via atomic operations. Otherwise, you risk concurrency problems. And really, this is the same as what you'd do in C/C++, except that in C/C++, it doesn't catch you when you operate on an object that's shared across threads with non-atomic operations (because the object isn't explicitly typed as shared), and you don't have to cast away shared to do non-atomic operations. So, having to cast away shared is the price of getting the protection against accidentally using non-atomic operations on a shared object as well as the price we pay to be able to have the type system distinguish between shared and thread-local objects so that it's able to optimize based on the knowledge that an object is thread-local. Ultimately though, you're doing the same thing that you'd do in C++ if you're handling concurrency safely. You just have to explicitly mark stuff as shared and carefully cast away shared in certain, protected contexts. Using __gshared in extern(D) code is just asking for it, because then you have an object that the compiler thinks is thread-local but isn't, and you risk subtle and nasty bugs as a result. __gshared is only intended for binding to extern(C), global variables. To an extent, you can get away with using it with extern(D) variables, but that's not its intended purpose, and you risk running afoul of the compiler and what it chooses to do based on the assumption that the object is thread-local. - Jonathan M Davis
Aug 13
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
 [snip]

 Now, the fact that the mutex objects don't handle shared 
 correctly is another issue entirely. Having to cast away shared 
 from mutexes is dumb, because you're obviously not going to be 
 protecting them with a mutex, and their operations have to be 
 atomic anyway for them to do what they do. So, that definitely 
 needs to be fixed. However, I believe that it _has_ been fixed 
 in master, and it might have made it into a release now, but 
 I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable 
 as shared like it should be.
My fixes for shared(Mutex) - https://github.com/dlang/druntime/pull/1728 - are part of the DMD v2.074.0 release, and should propagate to LDC with their 1.4 release respectively. GDC master now at druntime/phobos 2.074.1 - https://github.com/D-Programming-GDC/GDC/pull/539 and they're in the process of moving to 2.075.0 - https://github.com/D-Programming-GDC/GDC/pull/542. Unfortunately, I forgot to write a changelog entry when I was working on the PR and later didn't have time to do so, before the 2.074.0 release. So shared (Mutex) should be in good shape now. I should supplement the ddoc unittest to address the transitive you issues you talked about. Currently it only shows synchronized use of a plain value type without indirections, which is uninteresting. The example I'm thinking about is casting with HeadUnsharedOf / TailSharedOf a pointer to a singly-linked list, because with pointers you could show the shared(Node*) -> shared(Node)* transition in the type system, which unfortunately can't be demonstrated with implicitly reference types (classes). Speaking of TailSharedOf, Jonathan could take a look at this PR: https://github.com/dlang/druntime/pull/1605? I and Andrei already approved it, but I'd like to get a third opinion since this is a breaking change and Martin seems be too busy the moment.
Aug 14
prev sibling next sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:
 On Sunday, August 13, 2017 16:40:03 crimaniak via 
 Digitalmars-d-learn wrote:
 More of this, I think, you can't avoid __gshared for any 
 complex work. Even mutexes from Phobos doesn't support shared, 
 so I had to 'cowboy with __gshared' when implementing my site 
 engine.
The way to handle shared is to protect the section of code that's using the shared object with either a mutex or synchronized block, and then you cast away shared from the object within that section and operate on it as thread-local. When you're done, you make sure that you don't have any thread-local references to the data, and you release the mutex or exit the synchronized block. e.g. something like shared T sharedObj = getSharedObj(); synchronized(mutex) { T nonSharedObj = cast(T)sharedObject // do stuff... // make sure that no references to nonSharedObj have escaped } // now, there's just the shared version of the object
Yeah, and this is what i'm doing now (more or less). To be more precise, I don't even want to synchronize access to the shared resource between the threads. I just want to move the object from one thread to another. Of course I could copy the local object, but my obcjecs has indirect references to others, forming kind of tree. I like the idea of channels in Go. I've tried to get something similiar with send/receive. In this case, object could be also immutable, because once there are created (in the deserialization process) they will no be modified. I just have to emit them into another task (to be honest, I use fiber, so it's not even another thread, and they will not be accessed in parallel). But all this language protections makes the issue unexpectedly complicated.
 And no, this isn't ideal, but the only semi-decent solution 
 that's been proposed that safely casts away shared for you is 
 synchronized classes, which Andrei describes in TDPL but have 
 never been implemented. And because they can only safely strip 
 off the outermost layer of shared, they're of questionable 
 usefulness anyway. Ultimately, even with synchronized classes, 
 in many situations, the programmer is going to have to 
 carefully cast away shared to operate on the object within a 
 protected context.

 Now, the fact that the mutex objects don't handle shared 
 correctly is another issue entirely. Having to cast away shared 
 from mutexes is dumb, because you're obviously not going to be 
 protecting them with a mutex, and their operations have to be 
 atomic anyway for them to do what they do. So, that definitely 
 needs to be fixed. However, I believe that it _has_ been fixed 
 in master, and it might have made it into a release now, but 
 I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable 
 as shared like it should be.

 In general though, the idea is that you simply don't operate on 
 shared objects except via atomic operations. Otherwise, you 
 risk concurrency problems. And really, this is the same as what 
 you'd do in C/C++, except that in C/C++, it doesn't catch you 
 when you operate on an object that's shared across threads with 
 non-atomic operations (because the object isn't explicitly 
 typed as shared), and you don't have to cast away shared to do 
 non-atomic operations. So, having to cast away shared is the 
 price of getting the protection against accidentally using 
 non-atomic operations on a shared object as well as the price 
 we pay to be able to have the type system distinguish between 
 shared and thread-local objects so that it's able to optimize 
 based on the knowledge that an object is thread-local. 
 Ultimately though, you're doing the same thing that you'd do in 
 C++ if you're handling concurrency safely. You just have to 
 explicitly mark stuff as shared and carefully cast away shared 
 in certain, protected contexts.

 Using __gshared in extern(D) code is just asking for it, 
 because then you have an object that the compiler thinks is 
 thread-local but isn't, and you risk subtle and nasty bugs as a 
 result. __gshared is only intended for binding to extern(C), 
 global variables. To an extent, you can get away with using it 
 with extern(D) variables, but that's not its intended purpose, 
 and you risk running afoul of the compiler and what it chooses 
 to do based on the assumption that the object is thread-local.

 - Jonathan M Davis
Thanks for the explanation! It would be good to have a comprehensive article on this subject. Arek
Aug 14
prev sibling next sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:

 The way to handle shared is to protect the section of code 
 that's using the shared object with either a mutex or 
 synchronized block, and then you cast away shared from the 
 object within that section and operate on it as thread-local. 
 When you're done, you make sure that you don't have any 
 thread-local references to the data, and you release the mutex 
 or exit the synchronized block. e.g. something like

 shared T sharedObj = getSharedObj();

 synchronized(mutex)
 {
     T nonSharedObj = cast(T)sharedObject

     // do stuff...

     // make sure that no references to nonSharedObj have escaped
 }
Casting objects just to work with it every time is so bad style for me that I even didn't consider such possibility. In fact, I did make something like this but with __gshared both object and mutex. I think I need to review this part of site engine and my programming habits too. I wonder if it possible and usable to make some template to support this pattern, where we give mutex(es), shared object(s) and delegate to operate with objects as non-shared.
Aug 16
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote:
 I wonder if it possible and usable to make some template to 
 support this pattern, where we give mutex(es), shared object(s) 
 and delegate to operate with objects as non-shared.
https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it helps, don't forget that it's a casted shared object.
Aug 17
parent crimaniak <crimaniak gmail.com> writes:
On Thursday, 17 August 2017 at 13:09:29 UTC, Kagamin wrote:
 On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote:
 I wonder if it possible and usable to make some template to 
 support this pattern, where we give mutex(es), shared 
 object(s) and delegate to operate with objects as non-shared.
https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it helps, don't forget that it's a casted shared object.
Yes, something like this. In general, I thought about the possibility of using several shared objects in this block but then realized that everything can be reduced to the case of one object.
Aug 20
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:

 And no, this isn't ideal, but the only semi-decent solution 
 that's been proposed that safely casts away shared for you is 
 synchronized classes, which Andrei describes in TDPL but have 
 never been implemented.
After reading this I did some experiment to understand the situation better. I make a simple class and unittest: // dmd sync1.d -unittest -main unittest { import std.stdio; synchronized class A { private int a; void inc() { ++a; } int get(){ return a;} } shared A a; for(int i=0; i<100; ++i) a.inc(); writeln(a.get); } Oops! Deprecation: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(this.a, 1) instead. Why use atomic operations if the class already synchronized? Well.. ... import core.atomic: atomicOp; ... // ++a; // Deprecation: read-modify-write operations are not allowed for shared variables. Use core.atomic.atomicOp!"+="(this.a, 1) instead. atomicOp!"+="(this.a, 1); ... ok, works. But it works by the way as if synchronized just makes all methods shared, but does not provide the object methods with a mutex lock, as Java does. Am I right here? And what preventing to implement it right, lack of manpower or some ideologic problems?
Aug 20
parent crimaniak <crimaniak gmail.com> writes:
On Monday, 21 August 2017 at 02:17:57 UTC, crimaniak wrote:
...
     shared A a;
... Sorry, accidental delete, read this as shared A a = new shared(A);
Aug 20
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. - Jonathan M Davis
Aug 13
next sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 03:40:26 UTC, Jonathan M Davis wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally. - Jonathan M Davis
That's what I suspected. Thanks for the confirmation. I was not sure if I was doing everything right. Arek
Aug 14
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared. -Steve
Aug 14
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 19:22:23 UTC, Steven Schveighoffer 
wrote:
 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn 
 wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared. -Steve
First of all, I'm not native English speaker, so forgive me possible misunderstanding In my opinion the whole problem of 'shared' is that when I use send/receive operation (I believe it's should be something similar to go's channels or erlang messages) I do not want to share the memory so any "shared" concept is useless. What I really would like to get is possibility to make: // in first thread Unique!MyObject msg = new MyObject() send(msg.release); // in other thread: auto msg = receiveOnly!(Unique!MyObject)(); My object disappears in "send" operation and its new instance is constructed on receive. If I can ensure the uniqueness of the object, there is no need to "share" it or synchronize the access. But now such operation is impossible and the compiler forces me to cope with shared if MyObject has any references to other areas of memory. Anyway, thanks for all replies. Arek
Aug 14
parent Kagamin <spam here.lot> writes:
On Monday, 14 August 2017 at 20:13:28 UTC, Arek wrote:
 If I can ensure the uniqueness of the object, there is no need 
 to "share" it or synchronize the access.
You use manually managed multithreading, that's why you need shared. And because compiler can't verify uniqueness, you are requested to do it manually by casting.
Aug 15
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
learn wrote:
 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn 
wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared.
What's incorrect about it? It's a longstanding issue that because Variant can't contain shared data, send and receive do not work with shared. You can send and receive mutable data with no indirections, and AFAIK, you can send immutable data (though the OP is apparently having problems with that, so I guess that that doesn't work completely, though I definitely recall doing so previously), but you get compilation errors if you try to send shared data. For instance, this code import std.concurrency; void main() { static void func(Tid parent) { auto received = receiveOnly!(shared int[string]); } shared int[string] aa; auto tid = spawn(&func, thisTid); send(tid, aa); } will fail to compile with /usr/local/include/dmd/std/variant.d(625): Error: function core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong n) is not callable using argument types (ubyte[32]*, shared(int[string])*, ulong) /usr/local/include/dmd/std/variant.d(625): Error: function core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong n) is not callable using argument types (ubyte[32]*, shared(int)*, ulong) /usr/local/include/dmd/std/variant.d(424): Error: template instance std.variant.VariantN!32LU.VariantN.opAssign!(shared(int)) error instantiating /usr/local/include/dmd/std/variant.d(645): instantiated from here: handler!(shared(int[string])) /usr/local/include/dmd/std/variant.d(559): instantiated from here: opAssign!(shared(int[string])) /usr/local/include/dmd/std/concurrency.d(101): instantiated from here: __ctor!(shared(int[string])) /usr/local/include/dmd/std/concurrency.d(606): ... (2 instantiations, -v to show) ... /usr/local/include/dmd/std/concurrency.d(576): instantiated from here: _send!(shared(int[string])) q.d(13): instantiated from here: send!(shared(int[string])) whereas if you use something like int, it has no problem. It won't even compile if you give it a shared int, and that arguably should just be converting to and from int, since shared int has no indirections. There are at least two bugs reported on the issue: https://issues.dlang.org/show_bug.cgi?id=13262 https://issues.dlang.org/show_bug.cgi?id=14893 I suspect that the solution involves making it so that std.concurrency uses something other than Variant to store the data being sent, but regardless, no one has fixed the issue yet. So, while send and receive work fine for simple message, they're fairly crippled when it comes to complex data and have been for their entire existence AFAIK. - Jonathan M Davis
Aug 14
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 21:27:48 UTC, Jonathan M Davis wrote:
 On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via 
 Digitalmars-d- learn wrote:
 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn 
 wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn
wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared.
What's incorrect about it? It's a longstanding issue that because Variant can't contain shared data, send and receive do not work with shared. You can send and receive mutable data with no indirections, and AFAIK, you can send immutable data (though the OP is apparently having problems with that, so I guess that that doesn't work completely, though I definitely recall doing so previously), but you get compilation errors if you try to send shared data. For instance, this code import std.concurrency; void main() { static void func(Tid parent) { auto received = receiveOnly!(shared int[string]); } shared int[string] aa; auto tid = spawn(&func, thisTid); send(tid, aa); }
I've found some simple workaround for this problem: import std.stdio; import std.concurrency; struct Envelope(T) if (is(T == class)) // for simplicity of this example, only classes { shared(T)[] obj; this(shared T o) { this.obj = [o]; } T get() property nothrow nogc { return cast() obj[0]; } } class A { } void consumer() { auto r = receiveOnly!(Envelope!(A))(); writeln("Got: ", typeof(r).stringof); } void main() { auto cons = spawn(&consumer); auto o = Envelope!A(new A()); send(cons, o); } Shared object can be encapsulated in the array. In case of other (non-class) types the pointer can be used, and get() should return ref to the pointed object (after stripping off the shared qualifier). send() could encapsulate itself shared objectes. Arek
Aug 14
parent reply Kagamin <spam here.lot> writes:
On Monday, 14 August 2017 at 22:22:58 UTC, Arek wrote:
 I've found some simple workaround for this problem:

 import std.stdio;
 import std.concurrency;

 struct Envelope(T) if (is(T == class)) // for simplicity of 
 this example, only classes
 {
 	shared(T)[] obj;

 	this(shared T o)
 	{
 		this.obj = [o];
 	}

 	T get()  property nothrow  nogc
 	{
 		return cast() obj[0];
 	}
 }

 class A
 {

 }

 void consumer()
 {
 	auto r = receiveOnly!(Envelope!(A))();
 	writeln("Got: ", typeof(r).stringof);
 }

 void main()
 {
 	auto cons = spawn(&consumer);
 	auto o = Envelope!A(new A());
 	send(cons, o);
 }

 Shared object can be encapsulated in the array. In case of 
 other (non-class) types the pointer can be used, and get() 
 should return ref to the pointed object (after stripping off 
 the shared qualifier).
Rather like this: struct Sendable(T) { shared T o; alias o this; } import std.concurrency; class A { int method() shared; } void consumer() { shared A a = receiveOnly!(Sendable!(A))(); } void producer() { auto cons = spawn(&consumer); shared A a = new shared A(); send(cons, Sendable!A(a)); }
Aug 15
parent reply Kagamin <spam here.lot> writes:
Well, no wrapper is actually needed here:

class A
{
	int method() shared;
}

void consumer()
{
	shared a = receiveOnly!(shared A)();
}

void producer()
{
	auto cons = spawn(&consumer);
	send(cons, new shared A());
}
Aug 15
parent reply Arek <arychlinski gmail.com> writes:
On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
 Well, no wrapper is actually needed here:

 class A
 {
 	int method() shared;
 }

 void consumer()
 {
 	shared a = receiveOnly!(shared A)();
 }

 void producer()
 {
 	auto cons = spawn(&consumer);
 	send(cons, new shared A());
 }
Yes, but this doesn't compile: import std.stdio; import std.concurrency; struct A { int t; int r; int method() shared { return 0; } } void consumer() { shared a = receiveOnly!(shared A)(); } void main() { auto cons = spawn(&consumer); send(cons, shared A()); } This very simple code also doesn't compile: shared struct S { int i; ~this() { } } void main() { shared s = shared S(); } In general, shared structs with postblit and destructor make problems. Arek
Aug 15
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/15/17 5:27 PM, Arek wrote:
 On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
 Well, no wrapper is actually needed here:

 class A
 {
     int method() shared;
 }

 void consumer()
 {
     shared a = receiveOnly!(shared A)();
 }

 void producer()
 {
     auto cons = spawn(&consumer);
     send(cons, new shared A());
 }
Yes, but this doesn't compile: import std.stdio; import std.concurrency; struct A { int t; int r; int method() shared { return 0; } } void consumer() { shared a = receiveOnly!(shared A)(); } void main() { auto cons = spawn(&consumer); send(cons, shared A()); }
The issue is that send cannot handle shared value types due to a bug in the implementation. In essence, there is no reason to send a shared A -- it's an unrelated copy and not actually shared. You can send a shared reference to an A just fine: static shared A a; send(tid, &a); // works You *should* be able to send a shared(A). There is no reason to disallow it. But again, it's not super useful.
 This very simple code also doesn't compile:
 
 shared struct S
 {
      int i;
 
      ~this()
      {
      }
 }
 
 void main()
 {
      shared s = shared S();
 }
 
 In general, shared structs with postblit and destructor make problems.
postblit should work. It does look like destructors are a problem. It appears the compiler attempts to call the destructor while unshared. -Steve
Aug 15
parent Arek <arychlinski gmail.com> writes:
On Tuesday, 15 August 2017 at 21:54:23 UTC, Steven Schveighoffer 
wrote:
 On 8/15/17 5:27 PM, Arek wrote:
 On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:
 Well, no wrapper is actually needed here:
[...]
 The issue is that send cannot handle shared value types due to 
 a bug in the implementation. In essence, there is no reason to 
 send a shared A -- it's an unrelated copy and not actually 
 shared.

 You can send a shared reference to an A just fine:

 static shared A a;

 send(tid, &a); // works

 You *should* be able to send a shared(A). There is no reason to 
 disallow it. But again, it's not super useful.
Yes, I absolutely agree. The object may be shared due to other requirements and there is no reason to reject 'share' qualifier what I want to send/receive it.
 This very simple code also doesn't compile:
 
 shared struct S
 {
      int i;
 
      ~this()
      {
      }
 }
 
 void main()
 {
      shared s = shared S();
 }
 
 In general, shared structs with postblit and destructor make 
 problems.
postblit should work. It does look like destructors are a problem. It appears the compiler attempts to call the destructor while unshared. -Steve
But I have no idea how and when the object loses its shared nature. Yes, it looks like a bug in dmd. ldc 1.3 doesn't complain in this case. Arek
Aug 16
prev sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 15 August 2017 at 21:27:49 UTC, Arek wrote:
 Yes, but this doesn't compile:

 import std.stdio;
 import std.concurrency;

 struct A
 {
 	int t;
 	int r;
 	int method() shared
 	{
 		return 0;
 	}
 }

 void consumer()
 {
 	shared a = receiveOnly!(shared A)();
 }

 void main()
 {
 	auto cons = spawn(&consumer);
 	send(cons, shared A());
 }
AIU you use struct as a Unique-like wrapper for shared class object. Extract the object and send it, then wrap again on arrival.
Aug 16
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/14/17 5:27 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
 On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
 learn wrote:
 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:
 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn
wrote:
 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?
Unfortunately, send and receive do not currently work with shared because of issues with Variant, which they use internally.
This can't be a correct statement. This is the whole point of shared.
What's incorrect about it? It's a longstanding issue that because Variant can't contain shared data, send and receive do not work with shared.
The implementation details aren't important. From the documentation (and no, this documentation is not wrong): From spawn: args must not have unshared aliasing. In other words, all arguments to fn must either be shared or immutable or have no pointer indirection. This is necessary for enforcing isolation among threads. From send: As with std.concurrency.spawn, T must not have unshared aliasing. So clearly passing shared pointers or things containing shared pointers should work fine. As I was building code to test, I found that it does actually work for shared int pointers: import std.concurrency; import std.typecons; import std.stdio; import core.thread; void threadfunc() { bool done = false; while(!done) { receive( (shared(int)*foo) {*foo = 5; done = true;}, (Variant v) {} ); } } shared int f; void main() { auto tid = spawn(&threadfunc); tid.send(&f); Thread.sleep(1.seconds); writeln(f); } No error, completes as expected, and outputs 5. So it looks like this is really a straight up bug and has nothing to do with the shared type qualifier. It is documented as working, and does work in some cases. I think if the shared item is not a reference, it is doing something different, and this is incompatible with something in the implementation. Indeed, if you attempt to send unshared references, you get an assert. However, if you send a shared int (not a pointer), you get a bunch of compiler errors. Clearly the implementation expects it to work, as it doesn't fail the logical checks. -Steve
Aug 15
parent reply Kagamin <spam here.lot> writes:
https://github.com/dlang/phobos/blob/master/std/variant.d#L623
memcpy(&store, cast(const(void*)) &rhs, rhs.sizeof);

should be ok to cast unconditionally
Aug 15
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/15/17 10:42 AM, Kagamin wrote:
 https://github.com/dlang/phobos/blob/master/std/variant.d#L623
 memcpy(&store, cast(const(void*)) &rhs, rhs.sizeof);
 
 should be ok to cast unconditionally
Agreed, the T value being copied has already been copied onto the stack, so it's not really shared. However, I'm not sure about the postblit being called afterward. Does a postblit need to be marked shared in order to work for shared types? I think also I understand why it's working for shared int* but not shared int -- IFTI automatically infers tail-modified for such things to cut down on instantiations. tail-modified means the head is not shared (which is in this case more accurate). This means the cast is not necessary, since you are copying unqualified data. -Steve
Aug 15
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer 
wrote:
 However, I'm not sure about the postblit being called 
 afterward. Does a postblit need to be marked shared in order to 
 work for shared types?
Ideally yes, but it's difficult to come up with a good shared postblit, send and receive is probably the only use case for it :)
Aug 16
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 6:23 AM, Kagamin wrote:
 On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer wrote:
 However, I'm not sure about the postblit being called afterward. Does 
 a postblit need to be marked shared in order to work for shared types?
Ideally yes, but it's difficult to come up with a good shared postblit, send and receive is probably the only use case for it :)
Use cases don't matter. What matters is: is it proper for Variant to call the postblit (as it does currently) without regard for the qualifiers? However, I have found a better way to call postblit that involves the qualifiers than the way Variant currently does it. I'm going to submit a PR to fix these issues. -Steve
Aug 16
next sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven 
Schveighoffer wrote:
 Use cases don't matter. What matters is: is it proper for 
 Variant to call the postblit (as it does currently) without 
 regard for the qualifiers?
Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared reference counter with (probably) good enough postblit.
Aug 16
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 9:04 AM, Kagamin wrote:
 On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven Schveighoffer wrote:
 Use cases don't matter. What matters is: is it proper for Variant to 
 call the postblit (as it does currently) without regard for the 
 qualifiers?
Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared reference counter with (probably) good enough postblit.
I have found that shared dtor doesn't work: struct S { ~this() shared {} } void main() { shared S s; // Error: shared method testshared.S.~this is not callable using a non-shared object } But shared postblit does work. Kind of a bummer. Though I think we need some work on the destructors and postblits -- you can't overload them based on qualifiers. But that isn't a concern for Variant. It is only calling the postblit, which does work. -Steve
Aug 16
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven 
Schveighoffer wrote:
 But that isn't a concern for Variant. It is only calling the 
 postblit, which does work.
Shouldn't it call destructor when it goes out of scope?
Aug 17
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/17/17 8:41 AM, Kagamin wrote:
 On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven Schveighoffer wrote:
 But that isn't a concern for Variant. It is only calling the postblit, 
 which does work.
Shouldn't it call destructor when it goes out of scope?
You're right, and it does. It uses the typeid to destroy it, which I think ignores any attributes. I've updated my PR to switch to the __xdtor method, which takes into account attributes of the method. This doesn't suffer from the same incorrect assumption the compiler makes when destroying something in a scope, so ironically, using a Variant to wrap a shared type with a destructor is going to work better than using the stack :) -Steve
Aug 17
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 8:58 AM, Steven Schveighoffer wrote:
 However, I have found a better way to call postblit that involves the 
 qualifiers than the way Variant currently does it. I'm going to submit a 
 PR to fix these issues.
https://github.com/dlang/phobos/pull/5694 -Steve
Aug 16
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 11:23 AM, Steven Schveighoffer wrote:
 On 8/16/17 8:58 AM, Steven Schveighoffer wrote:
 However, I have found a better way to call postblit that involves the 
 qualifiers than the way Variant currently does it. I'm going to submit 
 a PR to fix these issues.
https://github.com/dlang/phobos/pull/5694
This has been merged, so you should now be able to send shared types properly through send/receive on master dmd. Don't think it made it into 2.076 beta though. -Steve
Aug 21
next sibling parent reply Kagamin <spam here.lot> writes:
https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? 
How various opIndex will behave now?
Aug 22
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/22/17 12:15 PM, Kagamin wrote:
 https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? How 
 various opIndex will behave now?
Seems to work. I closed as a duplicate. -Steve
Aug 22
prev sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 21 August 2017 at 18:50:50 UTC, Steven Schveighoffer 
wrote:
 On 8/16/17 11:23 AM, Steven Schveighoffer wrote:
 On 8/16/17 8:58 AM, Steven Schveighoffer wrote:
 However, I have found a better way to call postblit that 
 involves the qualifiers than the way Variant currently does 
 it. I'm going to submit a PR to fix these issues.
https://github.com/dlang/phobos/pull/5694
This has been merged, so you should now be able to send shared types properly through send/receive on master dmd. Don't think it made it into 2.076 beta though. -Steve
Great! Thanks. Arek
Aug 22