www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Does anyone understand how to use "shared" types with concurrency

reply Arek <arychlinski gmail.com> writes:
I have the folowing problem:
I like to envelope the class object in struct to control the 
destruction moment and then send this object to another 
thread/fiber (or task, cause I use vibe-d).

I can't find any method to make it working. Any ideas?

dmd (version 075) gives so stupid results, I belive it's broken 
(I've created the issue 
https://issues.dlang.org/show_bug.cgi?id=17749 )

ldc2 returns some errors:

.../import/std/variant.d(610,27): Error: function 
core.stdc.string.memcpy (void* s1, const(void*) s2, ulong n) is 
not callable using argument types (ubyte[32]*, 
shared(Env!(shared(A)))*, ulong)
.../import/std/conv.d(4186,9): Error: static assert  "Cannot 
emplace a Env!(shared(A)) because Env!(shared(A)).this(this) is 
annotated with  disable."
.../import/std/conv.d(4198,24):        instantiated from here: 
emplaceRef!(Env!(shared(A)), Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(301,35):        instantiated from here: 
emplaceRef!(Env!(shared(A)), Env!(shared(A)))
.../import/std/variant.d(630,21):        instantiated from here: 
handler!(shared(Env!(shared(A))))
.../import/std/variant.d(544,17):        ... (5 instantiations, 
-v to show) ...
../../../.dub/packages/vibe-core-1.1.1/vibe-core/source/vibe/core/con
urrency.d(1223,64):        instantiated from here:
send!(shared(Env!(shared(A))))
app.d(74,7):        instantiated from here: 
send!(shared(Env!(shared(A))))



The code may be complied like this: dub --single --compiler=ldc2 
./app.d   (assuming it's saved in app.d file).


/+
dub.sdl:
name "simple"
dependency "vibe-core" version="~>1.1.0"
+/
import std.stdio;
import std.format;

import vibe.core.core;
import vibe.core.task;
import vibe.core.concurrency;

// simple class with destructor
shared class A
{
	int i;
	this(int i)
	{
		this.i = i;
	}

	~this()
	{
		writeln("destruct ", i);
	}

}

shared struct Env(T) if (is(T == class) && is(T == shared))
{
	T obj;

	this(T o)
	{
		obj = obj;
	}

	this(this)
	{
		// some magic here
	}

	auto opAssign(shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	auto opAssign(ref shared(Env!T) other)
	{
		obj = other.obj;
		return this;
	}

	~this()
	{
		if (obj !is null)
			destroy(obj);
		obj = null;
	}
}

void main()
{
	auto consumer = runTask({
		auto got = receiveOnly!(shared Env!(shared A))();
		writeln(typeof(got).stringof);
	});

	auto producer = runTask({
		auto a = new shared A(1);
		shared b = shared Env!(shared A)(a);
		writeln(typeof(b).stringof);
		send(consumer, b);
	});
	runApplication();
}
Aug 12 2017
next sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the 
 destruction moment and then send this object to another 
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


I tried it too some time ago. Then I read Alexandrescu book and 
realized that the authors of the language do not want anyone to 
do this. Long story short, just plan your application so that 
each complex object is monitored by only one thread/task, and 
pass not objects, but messages (immutable structs) about what to 
do with them.
Aug 12 2017
parent reply Arek <arychlinski gmail.com> writes:
On Sunday, 13 August 2017 at 02:50:13 UTC, crimaniak wrote:

 On Saturday, 12 August 2017 at 18:57:44 UTC, Arek wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the 
 destruction moment and then send this object to another 
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 I tried it too some time ago. Then I read Alexandrescu book and 
 realized that the authors of the language do not want anyone to 
 do this. Long story short, just plan your application so that 
 each complex object is monitored by only one thread/task, and 
 pass not objects, but messages (immutable structs) about what 
 to do with them.


Yeah, I've read this. But conurrency.send cannot pass immutable 
object. The same story with Unique.
What means, that if I have complex objects which I want to 
transfer between threads I must keep them somewhere in global 
memory and share among threads. I can use send/receive only to 
signal availability of their existance. What I need is kind of 
"move" operation beetwen threads.

More over, "shared" looks rather like unfinished concept. It is 
really difficult to create proper struct (with postblit and 
destructor) working as shared object.

I even have no clue what is exact semantics of "shared" types. 
Language specification is a little laconic.

Anyway, _gshared looks very promising, so I will try to work out 
any approach.

Thanks for answer.
Arek
Aug 13 2017
parent reply crimaniak <crimaniak gmail.com> writes:
On Sunday, 13 August 2017 at 11:35:05 UTC, Arek wrote:

 Yeah, I've read this. But conurrency.send cannot pass immutable 
 object. The same story with Unique.


  Sorry, read this as 'efficient immutable'.


 More over, "shared" looks rather like unfinished concept.


  Yes, exactly.


 Anyway, _gshared looks very promising, so I will try to work 
 out any approach.


More of this, I think, you can't avoid __gshared for any complex 
work. Even mutexes from Phobos doesn't support shared, so I had 
to 'cowboy with __gshared' when implementing my site engine.
Aug 13 2017
parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Sunday, August 13, 2017 16:40:03 crimaniak via Digitalmars-d-learn wrote:

 More of this, I think, you can't avoid __gshared for any complex
 work. Even mutexes from Phobos doesn't support shared, so I had
 to 'cowboy with __gshared' when implementing my site engine.


The way to handle shared is to protect the section of code that's using the
shared object with either a mutex or synchronized block, and then you cast
away shared from the object within that section and operate on it as
thread-local. When you're done, you make sure that you don't have any
thread-local references to the data, and you release the mutex or exit the
synchronized block. e.g. something like

shared T sharedObj = getSharedObj();

synchronized(mutex)
{
    T nonSharedObj = cast(T)sharedObject

    // do stuff...

    // make sure that no references to nonSharedObj have escaped
}

// now, there's just the shared version of the object

And no, this isn't ideal, but the only semi-decent solution that's been
proposed that safely casts away shared for you is synchronized classes,
which Andrei describes in TDPL but have never been implemented. And because
they can only safely strip off the outermost layer of shared, they're of
questionable usefulness anyway. Ultimately, even with synchronized classes,
in many situations, the programmer is going to have to carefully cast away
shared to operate on the object within a protected context.

Now, the fact that the mutex objects don't handle shared correctly is
another issue entirely. Having to cast away shared from mutexes is dumb,
because you're obviously not going to be protecting them with a mutex, and
their operations have to be atomic anyway for them to do what they do. So,
that definitely needs to be fixed. However, I believe that it _has_ been
fixed in master, and it might have made it into a release now, but I'm not
sure. So, core.sync.mutex.Mutex _should_ now be useable as shared like it
should be.

In general though, the idea is that you simply don't operate on shared
objects except via atomic operations. Otherwise, you risk concurrency
problems. And really, this is the same as what you'd do in C/C++, except
that in C/C++, it doesn't catch you when you operate on an object that's
shared across threads with non-atomic operations (because the object isn't
explicitly typed as shared), and you don't have to cast away shared to do
non-atomic operations. So, having to cast away shared is the price of
getting the protection against accidentally using non-atomic operations on a
shared object as well as the price we pay to be able to have the type system
distinguish between shared and thread-local objects so that it's able to
optimize based on the knowledge that an object is thread-local. Ultimately
though, you're doing the same thing that you'd do in C++ if you're handling
concurrency safely. You just have to explicitly mark stuff as shared and
carefully cast away shared in certain, protected contexts.

Using __gshared in extern(D) code is just asking for it, because then you
have an object that the compiler thinks is thread-local but isn't, and you
risk subtle and nasty bugs as a result. __gshared is only intended for
binding to extern(C), global variables. To an extent, you can get away with
using it with extern(D) variables, but that's not its intended purpose, and
you risk running afoul of the compiler and what it chooses to do based on
the assumption that the object is thread-local.

- Jonathan M Davis
Aug 13 2017
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:

 [snip]

 Now, the fact that the mutex objects don't handle shared 
 correctly is another issue entirely. Having to cast away shared 
 from mutexes is dumb, because you're obviously not going to be 
 protecting them with a mutex, and their operations have to be 
 atomic anyway for them to do what they do. So, that definitely 
 needs to be fixed. However, I believe that it _has_ been fixed 
 in master, and it might have made it into a release now, but 
 I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable 
 as shared like it should be.


My fixes for shared(Mutex) - 
https://github.com/dlang/druntime/pull/1728 - are part of the DMD 
v2.074.0 release, and should propagate to LDC with their 1.4 
release respectively. GDC master now at druntime/phobos 2.074.1 - 
https://github.com/D-Programming-GDC/GDC/pull/539 and they're in 
the process of moving to 2.075.0 - 
https://github.com/D-Programming-GDC/GDC/pull/542.

Unfortunately, I forgot to write a changelog entry when I was 
working on the PR and later didn't have time to do so, before the 
2.074.0 release.

So shared (Mutex) should be in good shape now. I should 
supplement the ddoc unittest to address the transitive you issues 
you talked about. Currently it only shows synchronized use of a 
plain value type without indirections, which is uninteresting.
The example I'm thinking about is casting with HeadUnsharedOf / 
TailSharedOf a pointer to a singly-linked list, because with 
pointers you could show the shared(Node*) -> shared(Node)* 
transition in the type system, which unfortunately can't be 
demonstrated with implicitly reference types (classes).

Speaking of TailSharedOf, Jonathan could take a look at this PR: 
https://github.com/dlang/druntime/pull/1605? I and Andrei already 
approved it, but I'd like to get a third opinion since this is a 
breaking change and Martin seems be too busy the moment.
Aug 14 2017
prev sibling next sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:

 On Sunday, August 13, 2017 16:40:03 crimaniak via 
 Digitalmars-d-learn wrote:

 More of this, I think, you can't avoid __gshared for any 
 complex work. Even mutexes from Phobos doesn't support shared, 
 so I had to 'cowboy with __gshared' when implementing my site 
 engine.


 The way to handle shared is to protect the section of code 
 that's using the shared object with either a mutex or 
 synchronized block, and then you cast away shared from the 
 object within that section and operate on it as thread-local. 
 When you're done, you make sure that you don't have any 
 thread-local references to the data, and you release the mutex 
 or exit the synchronized block. e.g. something like

 shared T sharedObj = getSharedObj();

 synchronized(mutex)
 {
     T nonSharedObj = cast(T)sharedObject

     // do stuff...

     // make sure that no references to nonSharedObj have escaped
 }

 // now, there's just the shared version of the object


Yeah, and this is what i'm  doing now (more or less).
To be more precise, I don't even want to synchronize access to 
the shared resource between the threads. I just want to move the 
object from one thread to another.
Of course I could copy the local object, but my obcjecs has 
indirect references to others, forming kind of tree.

I like the idea of channels in Go. I've tried to get something 
similiar with send/receive.
In this case, object could be also immutable, because once there 
are created (in the deserialization process) they will no be 
modified. I just have to emit them into another task (to be 
honest, I use fiber, so it's not even another thread, and they 
will not be accessed in parallel).

But all this language protections makes the issue unexpectedly 
complicated.




 And no, this isn't ideal, but the only semi-decent solution 
 that's been proposed that safely casts away shared for you is 
 synchronized classes, which Andrei describes in TDPL but have 
 never been implemented. And because they can only safely strip 
 off the outermost layer of shared, they're of questionable 
 usefulness anyway. Ultimately, even with synchronized classes, 
 in many situations, the programmer is going to have to 
 carefully cast away shared to operate on the object within a 
 protected context.

 Now, the fact that the mutex objects don't handle shared 
 correctly is another issue entirely. Having to cast away shared 
 from mutexes is dumb, because you're obviously not going to be 
 protecting them with a mutex, and their operations have to be 
 atomic anyway for them to do what they do. So, that definitely 
 needs to be fixed. However, I believe that it _has_ been fixed 
 in master, and it might have made it into a release now, but 
 I'm not sure. So, core.sync.mutex.Mutex _should_ now be useable 
 as shared like it should be.

 In general though, the idea is that you simply don't operate on 
 shared objects except via atomic operations. Otherwise, you 
 risk concurrency problems. And really, this is the same as what 
 you'd do in C/C++, except that in C/C++, it doesn't catch you 
 when you operate on an object that's shared across threads with 
 non-atomic operations (because the object isn't explicitly 
 typed as shared), and you don't have to cast away shared to do 
 non-atomic operations. So, having to cast away shared is the 
 price of getting the protection against accidentally using 
 non-atomic operations on a shared object as well as the price 
 we pay to be able to have the type system distinguish between 
 shared and thread-local objects so that it's able to optimize 
 based on the knowledge that an object is thread-local. 
 Ultimately though, you're doing the same thing that you'd do in 
 C++ if you're handling concurrency safely. You just have to 
 explicitly mark stuff as shared and carefully cast away shared 
 in certain, protected contexts.

 Using __gshared in extern(D) code is just asking for it, 
 because then you have an object that the compiler thinks is 
 thread-local but isn't, and you risk subtle and nasty bugs as a 
 result. __gshared is only intended for binding to extern(C), 
 global variables. To an extent, you can get away with using it 
 with extern(D) variables, but that's not its intended purpose, 
 and you risk running afoul of the compiler and what it chooses 
 to do based on the assumption that the object is thread-local.

 - Jonathan M Davis


Thanks for the explanation! It would be good to have a 
comprehensive article on this subject.

Arek
Aug 14 2017
prev sibling next sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:


 The way to handle shared is to protect the section of code 
 that's using the shared object with either a mutex or 
 synchronized block, and then you cast away shared from the 
 object within that section and operate on it as thread-local. 
 When you're done, you make sure that you don't have any 
 thread-local references to the data, and you release the mutex 
 or exit the synchronized block. e.g. something like

 shared T sharedObj = getSharedObj();

 synchronized(mutex)
 {
     T nonSharedObj = cast(T)sharedObject

     // do stuff...

     // make sure that no references to nonSharedObj have escaped
 }


Casting objects just to work with it every time is so bad style 
for me that I even didn't consider such possibility. In fact, I 
did make something like this but with __gshared both object and 
mutex. I think I need to review this part of site engine and my 
programming habits too.
I wonder if it possible and usable to make some template to 
support this pattern, where we give mutex(es), shared object(s) 
and delegate to operate with objects as non-shared.
Aug 16 2017
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote:

 I wonder if it possible and usable to make some template to 
 support this pattern, where we give mutex(es), shared object(s) 
 and delegate to operate with objects as non-shared.


https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it 
helps, don't forget that it's a casted shared object.
Aug 17 2017
parent crimaniak <crimaniak gmail.com> writes:
On Thursday, 17 August 2017 at 13:09:29 UTC, Kagamin wrote:

 On Wednesday, 16 August 2017 at 23:15:10 UTC, crimaniak wrote:

 I wonder if it possible and usable to make some template to 
 support this pattern, where we give mutex(es), shared 
 object(s) and delegate to operate with objects as non-shared.


 https://dpaste.dzfl.pl/8b3b05c8ec0a like this? Not sure if it 
 helps, don't forget that it's a casted shared object.


  Yes, something like this. In general, I thought about the 
possibility of using several shared objects in this block but 
then realized that everything can be reduced to the case of one 
object.
Aug 20 2017
prev sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Monday, 14 August 2017 at 03:59:48 UTC, Jonathan M Davis wrote:


 And no, this isn't ideal, but the only semi-decent solution 
 that's been proposed that safely casts away shared for you is 
 synchronized classes, which Andrei describes in TDPL but have 
 never been implemented.


  After reading this I did some experiment to understand the 
situation better. I make a simple class and unittest:

// dmd sync1.d -unittest -main

unittest
{
     import std.stdio;

synchronized
class A
{
     private int a;

     void inc()
     {
	++a;
     }

     int get(){ return a;}
}

     shared A a;

     for(int i=0; i<100; ++i)
	a.inc();
	
     writeln(a.get);	

}

Oops! Deprecation: read-modify-write operations are not allowed 
for shared variables. Use core.atomic.atomicOp!"+="(this.a, 1) 
instead.

Why use atomic operations if the class already synchronized? 
Well..

...
    import core.atomic: atomicOp;
...
	// ++a; // Deprecation: read-modify-write operations are not 
allowed for shared variables. Use 
core.atomic.atomicOp!"+="(this.a, 1) instead.
	atomicOp!"+="(this.a, 1);
...

ok, works. But it works by the way as if synchronized just makes 
all methods shared, but does not provide the object methods with 
a mutex lock, as Java does. Am I right here? And what preventing 
to implement it right, lack of manpower or some ideologic 
problems?
Aug 20 2017
parent crimaniak <crimaniak gmail.com> writes:
On Monday, 21 August 2017 at 02:17:57 UTC, crimaniak wrote:
...

     shared A a;


...
  Sorry, accidental delete, read this as shared A a = new 
shared(A);
Aug 20 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


Unfortunately, send and receive do not currently work with shared because of
issues with Variant, which they use internally.

- Jonathan M Davis
Aug 13 2017
next sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 03:40:26 UTC, Jonathan M Davis wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 Unfortunately, send and receive do not currently work with 
 shared because of issues with Variant, which they use 
 internally.

 - Jonathan M Davis


That's what I suspected. Thanks for the confirmation. I was not 
sure if I was doing everything right.

Arek
Aug 14 2017
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 
 Unfortunately, send and receive do not currently work with shared because of
 issues with Variant, which they use internally.


This can't be a correct statement. This is the whole point of shared.

-Steve
Aug 14 2017
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 19:22:23 UTC, Steven Schveighoffer 
wrote:

 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn 
 wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 
 Unfortunately, send and receive do not currently work with 
 shared because of
 issues with Variant, which they use internally.


 This can't be a correct statement. This is the whole point of 
 shared.

 -Steve


First of all, I'm not native English speaker, so forgive me 
possible misunderstanding

In my opinion the whole problem of 'shared' is that when I use 
send/receive operation (I believe it's should be something 
similar to go's channels or erlang messages) I do not want to 
share the memory so any "shared" concept is useless.

What I really would like to get is possibility to make:

// in first thread
Unique!MyObject msg = new MyObject()
send(msg.release);

// in other thread:
auto msg = receiveOnly!(Unique!MyObject)();


My object disappears in "send" operation and its new instance is 
constructed on receive.
If I can ensure the uniqueness of the object, there is no need to 
"share" it or synchronize the access.

But now such operation is impossible and the compiler forces me 
to cope with shared if MyObject has any references to other areas 
of memory.

Anyway, thanks for all replies.

Arek
Aug 14 2017
parent Kagamin <spam here.lot> writes:
On Monday, 14 August 2017 at 20:13:28 UTC, Arek wrote:

 If I can ensure the uniqueness of the object, there is no need 
 to "share" it or synchronize the access.


You use manually managed multithreading, that's why you need 
shared. And because compiler can't verify uniqueness, you are 
requested to do it manually by casting.
Aug 15 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
learn wrote:

 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn 




wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 Unfortunately, send and receive do not currently work with shared
 because of issues with Variant, which they use internally.


 This can't be a correct statement. This is the whole point of shared.


What's incorrect about it? It's a longstanding issue that because Variant
can't contain shared data, send and receive do not work with shared. You can
send and receive mutable data with no indirections, and AFAIK, you can send
immutable data (though the OP is apparently having problems with that, so I
guess that that doesn't work completely, though I definitely recall doing so
previously), but you get compilation errors if you try to send shared data.
For instance, this code

import std.concurrency;

void main()
{
    static void func(Tid parent)
    {
        auto received = receiveOnly!(shared int[string]);
    }

    shared int[string] aa;

    auto tid = spawn(&func, thisTid);
    send(tid, aa);
}

will fail to compile with

/usr/local/include/dmd/std/variant.d(625): Error: function 
core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong 
n) is not callable using argument types (ubyte[32]*, shared(int[string])*, 
ulong)
/usr/local/include/dmd/std/variant.d(625): Error: function 
core.stdc.string.memcpy (return scope void* s1, scope const(void*) s2, ulong 
n) is not callable using argument types (ubyte[32]*, shared(int)*, ulong)
/usr/local/include/dmd/std/variant.d(424): Error: template instance 
std.variant.VariantN!32LU.VariantN.opAssign!(shared(int)) error 
instantiating
/usr/local/include/dmd/std/variant.d(645):        instantiated from here: 
handler!(shared(int[string]))
/usr/local/include/dmd/std/variant.d(559):        instantiated from here: 
opAssign!(shared(int[string]))
/usr/local/include/dmd/std/concurrency.d(101):        instantiated from 
here: __ctor!(shared(int[string]))
/usr/local/include/dmd/std/concurrency.d(606):        ... (2 instantiations, 
-v to show) ...
/usr/local/include/dmd/std/concurrency.d(576):        instantiated from 
here: _send!(shared(int[string]))
q.d(13):        instantiated from here: send!(shared(int[string]))

whereas if you use something like int, it has no problem. It won't even
compile if you give it a shared int, and that arguably should just be
converting to and from int, since shared int has no indirections.

There are at least two bugs reported on the issue:

https://issues.dlang.org/show_bug.cgi?id=13262
https://issues.dlang.org/show_bug.cgi?id=14893

I suspect that the solution involves making it so that std.concurrency uses
something other than Variant to store the data being sent, but regardless,
no one has fixed the issue yet. So, while send and receive work fine for
simple message, they're fairly crippled when it comes to complex data and
have been for their entire existence AFAIK.

- Jonathan M Davis
Aug 14 2017
next sibling parent reply Arek <arychlinski gmail.com> writes:
On Monday, 14 August 2017 at 21:27:48 UTC, Jonathan M Davis wrote:

 On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via 
 Digitalmars-d- learn wrote:

 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn 
 wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via 
 Digitalmars-d-learn




 wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 Unfortunately, send and receive do not currently work with 
 shared because of issues with Variant, which they use 
 internally.


 This can't be a correct statement. This is the whole point of 
 shared.


 What's incorrect about it? It's a longstanding issue that 
 because Variant can't contain shared data, send and receive do 
 not work with shared. You can send and receive mutable data 
 with no indirections, and AFAIK, you can send immutable data 
 (though the OP is apparently having problems with that, so I 
 guess that that doesn't work completely, though I definitely 
 recall doing so previously), but you get compilation errors if 
 you try to send shared data. For instance, this code

 import std.concurrency;

 void main()
 {
     static void func(Tid parent)
     {
         auto received = receiveOnly!(shared int[string]);
     }

     shared int[string] aa;

     auto tid = spawn(&func, thisTid);
     send(tid, aa);
 }


I've found some simple workaround for this problem:

import std.stdio;
import std.concurrency;

struct Envelope(T) if (is(T == class)) // for simplicity of this 
example, only classes
{
	shared(T)[] obj;

	this(shared T o)
	{
		this.obj = [o];
	}

	T get()  property nothrow  nogc
	{
		return cast() obj[0];
	}
}

class A
{

}

void consumer()
{
	auto r = receiveOnly!(Envelope!(A))();
	writeln("Got: ", typeof(r).stringof);
}

void main()
{
	auto cons = spawn(&consumer);
	auto o = Envelope!A(new A());
	send(cons, o);
}

Shared object can be encapsulated in the array. In case of other 
(non-class) types the pointer can be used, and get() should 
return ref to the pointed object (after stripping off the shared 
qualifier).

send() could encapsulate itself shared objectes.

Arek
Aug 14 2017
parent reply Kagamin <spam here.lot> writes:
On Monday, 14 August 2017 at 22:22:58 UTC, Arek wrote:

 I've found some simple workaround for this problem:

 import std.stdio;
 import std.concurrency;

 struct Envelope(T) if (is(T == class)) // for simplicity of 
 this example, only classes
 {
 	shared(T)[] obj;

 	this(shared T o)
 	{
 		this.obj = [o];
 	}

 	T get()  property nothrow  nogc
 	{
 		return cast() obj[0];
 	}
 }

 class A
 {

 }

 void consumer()
 {
 	auto r = receiveOnly!(Envelope!(A))();
 	writeln("Got: ", typeof(r).stringof);
 }

 void main()
 {
 	auto cons = spawn(&consumer);
 	auto o = Envelope!A(new A());
 	send(cons, o);
 }

 Shared object can be encapsulated in the array. In case of 
 other (non-class) types the pointer can be used, and get() 
 should return ref to the pointed object (after stripping off 
 the shared qualifier).


Rather like this:

struct Sendable(T)
{
	shared T o;
	alias o this;
}

import std.concurrency;

class A
{
	int method() shared;
}

void consumer()
{
	shared A a = receiveOnly!(Sendable!(A))();
}

void producer()
{
	auto cons = spawn(&consumer);
	shared A a = new shared A();
	send(cons, Sendable!A(a));
}
Aug 15 2017
parent reply Kagamin <spam here.lot> writes:
Well, no wrapper is actually needed here:

class A
{
	int method() shared;
}

void consumer()
{
	shared a = receiveOnly!(shared A)();
}

void producer()
{
	auto cons = spawn(&consumer);
	send(cons, new shared A());
}
Aug 15 2017
parent reply Arek <arychlinski gmail.com> writes:
On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:

 Well, no wrapper is actually needed here:

 class A
 {
 	int method() shared;
 }

 void consumer()
 {
 	shared a = receiveOnly!(shared A)();
 }

 void producer()
 {
 	auto cons = spawn(&consumer);
 	send(cons, new shared A());
 }


Yes, but this doesn't compile:

import std.stdio;
import std.concurrency;

struct A
{
	int t;
	int r;
	int method() shared
	{
		return 0;
	}
}

void consumer()
{
	shared a = receiveOnly!(shared A)();
}

void main()
{
	auto cons = spawn(&consumer);
	send(cons, shared A());
}

This very simple code also doesn't compile:

shared struct S
{
	int i;

	~this()
	{
	}
}

void main()
{
	shared s = shared S();
}

In general, shared structs with postblit and destructor make 
problems.

Arek
Aug 15 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/15/17 5:27 PM, Arek wrote:

 On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:

 Well, no wrapper is actually needed here:

 class A
 {
     int method() shared;
 }

 void consumer()
 {
     shared a = receiveOnly!(shared A)();
 }

 void producer()
 {
     auto cons = spawn(&consumer);
     send(cons, new shared A());
 }


 
 Yes, but this doesn't compile:
 
 import std.stdio;
 import std.concurrency;
 
 struct A
 {
      int t;
      int r;
      int method() shared
      {
          return 0;
      }
 }
 
 void consumer()
 {
      shared a = receiveOnly!(shared A)();
 }
 
 void main()
 {
      auto cons = spawn(&consumer);
      send(cons, shared A());
 }


The issue is that send cannot handle shared value types due to a bug in 
the implementation. In essence, there is no reason to send a shared A -- 
it's an unrelated copy and not actually shared.

You can send a shared reference to an A just fine:

static shared A a;

send(tid, &a); // works

You *should* be able to send a shared(A). There is no reason to disallow 
it. But again, it's not super useful.


 This very simple code also doesn't compile:
 
 shared struct S
 {
      int i;
 
      ~this()
      {
      }
 }
 
 void main()
 {
      shared s = shared S();
 }
 
 In general, shared structs with postblit and destructor make problems.


postblit should work.

It does look like destructors are a problem. It appears the compiler 
attempts to call the destructor while unshared.

-Steve
Aug 15 2017
parent Arek <arychlinski gmail.com> writes:
On Tuesday, 15 August 2017 at 21:54:23 UTC, Steven Schveighoffer 
wrote:

 On 8/15/17 5:27 PM, Arek wrote:

 On Tuesday, 15 August 2017 at 10:37:08 UTC, Kagamin wrote:

 Well, no wrapper is actually needed here:






[...]

 The issue is that send cannot handle shared value types due to 
 a bug in the implementation. In essence, there is no reason to 
 send a shared A -- it's an unrelated copy and not actually 
 shared.

 You can send a shared reference to an A just fine:

 static shared A a;

 send(tid, &a); // works

 You *should* be able to send a shared(A). There is no reason to 
 disallow it. But again, it's not super useful.


Yes, I absolutely agree. The object may be shared due to other 
requirements and there is no reason to reject 'share' qualifier 
what I want to send/receive it.


 This very simple code also doesn't compile:
 
 shared struct S
 {
      int i;
 
      ~this()
      {
      }
 }
 
 void main()
 {
      shared s = shared S();
 }
 
 In general, shared structs with postblit and destructor make 
 problems.


 postblit should work.

 It does look like destructors are a problem. It appears the 
 compiler attempts to call the destructor while unshared.

 -Steve


But I have no idea how and when the object loses its shared 
nature. Yes, it looks like a bug in dmd. ldc 1.3 doesn't complain 
in this case.

Arek
Aug 16 2017
prev sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 15 August 2017 at 21:27:49 UTC, Arek wrote:

 Yes, but this doesn't compile:

 import std.stdio;
 import std.concurrency;

 struct A
 {
 	int t;
 	int r;
 	int method() shared
 	{
 		return 0;
 	}
 }

 void consumer()
 {
 	shared a = receiveOnly!(shared A)();
 }

 void main()
 {
 	auto cons = spawn(&consumer);
 	send(cons, shared A());
 }


AIU you use struct as a Unique-like wrapper for shared class 
object. Extract the object and send it, then wrap again on 
arrival.
Aug 16 2017
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/14/17 5:27 PM, Jonathan M Davis via Digitalmars-d-learn wrote:

 On Monday, August 14, 2017 15:22:23 Steven Schveighoffer via Digitalmars-d-
 learn wrote:

 On 8/13/17 11:40 PM, Jonathan M Davis via Digitalmars-d-learn wrote:

 On Saturday, August 12, 2017 18:57:44 Arek via Digitalmars-d-learn




 wrote:

 I have the folowing problem:
 I like to envelope the class object in struct to control the
 destruction moment and then send this object to another
 thread/fiber (or task, cause I use vibe-d).

 I can't find any method to make it working. Any ideas?


 Unfortunately, send and receive do not currently work with shared
 because of issues with Variant, which they use internally.


 This can't be a correct statement. This is the whole point of shared.


 
 What's incorrect about it? It's a longstanding issue that because Variant
 can't contain shared data, send and receive do not work with shared.


The implementation details aren't important. From the documentation (and 
no, this documentation is not wrong):

 From spawn:
args must not have unshared aliasing. In other words, all arguments to 
fn must either be shared or immutable or have no pointer indirection. 
This is necessary for enforcing isolation among threads.

 From send:
As with std.concurrency.spawn, T must not have unshared aliasing.

So clearly passing shared pointers or things containing shared pointers 
should work fine.

As I was building code to test, I found that it does actually work for 
shared int pointers:

import std.concurrency;
import std.typecons;
import std.stdio;
import core.thread;

void threadfunc()
{
     bool done = false;
     while(!done)
     {  receive(
                   (shared(int)*foo) {*foo = 5; done = true;},
                   (Variant v) {}
                   );
     }
}

shared int f;

void main()
{
     auto tid = spawn(&threadfunc);
     tid.send(&f);
     Thread.sleep(1.seconds);
     writeln(f);
}


No error, completes as expected, and outputs 5.

So it looks like this is really a straight up bug and has nothing to do 
with the shared type qualifier. It is documented as working, and does 
work in some cases.

I think if the shared item is not a reference, it is doing something 
different, and this is incompatible with something in the 
implementation. Indeed, if you attempt to send unshared references, you 
get an assert. However, if you send a shared int (not a pointer), you 
get a bunch of compiler errors. Clearly the implementation expects it to 
work, as it doesn't fail the logical checks.

-Steve
Aug 15 2017
parent reply Kagamin <spam here.lot> writes:
https://github.com/dlang/phobos/blob/master/std/variant.d#L623
memcpy(&store, cast(const(void*)) &rhs, rhs.sizeof);

should be ok to cast unconditionally
Aug 15 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/15/17 10:42 AM, Kagamin wrote:

 https://github.com/dlang/phobos/blob/master/std/variant.d#L623
 memcpy(&store, cast(const(void*)) &rhs, rhs.sizeof);
 
 should be ok to cast unconditionally


Agreed, the T value being copied has already been copied onto the stack, 
so it's not really shared.

However, I'm not sure about the postblit being called afterward. Does a 
postblit need to be marked shared in order to work for shared types?

I think also I understand why it's working for shared int* but not 
shared int -- IFTI automatically infers tail-modified for such things to 
cut down on instantiations. tail-modified means the head is not shared 
(which is in this case more accurate). This means the cast is not 
necessary, since you are copying unqualified data.

-Steve
Aug 15 2017
parent reply Kagamin <spam here.lot> writes:
On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer 
wrote:

 However, I'm not sure about the postblit being called 
 afterward. Does a postblit need to be marked shared in order to 
 work for shared types?


Ideally yes, but it's difficult to come up with a good shared 
postblit, send and receive is probably the only use case for it :)
Aug 16 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 6:23 AM, Kagamin wrote:

 On Tuesday, 15 August 2017 at 15:19:02 UTC, Steven Schveighoffer wrote:

 However, I'm not sure about the postblit being called afterward. Does 
 a postblit need to be marked shared in order to work for shared types?


 
 Ideally yes, but it's difficult to come up with a good shared postblit, 
 send and receive is probably the only use case for it :)


Use cases don't matter. What matters is: is it proper for Variant to 
call the postblit (as it does currently) without regard for the qualifiers?

However, I have found a better way to call postblit that involves the 
qualifiers than the way Variant currently does it. I'm going to submit a 
PR to fix these issues.

-Steve
Aug 16 2017
next sibling parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven 
Schveighoffer wrote:

 Use cases don't matter. What matters is: is it proper for 
 Variant to call the postblit (as it does currently) without 
 regard for the qualifiers?


Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared 
reference counter with (probably) good enough postblit.
Aug 16 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 9:04 AM, Kagamin wrote:

 On Wednesday, 16 August 2017 at 12:58:25 UTC, Steven Schveighoffer wrote:

 Use cases don't matter. What matters is: is it proper for Variant to 
 call the postblit (as it does currently) without regard for the 
 qualifiers?


 
 Looks like it isn't, https://dpaste.dzfl.pl/183e6dae9867 - shared 
 reference counter with (probably) good enough postblit.


I have found that shared dtor doesn't work:

struct S
{
   ~this() shared {}
}

void main()
{
    shared S s; // Error: shared method testshared.S.~this is not 
callable using a non-shared object
}

But shared postblit does work.

Kind of a bummer. Though I think we need some work on the destructors 
and postblits -- you can't overload them based on qualifiers.

But that isn't a concern for Variant. It is only calling the postblit, 
which does work.

-Steve
Aug 16 2017
parent reply Kagamin <spam here.lot> writes:
On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven 
Schveighoffer wrote:

 But that isn't a concern for Variant. It is only calling the 
 postblit, which does work.


Shouldn't it call destructor when it goes out of scope?
Aug 17 2017
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/17/17 8:41 AM, Kagamin wrote:

 On Wednesday, 16 August 2017 at 13:14:55 UTC, Steven Schveighoffer wrote:

 But that isn't a concern for Variant. It is only calling the postblit, 
 which does work.


 
 Shouldn't it call destructor when it goes out of scope?


You're right, and it does. It uses the typeid to destroy it, which I 
think ignores any attributes.

I've updated my PR to switch to the __xdtor method, which takes into 
account attributes of the method.

This doesn't suffer from the same incorrect assumption the compiler 
makes when destroying something in a scope, so ironically, using a 
Variant to wrap a shared type with a destructor is going to work better 
than using the stack :)

-Steve
Aug 17 2017
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 8:58 AM, Steven Schveighoffer wrote:

 However, I have found a better way to call postblit that involves the 
 qualifiers than the way Variant currently does it. I'm going to submit a 
 PR to fix these issues.


https://github.com/dlang/phobos/pull/5694

-Steve
Aug 16 2017
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/16/17 11:23 AM, Steven Schveighoffer wrote:

 On 8/16/17 8:58 AM, Steven Schveighoffer wrote:

 However, I have found a better way to call postblit that involves the 
 qualifiers than the way Variant currently does it. I'm going to submit 
 a PR to fix these issues.


 
 https://github.com/dlang/phobos/pull/5694


This has been merged, so you should now be able to send shared types 
properly through send/receive on master dmd. Don't think it made it into 
2.076 beta though.

-Steve
Aug 21 2017
next sibling parent reply Kagamin <spam here.lot> writes:
https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? 
How various opIndex will behave now?
Aug 22 2017
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 8/22/17 12:15 PM, Kagamin wrote:

 https://issues.dlang.org/show_bug.cgi?id=6585 is this fixed too? How 
 various opIndex will behave now?


Seems to work. I closed as a duplicate.

-Steve
Aug 22 2017
prev sibling parent Arek <arychlinski gmail.com> writes:
On Monday, 21 August 2017 at 18:50:50 UTC, Steven Schveighoffer 
wrote:

 On 8/16/17 11:23 AM, Steven Schveighoffer wrote:

 On 8/16/17 8:58 AM, Steven Schveighoffer wrote:

 However, I have found a better way to call postblit that 
 involves the qualifiers than the way Variant currently does 
 it. I'm going to submit a PR to fix these issues.


 
 https://github.com/dlang/phobos/pull/5694


 This has been merged, so you should now be able to send shared 
 types properly through send/receive on master dmd. Don't think 
 it made it into 2.076 beta though.

 -Steve


Great!

Thanks.

Arek
Aug 22 2017