www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - I'm porting some go code to D

reply Rory McGuire <rjmcguire gmail.com> writes:
So I'm porting so #golang code to #dlang and there is all these blasted
"go" statements.So I thought I'd give implmenting it in D a shot. What do
you guys think?
Fire away :).

import std.stdio;
import core.thread : Thread;
import core.sync.mutex : Mutex;
import core.time;

void main() {
    auto ch = chan!int(1);
    go!({
        foreach (i; 22..44) {
            ch._ = i;
        } });

    foreach (i; 0..10) {
        writeln("pop: ", ch._);
    }
}

// Implementation

void go(alias F)() {
    auto t = new Thread(F);
    t.isDaemon(true); // we don't care if this thread dies.
    t.start();
}

/**
 * chan allows messaging between threads without having to deal with locks,
similar to how chan works in golang
 */
class chan_(T) {
    shared Mutex lock;
    struct Container(T) {
        T value;
        Container!T* next;
    }
    shared Container!T* buf;
    shared Container!T* last;
    shared size_t length;
    shared void insert(shared T v) {
        shared Container!T* newItem = new shared Container!T();
        newItem.value = v;
        if (buf is null) {
            buf = newItem;
            last = newItem;
        } else {
            last.next = newItem;
            last = newItem;
        }
        length++;
    }
    shared T popFront() {
        T ret;
        synchronized (lock) {
            ret = buf.value;
            buf = buf.next;
            length--;
        }
        return ret;
    }
    size_t maxItems;
    bool blockOnFull = false;
    this(int maxItems = 1024, bool blockOnFull = true) {
        lock = cast(shared)new Mutex;
        length = 0;

        this.maxItems = maxItems;
        this.blockOnFull = blockOnFull;
    }

     property
    shared void _(T value) {
        bool done;
        while(true) {
            synchronized(lock) {
                if (!done && length < maxItems) {
                    insert(value);
                    done = true;
                } else if (!blockOnFull) {
                    throw new ChannelFull("Channel Full");
                }
                if (length <= maxItems-1) {
                    break;
                }
            }
            Thread.sleep(dur!"msecs"(5));
        }
    }
     property
    shared T _() {
        while(true) {
            size_t len;
            synchronized(lock) {
                len = length;
            }
            if (len > 0) {
                break;
            }
            Thread.sleep(dur!"msecs"(5));
        };
        auto r = popFront();
        return r;
    }
}
auto chan(T)(int n, bool blockOnFull = true) {
    return cast(shared)new chan_!T(n, blockOnFull);
}

class ChannelFull : Exception {
    this(string msg, string file = __FILE__, ulong line =
cast(ulong)__LINE__, Throwable next = null) {
        super(msg,file,line,next);
    }
}
Aug 23 2013
next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Friday, 23 August 2013 at 23:54:55 UTC, Rory McGuire wrote:

 So I'm porting so #golang code to #dlang and there is all these 
 blasted
 "go" statements.So I thought I'd give implmenting it in D a 
 shot. What do
 you guys think?
 Fire away :).


I'd suggest posting long snippets of code to 
https://gist.github.com/ or http://dpaste.dzfl.pl/

A couple lines is fine but whole implementations, definitely want 
a more color friendly solution.
Aug 23 2013
parent Rory McGuire <rjmcguire gmail.com> writes:
On 24 Aug 2013 03:15, "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> wrote:

 On Friday, 23 August 2013 at 23:54:55 UTC, Rory McGuire wrote:

 So I'm porting so #golang code to #dlang and there is all these blasted
 "go" statements.So I thought I'd give implmenting it in D a shot. What do
 you guys think?
 Fire away :).



 I'd suggest posting long snippets of code to https://gist.github.com/ or


http://dpaste.dzfl.pl/

 A couple lines is fine but whole implementations, definitely want a more


color friendly solution.

Good point.
Aug 24 2013
prev sibling next sibling parent reply David <d dav1d.de> writes:
Daemonic Threads often end with a segfault, so if your main thread
exists, the other threads will probably segfault.
Aug 24 2013
next sibling parent Rory McGuire <rjmcguire gmail.com> writes:
On 24 Aug 2013 11:00, "David" <d dav1d.de> wrote:

 Daemonic Threads often end with a segfault, so if your main thread
 exists, the other threads will probably segfault.


Thanks, I wonder what they're accessing that they shouldn't.
Aug 24 2013
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Aug 24, 2013, at 1:59 AM, David <d dav1d.de> wrote:


 Daemonic Threads often end with a segfault, so if your main thread
 exists, the other threads will probably segfault.


By default, sure.  But with daemon threads you really want to have some =
kind of shutdown mechanism inside a static dtor somewhere.  The goal is =
more to have threads that don't implicitly block app shutdown.=
Sep 20 2013
prev sibling next sibling parent reply "Moritz Maxeiner" <moritz ucworks.org> writes:
On Friday, 23 August 2013 at 23:54:55 UTC, Rory McGuire wrote:

 So I'm porting so #golang code to #dlang and there is all these 
 blasted
 "go" statements.So I thought I'd give implmenting it in D a 
 shot. What do
 you guys think?
 Fire away :).

 /**
  * chan allows messaging between threads without having to deal 
 with locks,
 similar to how chan works in golang
  */
 class chan_(T) {
     shared Mutex lock;
     struct Container(T) {
         T value;
         Container!T* next;


I'm probably missunderstanding somehting about the TLS model, but 
from what I know, for something like this, shouldn't you make the 
class instance itself go into shared storage instead instead of 
all the members?
Aug 24 2013
parent Rory McGuire <rjmcguire gmail.com> writes:
On 24 Aug 2013 11:25, "Moritz Maxeiner" <moritz ucworks.org> wrote:

 On Friday, 23 August 2013 at 23:54:55 UTC, Rory McGuire wrote:

 So I'm porting so #golang code to #dlang and there is all these blasted
 "go" statements.So I thought I'd give implmenting it in D a shot. What do
 you guys think?
 Fire away :).

 /**
  * chan allows messaging between threads without having to deal with




locks,

 similar to how chan works in golang
  */
 class chan_(T) {
     shared Mutex lock;
     struct Container(T) {
         T value;
         Container!T* next;



 I'm probably missunderstanding somehting about the TLS model, but from


what I know, for something like this, shouldn't you make the class instance
itself go into shared storage instead instead of all the members?


I have no idea if shared on the class makes all it's parts shared.
It was a struct but I had problems with passing it to the spawn func when I
was using std.concurrent .

I'm trying to port a cassandra cql library from go so it's really just to
help with that.

I'm having a hard time imagining how to implement select from go. Could
probably use std.concurrent.receive because that can handle multiple types
at once.

Sorry I'm rambling...



 Aug 24 2013



prev sibling  parent reply "David Nadlinger" <code klickverbot.at>  writes:


On Friday, 23 August 2013 at 23:54:55 UTC, Rory McGuire wrote:

 So I'm porting so #golang code to #dlang and there is all these 
 blasted
 "go" statements.So I thought I'd give implmenting it in D a 
 shot. What do you guys think?


It's a cute idea, but not really practical, I'm afraid – 
Goroutines are managed by a scheduler in the Go runtime library, 
whereas D threads directly map to OS threads. Thus, if the Go 
application you are porting uses many Goroutines (and the Go code 
I've seen usually does so very liberally), the performance of the 
D equivalent is going to be horrible.

David



 Aug 24 2013



  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


David Nadlinger:


 It's a cute idea, but not really practical, I'm afraid – 
 Goroutines are managed by a scheduler in the Go runtime 
 library, whereas D threads directly map to OS threads.


Can't Rory McGuire add a scheduler to his code? How much code 
does it take?

Bye,
bearophile



 Aug 24 2013



 next sibling parent  "Jonas Drewsen" <nospam4321 hotmail.com >  writes:


On Saturday, 24 August 2013 at 13:26:32 UTC, bearophile wrote:

 David Nadlinger:


 It's a cute idea, but not really practical, I'm afraid – 
 Goroutines are managed by a scheduler in the Go runtime 
 library, whereas D threads directly map to OS threads.


 Can't Rory McGuire add a scheduler to his code? How much code 
 does it take?


It would be very nice to have a builtin scheduler in the runtime. 
It would make async non-threaded programming so much nicer in D.

/Jonas



 Aug 24 2013



prev sibling next sibling parent reply Rory McGuire <rjmcguire gmail.com>  writes:


On Sat, Aug 24, 2013 at 3:26 PM, bearophile <bearophileHUGS lycos.com>wrote=
:


 David Nadlinger:


  It's a cute idea, but not really practical, I'm afraid =E2=80=93 Gorouti=


nes are

 managed by a scheduler in the Go runtime library, whereas D threads
 directly map to OS threads.


 Can't Rory McGuire add a scheduler to his code? How much code does it tak=


e?

 Bye,
 bearophile


I imagine that it will be fine on Linux because threads truly are
lightweight on Linux, but its not going to be great on windows.

Go's scheduler is basically the same as the way vibe.d works. Fibers. I
haven't tried it but I could use fibers to do the same sort of thing,
though it would be easiest to use in Vibe.d because even in go you have to
do something that will wait/sleep in order to have your program advance
unless you set GOPROCS to  a value greater than 1. Go's scheduler is not
preemptive so there are the equivalent of Fiber.yield() spread throughout
the standard library, I think.

I'm not sure how threads + fibers would have to interact in such a system.



 Aug 24 2013



  parent reply "Paulo Pinto" <pjmp progtools.org>  writes:


On Saturday, 24 August 2013 at 20:03:58 UTC, Rory McGuire wrote:

 On Sat, Aug 24, 2013 at 3:26 PM, bearophile 
 <bearophileHUGS lycos.com>wrote:


 David Nadlinger:


  It's a cute idea, but not really practical, I'm afraid – 
 Goroutines are

 managed by a scheduler in the Go runtime library, whereas D 
 threads
 directly map to OS threads.


 Can't Rory McGuire add a scheduler to his code? How much code 
 does it take?

 Bye,
 bearophile


 I imagine that it will be fine on Linux because threads truly 
 are
 lightweight on Linux, but its not going to be great on windows.


Funny, I always thought otherwise, because Windows only has 
threads.

Processes are just a means of grouping threads on Windows, as 
there
isn't the distinction between threads and processes that UNIX 
systems used to make.

Then again, I lost track how the performance on Linux systems 
changed across the whole Processes -> LinuxThreads -> NPTL -> 
Posix Threads evolution.

--
Paulo



 Aug 24 2013



  parent  Rory McGuire <rjmcguire gmail.com>  writes:


On 24 Aug 2013 23:55, "Paulo Pinto" <pjmp progtools.org> wrote:

 Funny, I always thought otherwise, because Windows only has threads.

 Processes are just a means of grouping threads on Windows, as there
 isn't the distinction between threads and processes that UNIX systems


used to make.

 Then again, I lost track how the performance on Linux systems changed


across the whole Processes -> LinuxThreads -> NPTL -> Posix Threads
evolution.

 --
 Paulo


Just did a quick search to see if I was right. I'm not sure if it's still
the case but it appears that is actually the scheduler that causes windows
to be much slower than Linux.

Ideally one would want a hybrid of threads and fibres anyway. I wonder how
much of the standard library would need to change.



 Aug 25 2013



prev sibling  parent reply Rory McGuire <rjmcguire gmail.com>  writes:


On Sat, Aug 24, 2013 at 3:26 PM, bearophile <bearophileHUGS lycos.com>wrote=
:


 David Nadlinger:


  It's a cute idea, but not really practical, I'm afraid =E2=80=93 Gorouti=


nes are

 managed by a scheduler in the Go runtime library, whereas D threads
 directly map to OS threads.


 Can't Rory McGuire add a scheduler to his code? How much code does it tak=


e?

 Bye,
 bearophile



Here is a basic co-operative scheduler based version of go!(F)  that uses a
channel to keep things simple, its co-op so Fiber.yield has to be called at
some point. If the channel detects its in a Fiber it calls yield if there
is nothing available in it:

void go(alias F)() {
    scheduler._ =3D new Fiber(F);
}

shared chan!Fiber scheduler; // channel contains Fibers waiting for their
time slice
static this () {
    if (scheduler is null) {
        scheduler =3D makeChan!Fiber(100);

        // create the workers
        auto goprocs =3D environment.get("GOPROCS");
        int num_threads =3D 1;
        if (goprocs !=3D null) {
            num_threads =3D to!int(goprocs);
        }
        foreach (i; 0..num_threads) {
            // create threads that process the live fibers
            auto t =3D new Thread(() {
                for (;;) {
                    auto fiber =3D scheduler._;
                    fiber.call();
                    if (fiber.state !=3D Fiber.State.TERM) {
                        scheduler._ (fiber);
                    }
                }
                });
            t.start();
        }

    }
}

Just need to figure out a way to detect when main() has exited.



 Aug 25 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 08/26/2013 12:55 AM, Rory McGuire wrote:

 shared chan!Fiber scheduler; // channel contains Fibers waiting for
 their time slice
 static this () {
      if (scheduler is null) {



You want 'shared static this' instead to avoid a race condition.



 Aug 25 2013



 next sibling parent  Rory McGuire <rjmcguire gmail.com>  writes:


Awesome thanks, thought what I did there was dodgy. It was really weird
when this() started multiple schedulers at least now I see the obvious
reason.

BTW: gist is at: https://gist.github.com/rjmcguire/6336931

Could someone point me at the correct linked list to use inside the
channel. I'd prefer to use a range container of some kind if it exists in
the std lib. I tried SList and had a bad experience hence the custom
implementation.



On Mon, Aug 26, 2013 at 1:21 AM, Timon Gehr <timon.gehr gmx.ch> wrote:


 On 08/26/2013 12:55 AM, Rory McGuire wrote:


 shared chan!Fiber scheduler; // channel contains Fibers waiting for
 their time slice
 static this () {
      if (scheduler is null) {



 You want 'shared static this' instead to avoid a race condition.





 Aug 25 2013



prev sibling  parent  Rory McGuire <rjmcguire gmail.com>  writes:


I've made the scheduler a bit more inteligent and the channel
implementation is now backed by a lock free queue.
https://github.com/rjmcguire/goport/blob/tip/wrapper2.d for a example using
libev

https://github.com/rjmcguire/goport/blob/tip/goroutine.d for the "go"
routines

https://github.com/rjmcguire/goport/blob/tip/channel.d channel
implementation, the channel still has a read lock because I didn't like
looping wasting cpu cycles/battery, I may change this to use the same loop
as the select(...) expression with a timeout that backs off.

https://github.com/rjmcguire/goport/blob/tip/concurrentlinkedqueue.d queue
implementation


there is a problem with the file EV handler in that it can get spurios READ
events and I can't seem to set it to non-blocking.

The select(...) implementation and usage is quite interesting because it
works very much like Go's select statement.


On Mon, Aug 26, 2013 at 1:28 AM, Rory McGuire <rjmcguire gmail.com> wrote:


 Awesome thanks, thought what I did there was dodgy. It was really weird
 when this() started multiple schedulers at least now I see the obvious
 reason.

 BTW: gist is at: https://gist.github.com/rjmcguire/6336931

 Could someone point me at the correct linked list to use inside the
 channel. I'd prefer to use a range container of some kind if it exists in
 the std lib. I tried SList and had a bad experience hence the custom
 implementation.



 On Mon, Aug 26, 2013 at 1:21 AM, Timon Gehr <timon.gehr gmx.ch> wrote:


 On 08/26/2013 12:55 AM, Rory McGuire wrote:


 shared chan!Fiber scheduler; // channel contains Fibers waiting for
 their time slice
 static this () {
      if (scheduler is null) {



 You want 'shared static this' instead to avoid a race condition.







 Sep 18 2013