www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DConf 2013 Day 1 Talk 3: Distributed Caching Compiler for D by Robert

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Watch, discuss, vote up!

http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


Andrei
May 13 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-05-13 14:01, Andrei Alexandrescu wrote:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


Awesome :)

-- 
/Jacob Carlborg
May 13 2013
prev sibling next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
Is there a way to find all the reddit links to these (not a frequent reddit
user, but I'm curious to look over the discussions each vid gets when I
have the chance).

--bb


On Mon, May 13, 2013 at 5:01 AM, Andrei Alexandrescu <
SeeWebsiteForEmail erdani.org> wrote:


 Watch, discuss, vote up!

 http://reddit.com/r/**programming/comments/1e8mwq/**
 dconf_2013_day_1_talk_3_**distributed_caching/<http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/>


 Andrei
May 13 2013
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Mon, 13 May 2013 10:01:03 -0700
Bill Baxter <wbaxter gmail.com> wrote:


 Is there a way to find all the reddit links to these (not a frequent
 reddit user, but I'm curious to look over the discussions each vid
 gets when I have the chance).
 


Added Reddit links here:

http://semitwist.com/download/misc/dconf2013/

Also added torrents for this new video.
May 13 2013
next sibling parent Bill Baxter <wbaxter gmail.com> writes:
Excellent.  Thanks!


On Mon, May 13, 2013 at 12:09 PM, Nick Sabalausky <
SeeWebsiteToContactMe semitwist.com> wrote:


 On Mon, 13 May 2013 10:01:03 -0700
 Bill Baxter <wbaxter gmail.com> wrote:


 Is there a way to find all the reddit links to these (not a frequent
 reddit user, but I'm curious to look over the discussions each vid
 gets when I have the chance).


 Added Reddit links here:

 http://semitwist.com/download/misc/dconf2013/

 Also added torrents for this new video.
May 13 2013
prev sibling parent reply "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-05-13, 21:09, Nick Sabalausky wrote:


 On Mon, 13 May 2013 10:01:03 -0700
 Bill Baxter <wbaxter gmail.com> wrote:


 Is there a way to find all the reddit links to these (not a frequent
 reddit user, but I'm curious to look over the discussions each vid
 gets when I have the chance).


 Added Reddit links here:

 http://semitwist.com/download/misc/dconf2013/

 Also added torrents for this new video.


The linky for original MP4 keynote torrent seems to be dead.

-- 
Simen
May 13 2013
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Mon, 13 May 2013 23:27:53 +0200
"Simen Kjaeraas" <simen.kjaras gmail.com> wrote:


 On 2013-05-13, 21:09, Nick Sabalausky wrote:
 

 Added Reddit links here:

 http://semitwist.com/download/misc/dconf2013/

 Also added torrents for this new video.


 
 The linky for original MP4 keynote torrent seems to be dead.
 


Thanks. Fixed.
May 13 2013
prev sibling next sibling parent "Maxim Fomin" <maxim maxim-fomin.ru> writes:
On Monday, 13 May 2013 at 12:01:54 UTC, Andrei Alexandrescu wrote:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


 Andrei


Is the download speed too low because the video is popular? :)
May 13 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Monday, 13 May 2013 at 12:01:54 UTC, Andrei Alexandrescu wrote:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


 Andrei


Very cool presentation.

Not sure if Robert visits news group, but maybe someone else have 
experience in this subject - I wonder how solution shown in 
presentation scales compared to similar products like 
ccache+distcc+make and j flag combo.

If it is much faster, maybe there would be value in bringing such 
functionality to our current compilers? DMD, GDC, LDC troika.

Best Regards
May 13 2013
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 13.05.2013 14:01, schrieb Andrei Alexandrescu:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


 Andrei


it seems that the http://www.phoronix.com/ guys miss the dconf 
completely - they write about rust, go, D in gcc, kernel udpates etc. - 
but absolutely nothing about dconf :(
May 13 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/13/2013 10:19 PM, dennis luehring wrote:

 Am 13.05.2013 14:01, schrieb Andrei Alexandrescu:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/



 Andrei


 it seems that the http://www.phoronix.com/ guys miss the dconf completely -
they
 write about rust, go, D in gcc, kernel udpates etc. - but absolutely nothing
 about dconf :(


Email them about it!
May 13 2013
parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 14.05.2013 08:18, schrieb Walter Bright:

 On 5/13/2013 10:19 PM, dennis luehring wrote:

 Am 13.05.2013 14:01, schrieb Andrei Alexandrescu:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/



 Andrei


 it seems that the http://www.phoronix.com/ guys miss the dconf completely -
they
 write about rust, go, D in gcc, kernel udpates etc. - but absolutely nothing
 about dconf :(


 Email them about it!


is there something like an overview page with slides, videos etc.? 
dconf.org - could someone change the main page to reflect the success of 
the conference :)
May 14 2013
parent =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 05/14/2013 12:25 AM, dennis luehring wrote:


 is there something like an overview page with slides, videos etc.?


The schedule page has links to both the slides and the videos:

   http://dconf.org/schedule/index.html

Individual presentation pages too:

   http://dconf.org/talks/schadek.html

Ali
May 14 2013
prev sibling next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
On Monday, 13 May 2013 at 12:01:54 UTC, Andrei Alexandrescu wrote:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


 Andrei


On promising consequence of such compiler daemonization I have 
always had in mind since got familiar with the concept is that 
you can have convenience of separate compilation system AND speed 
of single step compilation at the same time, even if no stuff is 
actually distributed network-wise.
May 14 2013
parent reply "Guillaume Chatelet" <chatelet.guillaume gmail.com> writes:
 On promising consequence of such compiler daemonization I have 
 always had in mind since got familiar with the concept is that 
 you can have convenience of separate compilation system AND 
 speed of single step compilation at the same time, even if no 
 stuff is actually distributed network-wise.


Dicebot, Sönke, how about a vibe.d based compiler ? :P

Compilation stages could be pipelined/distributed through 
localhost or distant url calls (of course it shouldn't go through 
the network stack in case of local calls). We could even make use 
of resource caching to speedup the compilation. Maybe it's a 
crazy idea but I see vibe.d as an excellent framework for a 
parallel compiler (local and distributed). Also it becomes almost 
trivial to distribute over a farm if you see a compilation stage 
as a web resource. Lexing could be streamed to a Parsing url. 
Semantic analysis could be done in parallel and load balanced, 
Linking of course... well... linking is the brutally sequential 
part - nothing fancy here. I'm not familiar enough with vibe.d to 
see if it could work, it's just an idea.

The RestFul model act as an interface and allows for thinking in 
terms of services (or components) and abstracts away the 
execution location as well as the implementation. Each url could 
be seen as a pure function, allowing for caching (memoizing). And 
with no state (restful) : distribution becomes almost free. It's 
a very simplistic view of a compiler and Walter will surely have 
myriads of objections, but I wanted to share my thoughts. This 
kind of infrastructure could pay on massive codebase compilation 
( like at Facebook or Google )

--
Guillaume
May 14 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Tuesday, 14 May 2013 at 19:57:12 UTC, Guillaume Chatelet wrote:

 Dicebot, Sönke, how about a vibe.d based compiler ? :P


I don't think networking part of of distributed compiler will be 
that demanding :) It is more about good pipe-lining, like 
described in Robert presentation. Concurrency level is always low 
and network speed tend to be much faster than compilation speed.

I'd like someone more proficient in this topic to correct me, but 
once compiler is daemonized and decent data model / pipe line is 
defined, adding distributed networking functionality on top is 
one of the easiest parts.
May 15 2013
parent "Dicebot" <m.strashun gmail.com> writes:
P.S. And even easier if std.concurrency starts to work over
network sockets :P
May 15 2013
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
13-May-2013 16:01, Andrei Alexandrescu пишет:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3_distributed_caching/


Nice one! Reminds me the gory days of our "compiler technologies" course :)

Our professor seemed obsessed with tabulated methods as well (Dragon 
book etc.) And above all - reverse polish notation as the one and holy 
AST representation. Don't you try to say recursive descent parser in the 
class...

Speaking of flat AST point in the talk  - reverse-polish notation is 
kind of neat in that it does represent flat AST and is easy to walk.
So one need not to have 2-array AST (childs + map to where are these).

Another advantage - you can slice off a sub-tree as simple as a chunk of 
an array! It should be good for CPU caches (all related stuff is in one 
block, no pointer chasing), but I never was close enough to a real 
compiler to run any meaningful benchmarks.

Let's see the idea of AST as polish-notation array applied to the tree 
in the talk. I show the forward one as it easier to read and less tricky 
to walk.

Using the following notation for each node:
<arity>, array of offsets * arity, type code+contents |  subtree(s)

Data is laid out byte-by-byte in raw-binary.
Offsets are in bytes starting right after type+contents.

So e.g. leaf node of int would be:
0, int
No offsets nor sub-trees, content/type tag etc. is marked as int

Then whole tree (tried to align for reading):

1, 0, S | 1, 0, DeclDefs |
   3, 0, <x>, <y>, Declarator |
     1, 0, BasicType | 0, int
     1, 0, Identifier | 0, main
     1, 0, DeclDefs | 1, 0, DeclDef | 1, 0 ReturnStatement | 0, "1"


<x> and <y> are sizes of sub-trees and can easily be calculated at 
construction. Insertion is simple array insertion + fix ups of offsets 
up the tree (you have to know your path in nodes anyway to back up).

Giving up on random access to child nodes would allow us to drop offset 
table completely. I have limited experience on semantic analysis to tell 
if it makes sense to require or drop it.


Will comment on the DFA & Unicode some time later, it's a neat topic in 
its own right.

-- 
Dmitry Olshansky
May 14 2013
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 14 May 2013 17:43:46 +0400
Dmitry Olshansky <dmitry.olsh gmail.com> wrote:


 13-May-2013 16:01, Andrei Alexandrescu =D0=BF=D0=B8=D1=88=D0=B5=D1=82:

 Watch, discuss, vote up!

 http://reddit.com/r/programming/comments/1e8mwq/dconf_2013_day_1_talk_3=




_distributed_caching/


=20
 Nice one! Reminds me the gory days of our "compiler technologies"
 course :)
=20


Yea, I found it to be a fascinating talk, too.


 Our professor seemed obsessed with tabulated methods as well (Dragon=20
 book etc.)


I'm quite partial to "Crafting a Compiler"
<http://www.pearsonhighered.com/educator/product/Crafting-A-Compiler/978013=
6067054.page>.

I'm not sure it's quite as complete as the dragon book, but it's
somewhat more programmer-friendly and much easier to understand,
whereas the dragon book (as well as all CS theory books that I've seen)
is geared specifically to CS researchers and mathemeticians.



=20
 So e.g. leaf node of int would be:
 0, int
 No offsets nor sub-trees, content/type tag etc. is marked as int
=20
 Then whole tree (tried to align for reading):
=20
 1, 0, S | 1, 0, DeclDefs |
    3, 0, <x>, <y>, Declarator |
      1, 0, BasicType | 0, int
      1, 0, Identifier | 0, main
      1, 0, DeclDefs | 1, 0, DeclDef | 1, 0 ReturnStatement | 0, "1"
=20


Interesting.


=20
 Will comment on the DFA & Unicode some time later, it's a neat topic
 in its own right.
=20


Somewhat of a naive approach, but transitioning on character ranges
instead of characters will drastically reduce the storage requirements
for generalized unicode lexer, at least for any remotely typical lexer
(there can be some pretty bad worst-case-scenarios, but nobody ever
defines a token like [acegikmoq...]+).

Although I imagine it's not nearly as fast as a pure table since you
have those extra conditionals for each character of input. FWIW.



 May 14 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 5/14/13 4:12 PM, Nick Sabalausky wrote:
[snip]

This is a great discussion - I encourage you all to hold future 
discussions on DConf topics to the associated reddit pages for greater 
visibility and participation.

Tomorrow morning comes another one - Ben's first talk!


Thanks,

Andrei



 May 14 2013



 next sibling parent  =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com>  writes:


On 05/14/2013 02:16 PM, Andrei Alexandrescu wrote:


 Tomorrow morning comes another one - Ben's first talk!


Has he been fooling us or did he forget? :) Ben had given an excellent 
talk for the ACCU Silicon Valley about 6 years ago (as well as the 
repeat of his DConf talk just last week):

   http://accu.org/index.php/accu_branches/accu_usa/past

Ali



 May 14 2013



prev sibling  parent  Iain Buclaw <ibuclaw ubuntu.com>  writes:


On 14 May 2013 22:16, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:


 On 5/14/13 4:12 PM, Nick Sabalausky wrote:
 [snip]

 This is a great discussion - I encourage you all to hold future
 discussions on DConf topics to the associated reddit pages for greater
 visibility and participation.

 Tomorrow morning comes another one - Ben's first talk!


Ben did not talk as if it was his first talk... :o)

-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';



 May 14 2013