www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - What's going on with std.experimental.lexer?

reply "Brian Schott" <briancschott gmail.com> writes:
I've been looking at ways to optimize the D lexer's operation 
using SIMD instructions. I'm not yet sure if I'll need to change 
the lexer generator's API to do this. I'm going to wait until I 
have my proof-of-concept code and some benchmark results before 
asking for a voting thread or creating a pull request.
Jun 04 2014
next sibling parent "Philpax" <phillip philliplarkson.com> writes:
I've been meaning to mention this, but I use 
std.experimental.lexer in a code generation tool for our project; 
it's worked well so far, and I'd happily recommend its use. 
Looking forward to further updates.
Jun 05 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.

I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion.
Jun 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/14, 2:15 PM, Dejan Lekic wrote:
 On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote:
 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
 I've been looking at ways to optimize the D lexer's operation using
 SIMD instructions. I'm not yet sure if I'll need to change the lexer
 generator's API to do this. I'm going to wait until I have my
 proof-of-concept code and some benchmark results before asking for a
 voting thread or creating a pull request.

I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion.

I am more for "stdx", which is what some developers already use as package name for experimental stuff.

The way I see it is instead of explaining "stdx stands for the experimental part of std" I'd rather use std.experimental which explains itself. We added "immutable" after having explained to a million people "invariant in this context means immutable". Andrei
Jun 09 2014
parent David Gileadi <gileadis NSPMgmail.com> writes:
On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote:
 I agree all the way with std.experimental as the package name.  Though
 I might throw in an alternative argument to stdx and instead promote
 unsafe.* or std.unsafe.  ;-)

The only issue I see with *.unsafe.* is that it sounds related to safety. So far I haven't seen a better name than std.experimental.
Jun 10 2014
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 13.06.2014 16:59, schrieb Dejan Lekic:
 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all those
 reasons
 already stated.

What's wrong with javax?

experimental is 100% clear and simple to understand beeing evil javax was interpreted as eXtendet or eXtra or whatever, so people have no problem in using experimental stuff all the while
Jun 13 2014
prev sibling next sibling parent reply "Brian Schott" <briancschott gmail.com> writes:
On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.

Implementing some SIMD code just in the lexWhitespace function causes a drop in total lexing time of roughly 3.7%. This looks promising so far, so I'm going to implement similar code in lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, and lexSlashPlusComment. So far this only uses SSE2 instructions, so all 64-bit x86 processors can benefit from this. (SSE4.2 would allow us to do more interesting things, but it's not as common)
Jun 05 2014
next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 07.06.2014 01:50, schrieb Brian Schott:
 On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote:
 Implementing some SIMD code just in the lexWhitespace function
 causes a drop in total lexing time of roughly 3.7%. This looks
 promising so far, so I'm going to implement similar code in
 lexStringLiteral, lexSlashStarComment, lexSlashSlashComment,
 and lexSlashPlusComment.

Some moe numbers: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that.

thats why im always puzzled when people start to optimze algorithms based on DMD results - currently one should always compare any results before optimization with GDC/LDC
Jun 08 2014
parent Martin Nowak <code dawg.eu> writes:
On 06/08/2014 08:21 PM, dennis luehring wrote:
 thats why im always puzzled when people start to optimze algorithms
 based on DMD results - currently one should always compare any results
 before optimization with GDC/LDC

I second that, dmd results are easily misleading as you often optimize around backend deficiencies.
Jun 08 2014
prev sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 09.06.2014 22:21, schrieb Brian Schott:
 On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote:
 SIMD reduces execution time by 5.15% with DMD.
 Compiling the non-SIMD code with GDC reduces execution time by
 42.39%.

 So... There's that.

Changing the code generator to output a set of if statements that implements a binary search did more-or-less nothing with the DMD timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled version executes in 51% of the time that the DMD-compiled version does)

the LLVM optimizer is also very good (sometimes far better then the gcc one) - what are your LDC timings?
Jun 09 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 05 Jun 2014 10:57:35 +0000
Dicebot via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
 I've been looking at ways to optimize the D lexer's operation
 using SIMD instructions. I'm not yet sure if I'll need to
 change the lexer generator's API to do this. I'm going to wait
 until I have my proof-of-concept code and some benchmark
 results before asking for a voting thread or creating a pull
 request.

I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion.

Yeah. std.experimental really should be for stuff that's gone through the review process and is essentially ready for inclusion in Phobos but needs more real world usage to make sure that it really is fully ready to be merged into std and then have its API frozen. It really doesn't make any sense to put anything that's in more flux than that in there, because the release cycle is just way too long for it. Now, Brian's lexer might effectively be at that point already, I don't know. It's been reviewed before, but I generally haven't had much time to look it over for those reviews and don't know what state it's currently in. So, if its API is close enough to final at this point, then it could probably be put in std.experimental, but if it's still in a lot of flux, then that probably isn't appropriate. We do need to figure out what the official process is on this though so that it's clear when it's appropriate for something to go in std.experimental, otherwise we'll have folks who keep trying to put stuff in there which really isn't ready yet, and we'll keep end up arguing over what can and can't go in there. - Jonathan M Davis
Jun 06 2014
prev sibling next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Friday, 6 June 2014 at 00:33:23 UTC, Brian Schott wrote:
 Implementing some SIMD code just in the lexWhitespace function 
 causes a drop in total lexing time of roughly 3.7%. This looks 
 promising so far, so I'm going to implement similar code in 
 lexStringLiteral, lexSlashStarComment, lexSlashSlashComment, 
 and lexSlashPlusComment.

Some moe numbers: SIMD reduces execution time by 5.15% with DMD. Compiling the non-SIMD code with GDC reduces execution time by 42.39%. So... There's that.
Jun 06 2014
prev sibling next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Friday, 6 June 2014 at 23:50:40 UTC, Brian Schott wrote:
 SIMD reduces execution time by 5.15% with DMD.
 Compiling the non-SIMD code with GDC reduces execution time by 
 42.39%.

 So... There's that.

Changing the code generator to output a set of if statements that implements a binary search did more-or-less nothing with the DMD timings, but brought GDC's lead up to 49%. (i.e. the GDC-compiled version executes in 51% of the time that the DMD-compiled version does)
Jun 09 2014
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Thursday, 5 June 2014 at 10:57:37 UTC, Dicebot wrote:
 On Wednesday, 4 June 2014 at 21:12:25 UTC, Brian Schott wrote:
 I've been looking at ways to optimize the D lexer's operation 
 using SIMD instructions. I'm not yet sure if I'll need to 
 change the lexer generator's API to do this. I'm going to wait 
 until I have my proof-of-concept code and some benchmark 
 results before asking for a voting thread or creating a pull 
 request.

I still thing we should use it more like `std.staging` - once your updates are ready, go through review/voting and keep module in `std.experimental` for at least one DMD release before adding to Phobos core. This also means relaxing API requirements a lot for initial inclusion.

I am more for "stdx", which is what some developers already use as package name for experimental stuff.
Jun 09 2014
prev sibling next sibling parent Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a11c258dc1ee0fe04fb6e47df
Content-Type: text/plain; charset=UTF-8

On Mon, Jun 9, 2014 at 2:15 PM, Dejan Lekic via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 I am more for "stdx", which is what some developers already use as package
 name for experimental stuff.

Please no. See: javax Spelling out 'experimental' is probably the best, for all those reasons already stated. --001a11c258dc1ee0fe04fb6e47df Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On M= on, Jun 9, 2014 at 2:15 PM, Dejan Lekic via Digitalmars-d <span dir=3D"ltr"=
&lt;<a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">digit=

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">I am more for &quot;stdx&quot;, which is wha= t some developers already use as package name for experimental stuff.<br> </blockquote></div><br></div><div class=3D"gmail_extra">Please no. =C2=A0Se= e: javax</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extr= a">Spelling out &#39;experimental&#39; is probably the best, for all those = reasons already stated.</div> </div> --001a11c258dc1ee0fe04fb6e47df--
Jun 09 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 09 Jun 2014 14:23:59 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 6/9/14, 2:15 PM, Dejan Lekic wrote:

 I am more for "stdx", which is what some developers already use as
 package name for experimental stuff.

The way I see it is instead of explaining "stdx stands for the experimental part of std" I'd rather use std.experimental which explains itself. We added "immutable" after having explained to a million people "invariant in this context means immutable".

That and the fact that we'd like to avoid the javax disaster. std.experimental _should_ be annoyingly long and obvious. We want people to use it, but it needs to be clear that it's temporary and _experimental_ until it actually ends up in std. - Jonathan M Davis
Jun 09 2014
prev sibling next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2014 07:26, Jonathan M Davis via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Mon, 09 Jun 2014 14:23:59 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 On 6/9/14, 2:15 PM, Dejan Lekic wrote:

 I am more for "stdx", which is what some developers already use as
 package name for experimental stuff.

The way I see it is instead of explaining "stdx stands for the experimental part of std" I'd rather use std.experimental which explains itself. We added "immutable" after having explained to a million people "invariant in this context means immutable".

That and the fact that we'd like to avoid the javax disaster. std.experimental _should_ be annoyingly long and obvious. We want people to use it, but it needs to be clear that it's temporary and _experimental_ until it actually ends up in std.

I agree all the way with std.experimental as the package name. Though I might throw in an alternative argument to stdx and instead promote unsafe.* or std.unsafe. ;-)
Jun 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 10 Jun 2014 10:31:59 -0400, David Gileadi <gileadis nspmgmail.com>  
wrote:

 On 6/10/14, 3:57 AM, Iain Buclaw via Digitalmars-d wrote:
 I agree all the way with std.experimental as the package name.  Though
 I might throw in an alternative argument to stdx and instead promote
 unsafe.* or std.unsafe.  ;-)

The only issue I see with *.unsafe.* is that it sounds related to safety. So far I haven't seen a better name than std.experimental.

If it weren't a keyword, volatile would be a very good package name for this. -Steve
Jun 10 2014
prev sibling next sibling parent Tom Browder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 I've been looking at ways to optimize the D lexer's operation using SIMD

Brian, i just found the lexer code repo (and fixed the broken code link on the wiki), but the review thread on the wiki looks very old. The wiki page: http://wiki.dlang.org/Review_Queue The review thread link: http://forum.dlang.org/post/aiipgaqyddjijpjiulfu forum.dlang.org Thanks. Best regards, -Tom
Jun 13 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 13 June 2014 at 12:01:00 UTC, Tom Browder via 
Digitalmars-d wrote:
 On Wed, Jun 4, 2014 at 4:12 PM, Brian Schott via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I've been looking at ways to optimize the D lexer's operation 
 using SIMD

Brian, i just found the lexer code repo (and fixed the broken code link on the wiki), but the review thread on the wiki looks very old. The wiki page: http://wiki.dlang.org/Review_Queue The review thread link: http://forum.dlang.org/post/aiipgaqyddjijpjiulfu forum.dlang.org Thanks. Best regards, -Tom

Last formal review has happened quite some time ago, this is correct. Much has been done since then. Anyway, I am ready to proceed with it as soon as Brian is.
Jun 13 2014
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all those 
 reasons
 already stated.

What's wrong with javax?
Jun 13 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 13 June 2014 at 14:59:55 UTC, Dejan Lekic wrote:
 Please no.  See: javax

 Spelling out 'experimental' is probably the best, for all 
 those reasons
 already stated.

What's wrong with javax?

The fact that it started as same experimental package but stuff there never got moved to main namespace or actually changed in breaking fashion. Because many people missed "experimental" part and started using it as normal standard library package.
Jun 13 2014