www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Regex in ctfe?

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
Possible to std.regex in ctfe?
I kinda had it in my mind that ctRegex was ctfe-able, but it seems
it's not. It just generates an efficient regex engine at compile time
which only actually works on data at runtime.

I have a string import destined for a mixin, and I want to parse it
with regex, but I haven't been able to make it work. The docs mention
nothing about this possibility.
Jan 25
next sibling parent reply Era Scarecrow <rtcvb32 yahoo.com> writes:
  What exactly are you trying to do i wonder?

  If it uses enough of a subset then maybe... I'll assume you 
tried and failed already though.

  It seems more likely that a external library like PCRE would be 
compiled into the compiler instead and you could hook into it; 
But with probable mismatches of the Regex engine and features I'd 
wonder if it would cause problems (at least until the compiler is 
converted to D and can implement it's own d libraries).
Jan 25
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 26 January 2016 at 12:58, Era Scarecrow via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
  What exactly are you trying to do i wonder?

  If it uses enough of a subset then maybe... I'll assume you tried and
 failed already though.

  It seems more likely that a external library like PCRE would be compiled
 into the compiler instead and you could hook into it; But with probable
 mismatches of the Regex engine and features I'd wonder if it would cause
 problems (at least until the compiler is converted to D and can implement
 it's own d libraries).
I thought DMD was fully converted to D? The compiler could make use of std.regex if it wanted to...
Jan 25
parent reply Era Scarecrow <rtcvb32 yahoo.com> writes:
On Tuesday, 26 January 2016 at 05:10:58 UTC, Manu wrote:
 I thought DMD was fully converted to D? The compiler could make 
 use of std.regex if it wanted to...
My experience is at least a year or two out of date; One of the DConf2015 talks someone was working on a C++ to D converter, and planned to swap over from C++ to D once it was complete enough. If that already happened, i was unaware of it.
Jan 26
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 26 January 2016 at 11:35:24 UTC, Era Scarecrow wrote:
 On Tuesday, 26 January 2016 at 05:10:58 UTC, Manu wrote:
 I thought DMD was fully converted to D? The compiler could 
 make use of std.regex if it wanted to...
My experience is at least a year or two out of date; One of the DConf2015 talks someone was working on a C++ to D converter, and planned to swap over from C++ to D once it was complete enough. If that already happened, i was unaware of it.
Yes the swap happend. In a recent git checkout you will have the cpp files and the .d files side by side. matching regex at ctfe should work with a bit of modification.
Jan 26
parent reply w0rp <devw0rp gmail.com> writes:
Unless I'm mistaken, I think the compiler for regex currently 
works at compile time, but not the matcher. Maybe someone who 
knows the module could add support for that.
Jan 26
parent reply Pierre Krafft <kpierre+dlang outlook.com> writes:
On Tuesday, 26 January 2016 at 12:47:26 UTC, w0rp wrote:
 Unless I'm mistaken, I think the compiler for regex currently 
 works at compile time, but not the matcher. Maybe someone who 
 knows the module could add support for that.
That's correct. I looked in to this a while ago and found out that the matcher uses trusted code with pointers. Getting support for ctfe would require a rewrite of most of the matching engine and would probably result in a rather big performance hit.
Jan 27
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 27, 2016 at 06:47:59PM +0000, Pierre Krafft via Digitalmars-d wrote:
 On Tuesday, 26 January 2016 at 12:47:26 UTC, w0rp wrote:
Unless I'm mistaken, I think the compiler for regex currently works
at compile time, but not the matcher. Maybe someone who knows the
module could add support for that.
That's correct. I looked in to this a while ago and found out that the matcher uses trusted code with pointers. Getting support for ctfe would require a rewrite of most of the matching engine and would probably result in a rather big performance hit.
An alternative would be to use `if (_ctfe)` branches in the code to switch to a different matching engine when in CTFE, and leaving the runtime code untouched. T -- Don't throw out the baby with the bathwater. Use your hands...
Jan 27
parent reply Era Scarecrow <rtcvb32 yahoo.com> writes:
On Wednesday, 27 January 2016 at 19:05:01 UTC, H. S. Teoh wrote:
 An alternative would be to use `if (_ctfe)` branches in the 
 code to switch to a different matching engine when in CTFE, and 
 leaving the runtime code untouched.
Hmmm... As I recall there's 2 major engines that run Regex (DFA & NFA). If you don't need the more complex engine (which has forward/back referencing, named capturing, etc) then maybe the simpler engine would work. But it would seem easier/simpler if the compiler just hooks in the Regex that's compiled into it instead and passes the data back and forth. It does mean you can't test Regex changes during the CTFE stage (assuming your actively changing/building it) but it wouldn't have the performance hit either.
Jan 27
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 27, 2016 at 07:54:21PM +0000, Era Scarecrow via Digitalmars-d wrote:
 On Wednesday, 27 January 2016 at 19:05:01 UTC, H. S. Teoh wrote:
An alternative would be to use `if (_ctfe)` branches in the code to
switch to a different matching engine when in CTFE, and leaving the
runtime code untouched.
Hmmm... As I recall there's 2 major engines that run Regex (DFA & NFA). If you don't need the more complex engine (which has forward/back referencing, named capturing, etc) then maybe the simpler engine would work. But it would seem easier/simpler if the compiler just hooks in the Regex that's compiled into it instead and passes the data back and forth. It does mean you can't test Regex changes during the CTFE stage (assuming your actively changing/building it) but it wouldn't have the performance hit either.
Currently, AFAIK, (d)dmd does not use anything from Phobos. I'm not sure if this is a temporary situation, or there's a strong reason for it. T -- Those who don't understand D are condemned to reinvent it, poorly. -- Daniel N
Jan 27
parent reply Era Scarecrow <rtcvb32 yahoo.com> writes:
On Wednesday, 27 January 2016 at 20:25:33 UTC, H. S. Teoh wrote:
 On Wed, Jan 27, 2016 at 07:54:21PM +0000, Era Scarecrow via 
 Digitalmars-d wrote:
  But it would seem easier/simpler if the compiler just hooks 
 in the  Regex that's compiled into it instead and passes the 
 data back and forth. It does mean you can't test Regex changes 
 during the CTFE stage (assuming your actively 
 changing/building it) but it wouldn't  have the performance 
 hit either.
Currently, AFAIK, (d)dmd does not use anything from Phobos. I'm not sure if this is a temporary situation, or there's a strong reason for it.
Probably since it was so recently changed from C++ to D that it just hasn't tried to include anything. And leaving the two projects (library and compiler) separate does allow a certain level of purity.
Jan 27
parent tsbockman <thomas.bockman gmail.com> writes:
On Wednesday, 27 January 2016 at 21:09:19 UTC, Era Scarecrow 
wrote:
 On Wednesday, 27 January 2016 at 20:25:33 UTC, H. S. Teoh wrote:
 Currently, AFAIK, (d)dmd does not use anything from Phobos. 
 I'm not sure if this is a temporary situation, or there's a 
 strong reason for it.
Probably since it was so recently changed from C++ to D that it just hasn't tried to include anything. And leaving the two projects (library and compiler) separate does allow a certain level of purity.
This was a deliberate decision to avoid creating circular dependencies between the compiler and Phobos, which are arguably too tightly coupled already. (I remember seeing it discussed somewhere around the time of the transition to DDMD; I'm too lazy to dig up a link right now.)
Jan 31
prev sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Tuesday, 26 January 2016 at 01:34:09 UTC, Manu wrote:
 Possible to std.regex in ctfe?
(...)
 I have a string import destined for a mixin, and I want to 
 parse it with regex, but I haven't been able to make it work. 
 The docs mention nothing about this possibility.
Hi Manu, a possible solution for that would be to use my Pegged (https://github.com/PhilippeSigaud/Pegged) project. It's a parser generator that creates compile-time (and runtime) parsers. It's not directly compatible with Phobos regexes, but it's quite easy to extract the matched strings. Bastiaan Veelo recently added support for left-recursive grammars, which means we can now generate parsers for many more grammars (although if your parsing need is covered by regexes, that probably won't interest you...) Philippe
Jan 31