www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - earthquake changes of std.regexp to come

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I'm quite unhappy with the API of std.regexp. It's a chaotic design that 
provides a hodgepodge of functionality and tries to use as many synonyms 
of "to find" in the dictionary (e.g. search, match). I could swear 
Walter never really cared for using regexps, and that is felt throughout 
the design: it fills the bullet point but it's asinine to use.

Besides std.regexp only works with (narrow) strings and we want it to 
work on streams of all widths and structures. One pet complaint I have 
is that std.regexp puts a class around it all as if everybody's favorite 
pastime would be to inherit Regexp and override some random function in it.

In the upcoming releases of D 2.0 there will be rather dramatic breaking 
changes of phobos. I just wanted to ask whether y'all could stomach yet 
another rewritten API or you'd rather use std.regexp as it is for the 
time being.


Andrei
Feb 17 2009
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Don't be too much hard with the good Walter, please :-) One good thing in his
designs (in D1) is that they are often simple to use: they give you back much
more than you give them. D2 seems to ask much more from the programmer.

I agree that the API of regexes in Phobos is not much good, but I think
designing a good API for it is quite hard.


 I just wanted to ask whether y'all could stomach yet 
 another rewritten API or you'd rather use std.regexp as it is for the 
 time being.
I have no problems in accepting changes here too. D2 is already essentially another language compared to D1. Regarding regexes of D1 Phobos, it has problems bigger than just the API, in the past I have found some common cases where it is O(n^2) or more. You can see a case of such behaviours here (look at my comments that show what parts are slow, I have also commented out versions that more logical but much slower): http://shootout.alioth.debian.org/debian/benchmark.php?test=regexdna&lang=gdc&id=4 If you want to test that code you can generate test data with this other code: http://shootout.alioth.debian.org/debian/benchmark.php?test=fasta&lang=dlang&id=1 Bye, bearophile
Feb 17 2009
parent reply "Joel C. Salomon" <joelcsalomon gmail.com> writes:
bearophile wrote:
 I agree that the API of regexes in Phobos is not much good, but I think
designing a good API for it is quite hard.
So steal one, rather than invent something new. My suggestion would be to expose the DFA object, as in Plan 9’s library (documentation at <http://plan9.bell-labs.com/magic/man2html/2/regexp>, implementation at <http://plan9.bell-labs.com/sources/plan9/sys/src/libregexp/>, discussion and links to a Unix implementation at <http://swtch.com/~rsc/regexp/>). Simple API: • regcomp: Compile a regexp DFA; • regexec: Apply it to a string, returning a slice of the string that matches the first hit (or an array of slices if parenthesized expressions are used); and • regsub: Apply substitutions to subexpressions of the matching slice. —Joel Salomon
Feb 17 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Joel C. Salomon wrote:
 bearophile wrote:
 I agree that the API of regexes in Phobos is not much good, but I think
designing a good API for it is quite hard.
So steal one, rather than invent something new. My suggestion would be to expose the DFA object, as in Plan 9’s library (documentation at <http://plan9.bell-labs.com/magic/man2html/2/regexp>, implementation at <http://plan9.bell-labs.com/sources/plan9/sys/src/libregexp/>, discussion and links to a Unix implementation at <http://swtch.com/~rsc/regexp/>). Simple API: • regcomp: Compile a regexp DFA; • regexec: Apply it to a string, returning a slice of the string that matches the first hit (or an array of slices if parenthesized expressions are used); and
s/string/input range/ Also returning a range instead of an array of slices is more flexible. Andrei
Feb 17 2009
prev sibling next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 I'm quite unhappy with the API of std.regexp. It's a chaotic design that
 provides a hodgepodge of functionality and tries to use as many synonyms
 of "to find" in the dictionary (e.g. search, match). I could swear
 Walter never really cared for using regexps, and that is felt throughout
 the design: it fills the bullet point but it's asinine to use.
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's favorite
 pastime would be to inherit Regexp and override some random function in it.
 In the upcoming releases of D 2.0 there will be rather dramatic breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the
 time being.
 Andrei
As I've said before, anyone who can't stomach breaking changes w/o complaining has no business using D2 at this point. I'd rather deal with the aggravation of stuff breaking in the sort run to have a nice language and libraries to go with it in the long run. This whole concept of ranges as you've created them seems to have achieved the the holy grail of both making simple things simple and complex things possible, where "complex things" includes needing code to be efficient, so I can see your reason for wanting to redo all kinds of stuff in them. This compares favorably to C++ STL iterators, which are very flexible and efficient but a huge PITA to use for simple things because the syntax is so low-level and ugly, and to the D1/early D2 way, which gives beautiful, simple notation for the more common cases (basic dynamic arrays), at the expense of flexiblity when doing more complicated things like streams, chaining, strides, etc.
Feb 17 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from dsimcha (dsimcha yahoo.com)'s article
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 I'm quite unhappy with the API of std.regexp. It's a chaotic design that
 provides a hodgepodge of functionality and tries to use as many synonyms
 of "to find" in the dictionary (e.g. search, match). I could swear
 Walter never really cared for using regexps, and that is felt throughout
 the design: it fills the bullet point but it's asinine to use.
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's favorite
 pastime would be to inherit Regexp and override some random function in it.
 In the upcoming releases of D 2.0 there will be rather dramatic breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the
 time being.
 Andrei
As I've said before, anyone who can't stomach breaking changes w/o complaining has no business using D2 at this point. I'd rather deal with the aggravation of stuff breaking in the sort run to have a nice language and libraries to go with it in the long run. This whole concept of ranges as you've created them seems to have achieved the the holy grail of both making simple things simple and complex things possible, where "complex things" includes needing code to be efficient, so I can see your reason for wanting to redo all kinds of stuff in them. This compares favorably to C++ STL iterators, which are very flexible and efficient but a huge PITA to use for simple things because the syntax is so low-level and ugly, and to the D1/early D2 way, which gives beautiful, simple notation for the more common cases (basic dynamic arrays), at the expense of flexiblity when doing more complicated things like streams, chaining, strides, etc.
BTW, can you elaborate on how arrays, both builtin and any library versions, will work when everything is finalized?
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:
 == Quote from dsimcha (dsimcha yahoo.com)'s article
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 I'm quite unhappy with the API of std.regexp. It's a chaotic design that
 provides a hodgepodge of functionality and tries to use as many synonyms
 of "to find" in the dictionary (e.g. search, match). I could swear
 Walter never really cared for using regexps, and that is felt throughout
 the design: it fills the bullet point but it's asinine to use.
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's favorite
 pastime would be to inherit Regexp and override some random function in it.
 In the upcoming releases of D 2.0 there will be rather dramatic breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the
 time being.
 Andrei
As I've said before, anyone who can't stomach breaking changes w/o complaining has no business using D2 at this point. I'd rather deal with the aggravation of stuff breaking in the sort run to have a nice language and libraries to go with it in the long run. This whole concept of ranges as you've created them seems to have achieved the the holy grail of both making simple things simple and complex things possible, where "complex things" includes needing code to be efficient, so I can see your reason for wanting to redo all kinds of stuff in them. This compares favorably to C++ STL iterators, which are very flexible and efficient but a huge PITA to use for simple things because the syntax is so low-level and ugly, and to the D1/early D2 way, which gives beautiful, simple notation for the more common cases (basic dynamic arrays), at the expense of flexiblity when doing more complicated things like streams, chaining, strides, etc.
BTW, can you elaborate on how arrays, both builtin and any library versions, will work when everything is finalized?
Well finalizations hinges not only on me but on Walter (bugfixes and a couple of new features) and on all of you with the continuous stream of great suggestions and ideas. Again, without being able to experiment much I don't have a clear idea on how arrays/containers should at best look like. The interesting challenge is accommodating good, precise semantics with the freedom given by garbage collection. Here are some highlights: * Today's T[] will be firmly an incarnation of the random-access range concept, to the extent that all code expecting a random-access range can always be passed a T[] without any impedance adaptation. * $ will be generalized to mean "end of range" even for infinite ranges. * We don't have a solution to address the perils of extending a slice by using ~=. We're considering adding the type T[new], but I'm not sure we should take the hit of a new built-in type constructor, particularly when it's implementable as a library. * Fixed-size arrays will in all likelihood be value types. We couldn't find any other semantics that works. * Containers will have value semantics. * "Resources come and go; memory is forever" is the likely default in D resource management. This means that destroying e.g. an array of File objects will close the underlying files, but will not deallocate the memory allocated for them. In essence, destroying values means calling the destructor but not delete-ing them (unless of course they're on the stack). This approach has a number of disadvantages, but plenty of advantages that compensate them in most applications. * std.matrix will define memory layouts for a variety of popular libraries and also the common means to iterate said layouts. * For those who want containers with reference semantics, they can use the type Class!(T) for any value type T. That includes built-in value types (int, float...) and whichever value containers we define. It's unclear to me whether this is enough to satisfy those in need for complex container hierarchies. Andrei
Feb 17 2009
next sibling parent Yigal Chripun <yigal100 gmail.com> writes:
Andrei Alexandrescu wrote:
 dsimcha wrote:
 == Quote from dsimcha (dsimcha yahoo.com)'s article
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article
 I'm quite unhappy with the API of std.regexp. It's a chaotic design
 that
 provides a hodgepodge of functionality and tries to use as many
 synonyms
 of "to find" in the dictionary (e.g. search, match). I could swear
 Walter never really cared for using regexps, and that is felt
 throughout
 the design: it fills the bullet point but it's asinine to use.
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's
 favorite
 pastime would be to inherit Regexp and override some random function
 in it.
 In the upcoming releases of D 2.0 there will be rather dramatic
 breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the
 time being.
 Andrei
As I've said before, anyone who can't stomach breaking changes w/o complaining has no business using D2 at this point. I'd rather deal with the aggravation of stuff breaking in the sort run to have a nice language and libraries to go with it in the long run. This whole concept of ranges as you've created them seems to have achieved the the holy grail of both making simple things simple and complex things possible, where "complex things" includes needing code to be efficient, so I can see your reason for wanting to redo all kinds of stuff in them. This compares favorably to C++ STL iterators, which are very flexible and efficient but a huge PITA to use for simple things because the syntax is so low-level and ugly, and to the D1/early D2 way, which gives beautiful, simple notation for the more common cases (basic dynamic arrays), at the expense of flexiblity when doing more complicated things like streams, chaining, strides, etc.
BTW, can you elaborate on how arrays, both builtin and any library versions, will work when everything is finalized?
Well finalizations hinges not only on me but on Walter (bugfixes and a couple of new features) and on all of you with the continuous stream of great suggestions and ideas. Again, without being able to experiment much I don't have a clear idea on how arrays/containers should at best look like. The interesting challenge is accommodating good, precise semantics with the freedom given by garbage collection. Here are some highlights: * Today's T[] will be firmly an incarnation of the random-access range concept, to the extent that all code expecting a random-access range can always be passed a T[] without any impedance adaptation. * $ will be generalized to mean "end of range" even for infinite ranges. * We don't have a solution to address the perils of extending a slice by using ~=. We're considering adding the type T[new], but I'm not sure we should take the hit of a new built-in type constructor, particularly when it's implementable as a library. * Fixed-size arrays will in all likelihood be value types. We couldn't find any other semantics that works. * Containers will have value semantics. * "Resources come and go; memory is forever" is the likely default in D resource management. This means that destroying e.g. an array of File objects will close the underlying files, but will not deallocate the memory allocated for them. In essence, destroying values means calling the destructor but not delete-ing them (unless of course they're on the stack). This approach has a number of disadvantages, but plenty of advantages that compensate them in most applications. * std.matrix will define memory layouts for a variety of popular libraries and also the common means to iterate said layouts. * For those who want containers with reference semantics, they can use the type Class!(T) for any value type T. That includes built-in value types (int, float...) and whichever value containers we define. It's unclear to me whether this is enough to satisfy those in need for complex container hierarchies. Andrei
I've got a few questions about the proposed container value semantics: a) I'd like to be able to do for instance: List lst = new LinkedList(); i.e use interfaces everywhere and especially in functions so that I can switch implementations easily when the need arises. In the above I can choose to use singly or doubly linked list without making changes throughout the code by using the List interface. Will this be possible and how? is D going to get proper struct interfaces? b) it is sometimes useful to have a container!(Base) store references to instances of derived classes, a caconical example of this is a container of Widget class in a UI framework, where you can, for instance iterate over the container and paint all the different kinds of widgets on the screen by calling the virtual paint method of the base class. How can this be implemented with your proposed Class template? -- Yigal
Feb 18 2009
prev sibling next sibling parent Yigal Chripun <yigal100 gmail.com> writes:
Andrei Alexandrescu wrote:
 dsimcha wrote:
 == Quote from dsimcha (dsimcha yahoo.com)'s article
 == Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s
 article
 I'm quite unhappy with the API of std.regexp. It's a chaotic design
 that
 provides a hodgepodge of functionality and tries to use as many
 synonyms
 of "to find" in the dictionary (e.g. search, match). I could swear
 Walter never really cared for using regexps, and that is felt
 throughout
 the design: it fills the bullet point but it's asinine to use.
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's
 favorite
 pastime would be to inherit Regexp and override some random function
 in it.
 In the upcoming releases of D 2.0 there will be rather dramatic
 breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the
 time being.
 Andrei
As I've said before, anyone who can't stomach breaking changes w/o complaining has no business using D2 at this point. I'd rather deal with the aggravation of stuff breaking in the sort run to have a nice language and libraries to go with it in the long run. This whole concept of ranges as you've created them seems to have achieved the the holy grail of both making simple things simple and complex things possible, where "complex things" includes needing code to be efficient, so I can see your reason for wanting to redo all kinds of stuff in them. This compares favorably to C++ STL iterators, which are very flexible and efficient but a huge PITA to use for simple things because the syntax is so low-level and ugly, and to the D1/early D2 way, which gives beautiful, simple notation for the more common cases (basic dynamic arrays), at the expense of flexiblity when doing more complicated things like streams, chaining, strides, etc.
BTW, can you elaborate on how arrays, both builtin and any library versions, will work when everything is finalized?
Well finalizations hinges not only on me but on Walter (bugfixes and a couple of new features) and on all of you with the continuous stream of great suggestions and ideas. Again, without being able to experiment much I don't have a clear idea on how arrays/containers should at best look like. The interesting challenge is accommodating good, precise semantics with the freedom given by garbage collection. Here are some highlights: * Today's T[] will be firmly an incarnation of the random-access range concept, to the extent that all code expecting a random-access range can always be passed a T[] without any impedance adaptation. * $ will be generalized to mean "end of range" even for infinite ranges. * We don't have a solution to address the perils of extending a slice by using ~=. We're considering adding the type T[new], but I'm not sure we should take the hit of a new built-in type constructor, particularly when it's implementable as a library. * Fixed-size arrays will in all likelihood be value types. We couldn't find any other semantics that works. * Containers will have value semantics. * "Resources come and go; memory is forever" is the likely default in D resource management. This means that destroying e.g. an array of File objects will close the underlying files, but will not deallocate the memory allocated for them. In essence, destroying values means calling the destructor but not delete-ing them (unless of course they're on the stack). This approach has a number of disadvantages, but plenty of advantages that compensate them in most applications. * std.matrix will define memory layouts for a variety of popular libraries and also the common means to iterate said layouts. * For those who want containers with reference semantics, they can use the type Class!(T) for any value type T. That includes built-in value types (int, float...) and whichever value containers we define. It's unclear to me whether this is enough to satisfy those in need for complex container hierarchies. Andrei
Another question regarding the container design - have you considered mutable containers vs. functional style imutable containers? does it make sense to provide both options?
Feb 18 2009
prev sibling parent reply Georg Wrede <georg.wrede iki.fi> writes:
Andrei Alexandrescu wrote:
 * "Resources come and go; memory is forever" is the likely default in D 
 resource management. This means that destroying e.g. an array of File 
 objects will close the underlying files, but will not deallocate the 
 memory allocated for them. In essence, destroying values means calling 
 the destructor but not delete-ing them (unless of course they're on the 
 stack). This approach has a number of disadvantages, but plenty of 
 advantages that compensate them in most applications.
I admit I'm tired right now... You mention disadvantages, the one I can't avoid thinking of is memory leak! Which means you can't write e.g. a simple web server that opens and closes files, instead of creating and managing a file object pool? Eventually it'll run out of memory, unless I'm way too tired now...
 * std.matrix will define memory layouts for a variety of popular 
 libraries and also the common means to iterate said layouts.
I assume this is for handy and practical rectangular (and cubic, etc.) "arrays". Which would be most welcome. This "memory is forever" philosophy, is this discussed in depth somewhere? (With the current amount of traffic here, I simply can't follow every thread anymore. :-( )
Feb 20 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Georg Wrede wrote:
 Andrei Alexandrescu wrote:
 * "Resources come and go; memory is forever" is the likely default in 
 D resource management. This means that destroying e.g. an array of 
 File objects will close the underlying files, but will not deallocate 
 the memory allocated for them. In essence, destroying values means 
 calling the destructor but not delete-ing them (unless of course 
 they're on the stack). This approach has a number of disadvantages, 
 but plenty of advantages that compensate them in most applications.
I admit I'm tired right now... You mention disadvantages, the one I can't avoid thinking of is memory leak! Which means you can't write e.g. a simple web server that opens and closes files, instead of creating and managing a file object pool? Eventually it'll run out of memory, unless I'm way too tired now...
Better said, I was too tired when I posted that. I gave too little detail. Files are resources, so they will "come and go", i.e. will be under deterministic control; there's no need to worry. Only memory will have a "lives forever" regime for safety reasons. It's not really forever as the GC collects it. In short, my proposed system is to admit that GC is good _only_ for memory, and that deterministic management must prevail for other resources. I'll get back later on this.
 * std.matrix will define memory layouts for a variety of popular 
 libraries and also the common means to iterate said layouts.
I assume this is for handy and practical rectangular (and cubic, etc.) "arrays". Which would be most welcome. This "memory is forever" philosophy, is this discussed in depth somewhere? (With the current amount of traffic here, I simply can't follow every thread anymore. :-( )
I decided to curb my posting as well. Beyond a point even passable content becomes just white noise. Also since we don't have an off-topic group, off-topic discussions tend to carry on here as well and are not trivial to ignore. I'm happy they are civilized (congrats to all involved). Andrei
Feb 20 2009
prev sibling next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Wed, Feb 18, 2009 at 3:36 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 Besides std.regexp only works with (narrow) strings and we want it to work
 on streams of all widths and structures. One pet complaint I have is that
 std.regexp puts a class around it all as if everybody's favorite pastime
 would be to inherit Regexp and override some random function in it.
So what do you think it should be, a struct? That would imply to me that everybody's favorite pastime is making value copies of regex structures, when in fact nobody does that. Regex is a class in order to give it reference semantics and provide encapsulation of some re-usable state. Maybe it should be a final class, but my impression is "final class" doesn't really work in D. --bb
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Bill Baxter wrote:
 On Wed, Feb 18, 2009 at 3:36 AM, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 
 Besides std.regexp only works with (narrow) strings and we want it to work
 on streams of all widths and structures. One pet complaint I have is that
 std.regexp puts a class around it all as if everybody's favorite pastime
 would be to inherit Regexp and override some random function in it.
So what do you think it should be, a struct?
Yes.
 That would imply to me that everybody's favorite pastime is making
 value copies of regex structures, when in fact nobody does that.
Well you'd be surprised. The RegEx class saves the state of the last search, which is a sensible thing to do. But then consider a simple range Splitter that, when iterated, nicely gives you... string a = ",a, bcd, def,gh,"; foreach (e; splitter(a, pattern(", *")) writeln("[", e, "]"); writes [] [a] [bcd] [def] [gh] This is similar to the function std.regex.split with the notable difference that no extra memory is allocated. Now Splitter is an input range. This means you wouldn't expect that you copy a Splitter and then have iterating the original value affect the copy. Well, that's exactly what happens when you use the "good" reference semantics of the RegEx stored inside splitter. Worse, RegExp has no cloning primitive, so I need to resort to storing the pattern and recompiling it from scratch at every copy of Splitter. So essentially the "good" semantics of RegEx are useless when it comes to composing it in larger objects.
 Regex is a class in order to give it reference semantics and provide
 encapsulation of some re-usable state.  Maybe it should be a final
 class, but my impression is "final class" doesn't really work in D.
Re-usable state is provided by structs too. In addition they can choose value vs. reference semantics with ease. Andrei
Feb 17 2009
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 string a = ",a,  bcd, def,gh,";
 foreach (e; splitter(a, pattern(", *"))
      writeln("[", e, "]");
(I often use xplit() that is like split but yields items lazily, for larger strings it's much faster). A better approach is to fuse the xsplit and such xsplitter function in a single lazy generator that can take as a second argument a string or char or RE pattern. A 3rd optional argument can be the max number of splits (so after such max it yields all the rest of the string). You can then add an eager splitter function with the same signature, that outputs an array. Bye, bearophile
Feb 17 2009
prev sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Wed, Feb 18, 2009 at 6:56 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 Bill Baxter wrote:
 On Wed, Feb 18, 2009 at 3:36 AM, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 Besides std.regexp only works with (narrow) strings and we want it to
 work
 on streams of all widths and structures. One pet complaint I have is that
 std.regexp puts a class around it all as if everybody's favorite pastime
 would be to inherit Regexp and override some random function in it.
So what do you think it should be, a struct?
Yes.
 That would imply to me that everybody's favorite pastime is making
 value copies of regex structures, when in fact nobody does that.
Well you'd be surprised. The RegEx class saves the state of the last search, which is a sensible thing to do. But then consider a simple range Splitter that, when iterated, nicely gives you... string a = ",a, bcd, def,gh,"; foreach (e; splitter(a, pattern(", *")) writeln("[", e, "]"); writes [] [a] [bcd] [def] [gh] This is similar to the function std.regex.split with the notable difference that no extra memory is allocated. Now Splitter is an input range. This means you wouldn't expect that you copy a Splitter and then have iterating the original value affect the copy. Well, that's exactly what happens when you use the "good" reference semantics of the RegEx stored inside splitter. Worse, RegExp has no cloning primitive, so I need to resort to storing the pattern and recompiling it from scratch at every copy of Splitter. So essentially the "good" semantics of RegEx are useless when it comes to composing it in larger objects.
So that sounds to me like RegEx should have a .dup, and then it would be fine, no? I agree it should have a dup for the odd occasion when you do want to make a copy for some reason.
 Regex is a class in order to give it reference semantics and provide
 encapsulation of some re-usable state.  Maybe it should be a final
 class, but my impression is "final class" doesn't really work in D.
 Re-usable state is provided by structs too. In addition they can choose
 value vs. reference semantics with ease.
I think this choice is not so much available with D1, plus the constructor situation with D1 is less than ideal. Given that, I think the choice of class for RegEx was apropriate. But if the struct problems are all going away in D2, then that's great. Sounds like you're saying we'll really be able to use D structs just like one uses a non-polymorphic C++ class. If so, then that's super. --bb
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Bill Baxter wrote:
 I think this choice is not so much available with D1, plus the
 constructor situation with D1 is less than ideal.  Given that, I think
 the choice of class for RegEx was apropriate.   But if the struct
 problems are all going away in D2, then that's great.  Sounds like
 you're saying we'll really be able to use D structs just like one uses
 a non-polymorphic C++ class.  If so, then that's super.
I lost that perspective when criticizing RegExp, you're right. But still the API is lousy - every single time I am using a RegExp, I find myself fumbling through the thoroughly overlapping primitives in the documentation, and never seem to find an idiom that's simple, comfortable, and memorable. Andrei
Feb 17 2009
next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Wed, Feb 18, 2009 at 7:44 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 Bill Baxter wrote:
 I think this choice is not so much available with D1, plus the
 constructor situation with D1 is less than ideal.  Given that, I think
 the choice of class for RegEx was apropriate.   But if the struct
 problems are all going away in D2, then that's great.  Sounds like
 you're saying we'll really be able to use D structs just like one uses
 a non-polymorphic C++ class.  If so, then that's super.
I lost that perspective when criticizing RegExp, you're right. But still the API is lousy - every single time I am using a RegExp, I find myself fumbling through the thoroughly overlapping primitives in the documentation, and never seem to find an idiom that's simple, comfortable, and memorable.
Ok. I'm certainly not in love with the API either. Though, the only RegEx API I've ever used that felt totally comfortable with was Perl's, which in large part is syntax instead of an API. Python's syntax I have to look over the documentation every time I use it, too. Maybe it's because of the "matching" vs "searching" distinction that I find impossible to remember. (http://docs.python.org/library/re.html) --bb
Feb 17 2009
parent bearophile <bearophileHUGS lycos.com> writes:
Bill Baxter:
Python's syntax I have to look over the documentation every time I use it, too.
Maybe it's because of the "matching" vs "searching" distinction that I find
impossible to remember.<
I agree, I too need the Python docs every time I want to use something more than the basics. The syntax for group catching too is bad (groups? group? itersomething? etc). I have proposed an improvement (using [5] to grab the 5th group() but it was not implemented. Such syntax is possible in D too *hint*). It's because of situations like this that I say that designing a good API for std.re isn't easy at all. It will require care, brain, and maybe two or more tries :-) Bye, bearophile
Feb 17 2009
prev sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 7:13 PM, Bill Baxter <wbaxter gmail.com> wrote:
 Ok.  I'm certainly not in love with the API either.  Though, the only
 RegEx API I've ever used that felt totally comfortable with was
 Perl's, which in large part is syntax instead of an API.  Python's
 syntax I have to look over the documentation every time I use it, too.
  Maybe it's because of the "matching" vs "searching" distinction that
 I find impossible to remember.
 (http://docs.python.org/library/re.html)
Is there ever a situation where you want to use a single regexp for both matching _and_ searching? And if not, couldn't you just use ^ to anchor it? I never understood why Python's API makes such a distinction.
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Jarrett Billingsley wrote:
 On Tue, Feb 17, 2009 at 7:13 PM, Bill Baxter <wbaxter gmail.com> wrote:
 Ok.  I'm certainly not in love with the API either.  Though, the only
 RegEx API I've ever used that felt totally comfortable with was
 Perl's, which in large part is syntax instead of an API.  Python's
 syntax I have to look over the documentation every time I use it, too.
  Maybe it's because of the "matching" vs "searching" distinction that
 I find impossible to remember.
 (http://docs.python.org/library/re.html)
Is there ever a situation where you want to use a single regexp for both matching _and_ searching? And if not, couldn't you just use ^ to anchor it? I never understood why Python's API makes such a distinction.
Ehm, that's odd. You'd think that after Perl has set the precedent, it would be hard to do major goofs in designing a regex API. By the way, the more I dig into std.regexp, the stiffer the hair on my neck gets. Get this: the API offers both global functions and member functions, with both RegExp and plain string arguments. The latter are carefully designed to maximize the number of clashes, potential confusions, and errors when using both std.string and std.regex. But wait, there's more. The API defines the following functions that all ostensibly do some sort of mattern patching (sic): find, search, test, match, and exec. I wish I were kidding. There's some opIndex and opEquals thrown in for good measure. Knuth wouldn't know what each of them does after studying them for a week and then watching an episode from "The Bachelor". And get this: global search() does not do what member search() does. Nope. Global search() does what member test() does. I have only contempt for such designs. Andrei
Feb 17 2009
next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Wed, Feb 18, 2009 at 11:38 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 Jarrett Billingsley wrote:
 On Tue, Feb 17, 2009 at 7:13 PM, Bill Baxter <wbaxter gmail.com> wrote:
 Ok.  I'm certainly not in love with the API either.  Though, the only
 RegEx API I've ever used that felt totally comfortable with was
 Perl's, which in large part is syntax instead of an API.  Python's
 syntax I have to look over the documentation every time I use it, too.
  Maybe it's because of the "matching" vs "searching" distinction that
 I find impossible to remember.
 (http://docs.python.org/library/re.html)
Is there ever a situation where you want to use a single regexp for both matching _and_ searching? And if not, couldn't you just use ^ to anchor it? I never understood why Python's API makes such a distinction.
Ehm, that's odd. You'd think that after Perl has set the precedent, it would be hard to do major goofs in designing a regex API. By the way, the more I dig into std.regexp, the stiffer the hair on my neck gets. Get this: the API offers both global functions and member functions, with both RegExp and plain string arguments. The latter are carefully designed to maximize the number of clashes, potential confusions, and errors when using both std.string and std.regex.
All I know is that I found one incantation that works and I've been copy-pasting that every since. :-)
 But wait, there's more. The API defines the following functions that all
 ostensibly do some sort of mattern patching (sic): find, search, test,
 match, and exec. I wish I were kidding. There's some opIndex and opEquals
 thrown in for good measure. Knuth wouldn't know what each of them does after
 studying them for a week and then watching an episode from "The Bachelor".
 And get this: global search() does not do what member search() does. Nope.
 Global search() does what member test() does. I have only contempt for such
 designs.
Maybe "design" is too strong a word. Most Phobos modules seem to have been put together rather hastily in order to fill a pressing need. Often *something* is better than nothing at all, even if the something is not so great. --bb
Feb 17 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Bill Baxter wrote:
 Maybe "design" is too strong a word.  Most Phobos modules seem to have
 been put together rather hastily in order to fill a pressing need.
 Often *something* is better than nothing at all, even if the something
 is not so great.
std.regexp evolved out of the ECMAscript regex functions - they have the same names and functionality. Layered on top of that was ruby-like names and functionality. It's a good (bad?) example of an api evolving without sacrificing backwards compatibility.
Feb 20 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter Bright wrote:
 Bill Baxter wrote:
 Maybe "design" is too strong a word.  Most Phobos modules seem to have
 been put together rather hastily in order to fill a pressing need.
 Often *something* is better than nothing at all, even if the something
 is not so great.
std.regexp evolved out of the ECMAscript regex functions - they have the same names and functionality. Layered on top of that was ruby-like names and functionality. It's a good (bad?) example of an api evolving without sacrificing backwards compatibility.
s/good \(bad\?\)/REALLY BAD/ Andrei
Feb 20 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 20 Feb 2009 16:35:54 +0300, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Walter Bright wrote:
 Bill Baxter wrote:
 Maybe "design" is too strong a word.  Most Phobos modules seem to have
 been put together rather hastily in order to fill a pressing need.
 Often *something* is better than nothing at all, even if the something
 is not so great.
std.regexp evolved out of the ECMAscript regex functions - they have the same names and functionality. Layered on top of that was ruby-like names and functionality. It's a good (bad?) example of an api evolving without sacrificing backwards compatibility.
s/good \(bad\?\)/REALLY BAD/ Andrei
Backward compatibility is almost always a bad thing. Look what's happened to C++ and OpenGL.
Feb 20 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Denis Koroskin wrote:
 On Fri, 20 Feb 2009 16:35:54 +0300, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:
 
 Walter Bright wrote:
 Bill Baxter wrote:
 Maybe "design" is too strong a word.  Most Phobos modules seem to have
 been put together rather hastily in order to fill a pressing need.
 Often *something* is better than nothing at all, even if the something
 is not so great.
std.regexp evolved out of the ECMAscript regex functions - they have the same names and functionality. Layered on top of that was ruby-like names and functionality. It's a good (bad?) example of an api evolving without sacrificing backwards compatibility.
s/good \(bad\?\)/REALLY BAD/ Andrei
Backward compatibility is almost always a bad thing. Look what's happened to C++ and OpenGL.
In this case it's even worse, as I don't think anyone expects to paste their Ruby code and compile it with dmd. Andrei
Feb 20 2009
prev sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 9:38 PM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 By the way, the more I dig into std.regexp, the stiffer the hair on my neck
 gets. Get this: the API offers both global functions and member functions,
 with both RegExp and plain string arguments. The latter are carefully
 designed to maximize the number of clashes, potential confusions, and errors
 when using both std.string and std.regex.

 But wait, there's more. The API defines the following functions that all
 ostensibly do some sort of mattern patching (sic): find, search, test,
 match, and exec. I wish I were kidding. There's some opIndex and opEquals
 thrown in for good measure. Knuth wouldn't know what each of them does after
 studying them for a week and then watching an episode from "The Bachelor".
 And get this: global search() does not do what member search() does. Nope.
 Global search() does what member test() does. I have only contempt for such
 designs.
Well I don't mean to, uh, toot my own horn but.. I recently bound libpcre to MiniD and came up with a relatively simple but powerful and orthogonal API. http://www.dsource.org/projects/minid/wiki/Addons/PcreLib#LibraryReference The regex object has a single "subject" string at a time, the string that it's matching against. The subject is set with "search" and "test" does everything. All other functions are basically defined in terms of those two. "test" looks for the next match of the regex in the subject and returns true if it matched. "match" returns match groups (0 for the whole regex and 1..n for subgroups, as well as string indices for named subgroups). opApply is just a quicker way of writing something like: re.search(someSubject) while(re.test()) // use re.match to get matches You'll notice that opApply is also just defined in terms of test. I've found it far more intuitive than other APIs. I've never used Perl and I doubt I ever will, though.
Feb 17 2009
prev sibling next sibling parent reply Bill Baxter <wbaxter gmail.com> writes:
On Wed, Feb 18, 2009 at 3:36 AM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 I'm quite unhappy with the API of std.regexp. It's a chaotic design that
 provides a hodgepodge of functionality and tries to use as many synonyms of
 "to find" in the dictionary (e.g. search, match). I could swear Walter never
 really cared for using regexps, and that is felt throughout the design: it
 fills the bullet point but it's asinine to use.

 Besides std.regexp only works with (narrow) strings and we want it to work
 on streams of all widths and structures. One pet complaint I have is that
 std.regexp puts a class around it all as if everybody's favorite pastime
 would be to inherit Regexp and override some random function in it.

 In the upcoming releases of D 2.0 there will be rather dramatic breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the time
 being.
Btw, I've got no problems with you breaking the API of 2.0 either. Though you might consider moving the current implementation to std.deprecated.regex and leaving it there for a year with a pragma(msg, "This module is deprecated"). That way making a quick fix to broken code is just a matter of inserting ".deprecated" into your import statements. --bb
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Bill Baxter wrote:
 On Wed, Feb 18, 2009 at 3:36 AM, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 I'm quite unhappy with the API of std.regexp. It's a chaotic design that
 provides a hodgepodge of functionality and tries to use as many synonyms of
 "to find" in the dictionary (e.g. search, match). I could swear Walter never
 really cared for using regexps, and that is felt throughout the design: it
 fills the bullet point but it's asinine to use.

 Besides std.regexp only works with (narrow) strings and we want it to work
 on streams of all widths and structures. One pet complaint I have is that
 std.regexp puts a class around it all as if everybody's favorite pastime
 would be to inherit Regexp and override some random function in it.

 In the upcoming releases of D 2.0 there will be rather dramatic breaking
 changes of phobos. I just wanted to ask whether y'all could stomach yet
 another rewritten API or you'd rather use std.regexp as it is for the time
 being.
Btw, I've got no problems with you breaking the API of 2.0 either. Though you might consider moving the current implementation to std.deprecated.regex and leaving it there for a year with a pragma(msg, "This module is deprecated"). That way making a quick fix to broken code is just a matter of inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok? Andrei
Feb 17 2009
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
Andrei Alexandrescu wrote:
 I was thinking of moving older stuff to etc, is that ok?
Yes. But you should also rename the new one, perhaps to std.regex. That way, legacy code will refuse to compile, rather than compile wrongly.
Feb 17 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter Bright wrote:
 Andrei Alexandrescu wrote:
 I was thinking of moving older stuff to etc, is that ok?
Yes. But you should also rename the new one, perhaps to std.regex. That way, legacy code will refuse to compile, rather than compile wrongly.
Terrific. I prefer "regex" to "regexp" because it's easier to pronounce, particularly if you're a foreigner. "Regex" sounds like a frog utterance by a forest lake, "regexp" sounds like nothing in particular. Andrei
Feb 17 2009
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:
 Terrific. I prefer "regex" to "regexp" because it's easier to pronounce, 
 particularly if you're a foreigner. "Regex" sounds like a frog utterance 
 by a forest lake, "regexp" sounds like nothing in particular.
I'd like std.re :-) Bye, bearophile
Feb 17 2009
prev sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Andrei Alexandrescu wrote:
 Walter Bright wrote:
 Andrei Alexandrescu wrote:
 I was thinking of moving older stuff to etc, is that ok?
Yes. But you should also rename the new one, perhaps to std.regex. That way, legacy code will refuse to compile, rather than compile wrongly.
Terrific. I prefer "regex" to "regexp" because it's easier to pronounce, particularly if you're a foreigner. "Regex" sounds like a frog utterance by a forest lake, "regexp" sounds like nothing in particular. Andrei
It sounds to me like a frog who, immediately post-utterance, just got gigged. I guess that makes "regex" sound even better... as its still alive (sounding). -- Chris Nicholson-Sauls -- Who so far agrees with pretty much everything you've said, and therefore has no real contribution...
Feb 17 2009
prev sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Andrei Alexandrescu, el 17 de febrero a las 13:56 me escribiste:
Btw, I've got no problems with you breaking the API of 2.0 either.
Though you might consider moving the current implementation to
std.deprecated.regex and leaving it there for a year with a
pragma(msg, "This module is deprecated").
That way making a quick fix to broken code is just a matter of
inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated", o something shorter like "old", or "d1" (this last one could be good for future deprecated libraries, like when D3 is available there probably be a "d2" too). -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Hey you, don't tell me there's no hope at all Together we stand, divided we fall.
Feb 19 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Leandro Lucarella wrote:
 Andrei Alexandrescu, el 17 de febrero a las 13:56 me escribiste:
 Btw, I've got no problems with you breaking the API of 2.0 either.
 Though you might consider moving the current implementation to
 std.deprecated.regex and leaving it there for a year with a
 pragma(msg, "This module is deprecated").
 That way making a quick fix to broken code is just a matter of
 inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated", o something shorter like "old", or "d1" (this last one could be good for future deprecated libraries, like when D3 is available there probably be a "d2" too).
In the words of George Costanza: "Because it's there!" Andrei
Feb 19 2009
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
Andrei Alexandrescu wrote:
 Leandro Lucarella wrote:
 Andrei Alexandrescu, el 17 de febrero a las 13:56 me escribiste:
 Btw, I've got no problems with you breaking the API of 2.0 either.
 Though you might consider moving the current implementation to
 std.deprecated.regex and leaving it there for a year with a
 pragma(msg, "This module is deprecated").
 That way making a quick fix to broken code is just a matter of
 inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated", o something shorter like "old", or "d1" (this last one could be good for future deprecated libraries, like when D3 is available there probably be a "d2" too).
In the words of George Costanza: "Because it's there!" Andrei
Shouldn't that be George Mallory?
Feb 19 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Ellery Newcomer wrote:
 Andrei Alexandrescu wrote:
 Leandro Lucarella wrote:
 Andrei Alexandrescu, el 17 de febrero a las 13:56 me escribiste:
 Btw, I've got no problems with you breaking the API of 2.0 either.
 Though you might consider moving the current implementation to
 std.deprecated.regex and leaving it there for a year with a
 pragma(msg, "This module is deprecated").
 That way making a quick fix to broken code is just a matter of
 inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated", o something shorter like "old", or "d1" (this last one could be good for future deprecated libraries, like when D3 is available there probably be a "d2" too).
In the words of George Costanza: "Because it's there!" Andrei
Shouldn't that be George Mallory?
No, he said "because it is there". George said "because it's there": http://www.classictvquotes.com/quotes/characters/george-costanza/page_14.html George: So, she fell, and then she started screaming, "My back! My back!" So, I picked her up and took her to the hospital. Elaine: How is she? George: She's in traction. Elaine: Okay, I'm sorry. George: It's not funny, Elaine. Elaine: I know. I'm sorry. I'm serious. George: Her back went out. She's gotta be there for a couple of days. All she said on the way over in the car was, "Why, George, why?!" I said, "Because it's there!" Andrei
Feb 19 2009
prev sibling parent reply Georg Wrede <georg.wrede iki.fi> writes:
Andrei Alexandrescu wrote:
 That way making a quick fix to broken code is just a matter of
 inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated"
In the words of George Costanza: "Because it's there!"
With the critique you've given to the existing regexp stuff, deprecated would be the obvious choice. Then we could have etc for Miscellaneous Stuff.
Feb 20 2009
next sibling parent Bill Baxter <wbaxter gmail.com> writes:
On Sat, Feb 21, 2009 at 2:38 AM, Georg Wrede <georg.wrede iki.fi> wrote:
 Andrei Alexandrescu wrote:
 That way making a quick fix to broken code is just a matter of
 inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated"
In the words of George Costanza: "Because it's there!"
With the critique you've given to the existing regexp stuff, deprecated would be the obvious choice. Then we could have etc for Miscellaneous Stuff.
Agreed. etc implies to me that it's stuff that might be useful sometimes but not very often. It does not suggest to me that you shouldn't use it if you can avoid it. Or how about make it std.etc.deprecated.regexp That way it's clear that it's *both* something that might be useful occasionally but something that you should avoid if possible. ... Deprecated is a keyword though, isn't it. Dang. :-P --bb
Feb 20 2009
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Georg Wrede, el 20 de febrero a las 19:38 me escribiste:
 Andrei Alexandrescu wrote:
That way making a quick fix to broken code is just a matter of
inserting ".deprecated" into your import statements.
I was thinking of moving older stuff to etc, is that ok?
What's the rationale for "etc"? Why not "deprecated"
In the words of George Costanza: "Because it's there!"
With the critique you've given to the existing regexp stuff, deprecated would be the obvious choice. Then we could have etc for Miscellaneous Stuff.
Why not "misc" for that? =) -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- TIGRE SE COMIO A EMPLEADO DE CIRCO: DETUVIERON A DUEÑO Y DOMADOR -- Crónica TV
Feb 20 2009
prev sibling next sibling parent reply BCS <ao pathlink.com> writes:
Reply to Andrei,

 I'm quite unhappy with the API of std.regexp. It's a chaotic design
 that provides a hodgepodge of functionality and tries to use as many
 synonyms of "to find" in the dictionary (e.g. search, match). I could
 swear Walter never really cared for using regexps, and that is felt
 throughout the design: it fills the bullet point but it's asinine to
 use.
 
 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's
 favorite pastime would be to inherit Regexp and override some random
 function in it.
 
 In the upcoming releases of D 2.0 there will be rather dramatic
 breaking changes of phobos. I just wanted to ask whether y'all could
 stomach yet another rewritten API or you'd rather use std.regexp as it
 is for the time being.
 
 Andrei
 
For what it's worth, I have a partial clone of the .NET API built on top of PCRE. I would have to ask my boss but I expect I could donate it if anyone want to use it as a basis.
Feb 17 2009
next sibling parent reply Daniel de Kok <me danieldk.org> writes:
On Tue, Feb 17, 2009 at 8:39 PM, BCS <ao pathlink.com> wrote:
 For what it's worth, I have a partial clone of the .NET API built on top of
 PCRE. I would have to ask my boss but I expect I could donate it if anyone
 want to use it as a basis.
Actually, I was wondering why nobody is considering real regular languages anymore, that can be compiled to a normal finite state recognizer or transducer. While this may not be as fancy as Perl-like extensions, they are much faster, and it's easier to do fun stuff such as composition. Take care, Daniel
Feb 17 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Daniel de Kok wrote:
 On Tue, Feb 17, 2009 at 8:39 PM, BCS <ao pathlink.com> wrote:
 For what it's worth, I have a partial clone of the .NET API built on top of
 PCRE. I would have to ask my boss but I expect I could donate it if anyone
 want to use it as a basis.
Actually, I was wondering why nobody is considering real regular languages anymore, that can be compiled to a normal finite state recognizer or transducer. While this may not be as fancy as Perl-like extensions, they are much faster, and it's easier to do fun stuff such as composition.
I am considering that. One nice feature of "classic" regexes is that they never backtrack, so they work with pure input iterators. This has crucial consequences with regard to where and how regexes fit the range concept hierarchy. Andrei
Feb 17 2009
prev sibling next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 2:47 PM, Daniel de Kok <me danieldk.org> wrote:
 On Tue, Feb 17, 2009 at 8:39 PM, BCS <ao pathlink.com> wrote:
 For what it's worth, I have a partial clone of the .NET API built on top of
 PCRE. I would have to ask my boss but I expect I could donate it if anyone
 want to use it as a basis.
Actually, I was wondering why nobody is considering real regular languages anymore, that can be compiled to a normal finite state recognizer or transducer. While this may not be as fancy as Perl-like extensions, they are much faster, and it's easier to do fun stuff such as composition.
Tango's regex engine is just that. It uses a tagged NFA method. http://www.dsource.org/projects/tango/docs/current/tango.text.Regex.html The problem with this method is that while it's certainly faster to match, it's MUCH slower to compile. There are no pathological matches; only pathological compiles ;) I'm talking 60-70 seconds to compile a more complex regex. This might be an acceptable tradeoff for when you need to compile a regex in a long-running app like a server, but it's completely unacceptable for most small, Perl-like text munging programs. Unless of course this slowdown is unique to Tango's implementation of this method!
Feb 17 2009
next sibling parent bearophile <bearophileHUGS lycos.com> writes:
Jarrett Billingsley:
I'm talking 60-70 seconds to compile a more complex regex.<
A modern CPU is able to do something like 60*2*2E9 operations in that time, DMD needs 6 seconds or less to compile about 60000-80000 lines of my D code, so I think it's a bit too much time (probably 100 or 1000 times too much). Bye, bearophile
Feb 17 2009
prev sibling parent reply BCS <ao pathlink.com> writes:
Reply to Jarrett,

 On Tue, Feb 17, 2009 at 2:47 PM, Daniel de Kok <me danieldk.org>
 wrote:
 
 Actually, I was wondering why nobody is considering real regular
 languages anymore, that can be compiled to a normal finite state
 recognizer or transducer. While this may not be as fancy as Perl-like
 extensions, they are much faster, and it's easier to do fun stuff
 such as composition.
 
Tango's regex engine is just that. It uses a tagged NFA method. http://www.dsource.org/projects/tango/docs/current/tango.text.Regex.ht ml The problem with this method is that while it's certainly faster to match, it's MUCH slower to compile. There are no pathological matches; only pathological compiles ;) I'm talking 60-70 seconds to compile a more complex regex.
could this be transitioned to CTFE? you could even have a debug mode that delays till runtime RegEx mather = new CTFERegEx!("some regex"); class CTFERegEx(char[] regex) : RegEx { debug(NoCTFE) static char[] done; else static const char[] done = CTFECompile(regex); public this() { debug(NoCTFE) if(done == null) done = CTFECompile(regex); base(done) } }
Feb 17 2009
next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 3:16 PM, BCS <ao pathlink.com> wrote:
 could this be transitioned to CTFE? you could even have a debug mode that
 delays till runtime

 RegEx mather = new CTFERegEx!("some regex");


 class CTFERegEx(char[] regex) : RegEx
 {
      debug(NoCTFE)  static char[] done;
      else     static const char[] done = CTFECompile(regex);

      public this()
      {
         debug(NoCTFE) if(done == null) done = CTFECompile(regex);

         base(done)
      }
 }
For what it's worth the Tango regexes actually have a method to output a D function that will implement the regex after it's compiled. So you _could_ precompile the regex into D code and use that. But seriously, man - if something takes 60 seconds to complete at _runtime_, making it CTFE would simply make your computer explode.
Feb 17 2009
next sibling parent BCS <ao pathlink.com> writes:
Reply to Jarrett,

 On Tue, Feb 17, 2009 at 3:16 PM, BCS <ao pathlink.com> wrote:
 
 could this be transitioned to CTFE? you could even have a debug mode
 that delays till runtime
 
 RegEx mather = new CTFERegEx!("some regex");
 
 class CTFERegEx(char[] regex) : RegEx
 {
 debug(NoCTFE)  static char[] done;
 else     static const char[] done = CTFECompile(regex);
 public this()
 {
 debug(NoCTFE) if(done == null) done = CTFECompile(regex);
 base(done)
 }
 }
For what it's worth the Tango regexes actually have a method to output a D function that will implement the regex after it's compiled. So you _could_ precompile the regex into D code and use that. But seriously, man - if something takes 60 seconds to complete at _runtime_, making it CTFE would simply make your computer explode.
For any kind of debug, yeah, that's a problem. OTOH for release, as long as it /does/ compile, who cares? How many real release builds does anyone do a week?
Feb 17 2009
prev sibling parent Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Jarrett Billingsley wrote:
 On Tue, Feb 17, 2009 at 3:16 PM, BCS <ao pathlink.com> wrote:
 could this be transitioned to CTFE? you could even have a debug mode that
 delays till runtime

 RegEx mather = new CTFERegEx!("some regex");


 class CTFERegEx(char[] regex) : RegEx
 {
      debug(NoCTFE)  static char[] done;
      else     static const char[] done = CTFECompile(regex);

      public this()
      {
         debug(NoCTFE) if(done == null) done = CTFECompile(regex);

         base(done)
      }
 }
For what it's worth the Tango regexes actually have a method to output a D function that will implement the regex after it's compiled. So you _could_ precompile the regex into D code and use that.
I feature which I *adore* by the way. So long as the precompiled regex is "guaranteed" to run at best possible performance (hand-rolled, hand-optimized solutions notwithstanding) I for one prefer them. -- Chris Nicholson-Sauls
Feb 17 2009
prev sibling parent Daniel de Kok <me danieldk.org> writes:
On Tue, Feb 17, 2009 at 9:26 PM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:
 For what it's worth the Tango regexes actually have a method to output
 a D function that will implement the regex after it's compiled.  So
 you _could_ precompile the regex into D code and use that.
I have only been tinkering with Phobos, but that's good to hear, thanks!
Feb 17 2009
prev sibling next sibling parent Daniel de Kok <me danieldk.org> writes:
On Tue, Feb 17, 2009 at 8:57 PM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:
 The problem with this method is that while it's certainly faster to
 match, it's MUCH slower to compile.  There are no pathological
 matches; only pathological compiles ;)  I'm talking 60-70 seconds to
 compile a more complex regex. This might be an acceptable tradeoff
 for when you need to compile a regex in a long-running app like a
 server, but it's completely unacceptable for most small, Perl-like
 text munging programs.
Hmmm, define "complex", I suppose it's ok for the general line-splitting/matching stuff? I got into trouble (time-wise) when we compiled a part of speech tagger into a transducer. In those cases we generally pre-compile stuff, and output it as a large struct in the target language. Of course, it would be fun if we can do it at compile-time ;). Besides that, if we'd have a good general recognizer/transducer implementation it could also be used for compact dictionary storage, perfect hashing automata, etc. Take care, Daniel
Feb 17 2009
prev sibling next sibling parent reply Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Feb 17, 2009 at 3:30 PM, Daniel de Kok <me danieldk.org> wrote:
 Hmmm, define "complex"
\w+([\-+.]\w+)* \w+([\-.]\w+)*\.\w+([\-.]\w+)* This is a simple email regexp. This takes about 4 or 5 seconds to compile on my lappy (Pentium M). It only goes up from there.
Feb 17 2009
parent BCS <ao pathlink.com> writes:
Reply to Jarrett,

 On Tue, Feb 17, 2009 at 3:30 PM, Daniel de Kok <me danieldk.org>
 wrote:
 
 Hmmm, define "complex"
 
\w+([\-+.]\w+)* \w+([\-.]\w+)*\.\w+([\-.]\w+)* This is a simple email regexp. This takes about 4 or 5 seconds to compile on my lappy (Pentium M). It only goes up from there.
I wonder how well it would work on this: http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html :b
Feb 17 2009
prev sibling next sibling parent Daniel de Kok <me danieldk.org> writes:
On Tue, Feb 17, 2009 at 9:50 PM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:
 On Tue, Feb 17, 2009 at 3:30 PM, Daniel de Kok <me danieldk.org> wrote:
 Hmmm, define "complex"
\w+([\-+.]\w+)* \w+([\-.]\w+)*\.\w+([\-.]\w+)* This is a simple email regexp. This takes about 4 or 5 seconds to compile on my lappy (Pentium M).
Hmm, odd. I have translated that regexp to the syntax of the tool that we used, that is written in Prolog (it is generally a constant factor slower than C/C++/D equivalents). Generating a minimized DFA takes far less than a second. I used the following expression (abstracted a bit with macros): --- macro(letter, {a..z, 'A'..'Z'}). macro(punctlet,[{-,+,.},letter+]). macro(dompunctlet,[{-,.},letter+]). macro(email,[letter+,punctlet*, ,letter+,dompunctlet*,.,letter+,dompunctlet*]). --- The software is available from: http://www.let.rug.nl/~vannoord/Fsa/fsa.html Take care, Daniel
Feb 17 2009
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
BCS wrote:
 Reply to Andrei,
 
 I'm quite unhappy with the API of std.regexp. It's a chaotic design
 that provides a hodgepodge of functionality and tries to use as many
 synonyms of "to find" in the dictionary (e.g. search, match). I could
 swear Walter never really cared for using regexps, and that is felt
 throughout the design: it fills the bullet point but it's asinine to
 use.

 Besides std.regexp only works with (narrow) strings and we want it to
 work on streams of all widths and structures. One pet complaint I have
 is that std.regexp puts a class around it all as if everybody's
 favorite pastime would be to inherit Regexp and override some random
 function in it.

 In the upcoming releases of D 2.0 there will be rather dramatic
 breaking changes of phobos. I just wanted to ask whether y'all could
 stomach yet another rewritten API or you'd rather use std.regexp as it
 is for the time being.

 Andrei
For what it's worth, I have a partial clone of the .NET API built on top of PCRE. I would have to ask my boss but I expect I could donate it if anyone want to use it as a basis.
That would be cool; I find the engine in std.regexp rather hard to understand. Andrei
Feb 17 2009
prev sibling parent Derek Parnell <derek psych.ward> writes:
On Tue, 17 Feb 2009 10:36:06 -0800, Andrei Alexandrescu wrote:

 I'm quite unhappy with the API of std.regexp.
I was so happy with using it I wrote my own simplified regex ;-)
 In the upcoming releases of D 2.0 there will be rather dramatic breaking 
 changes of phobos. I just wanted to ask whether y'all could stomach yet 
 another rewritten API or you'd rather use std.regexp as it is for the 
 time being.
If your changes are going to make things better for coding and maintenance then go for it. -- Derek Parnell Melbourne, Australia skype: derek.j.parnell
Feb 17 2009