www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - What type does byGrapheme() return?

reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
I love these documentation lines in the D docs:

	auto byGrapheme(Range)(Range range)

How should I know what auto is? Why not write the explicit type so that 
I know what to expect? When declaring a variable as class/struct member 
I can't use auto but need the explicit type...

I used typeof() but that doesn't help a lot:

gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string

I want to iterate a string byGrapheme so that I can add, delete, change 
graphemes.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Dec 27 2019
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/27/19 12:26 PM, Robert M. Münch wrote:

 I love these documentation lines in the D docs:
 
      auto byGrapheme(Range)(Range range)
 
 How should I know what auto is? Why not write the explicit type so that 
 I know what to expect? When declaring a variable as class/struct member 
 I can't use auto but need the explicit type...
 
 I used typeof() but that doesn't help a lot:
 
 gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string
 
 I want to iterate a string byGrapheme so that I can add, delete, change 
 graphemes.
 


This is the rub with ranges. You need to use typeof. There's no other 
way to do it, because the type returned by byGrapheme depends on the 
type of Range.

If you know what type Range is, it would be:

struct S
{
    typeof(string.init.byGrapheme()) gText;
    // or
    alias GRange = typeof(string.init.byGrapheme());
    GRange gText;
}

Subbing in whatever your real range for `string`. Or if it's the result 
of a bunch of adapters, use the whole call chain with typeof.

Why not just declare the real range type? Because it's going to be ugly, 
especially if your underlying range is the result of other range 
algorithms. And really, typeof is going to be the better mechanism, even 
if it's not the best looking thing.

-Steve
Dec 27 2019
parent =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2019-12-27 17:54:28 +0000, Steven Schveighoffer said:


 This is the rub with ranges. You need to use typeof. There's no other 
 way to do it, because the type returned by byGrapheme depends on the 
 type of Range.


Hi, ok, thanks a lot and IMO these are the fundamental important 
information for people using D (beginners, causual programmers, etc.) 
to understand how things fit together.


 If you know what type Range is, it would be:
 
 struct S
 {
     typeof(string.init.byGrapheme()) gText;
     // or
     alias GRange = typeof(string.init.byGrapheme());
     GRange gText;
 }


Ah... I didn't know that I can use a basic type "string" combined with 
".init" to manually build the type. Neat...


 Subbing in whatever your real range for `string`. Or if it's the result 
 of a bunch of adapters, use the whole call chain with typeof.


Ok, and these are good candidates for alias definitions to avoid 
re-typing it many times.


 Why not just declare the real range type? Because it's going to be 
 ugly, especially if your underlying range is the result of other range 
 algorithms. And really, typeof is going to be the better mechanism, 
 even if it's not the best looking thing.


I think I got it... thanks a lot.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Dec 29 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Dec 27, 2019 at 06:26:58PM +0100, Robert M. Münch via
Digitalmars-d-learn wrote:

 I love these documentation lines in the D docs:
 
 	auto byGrapheme(Range)(Range range)
 
 How should I know what auto is? Why not write the explicit type so
 that I know what to expect? When declaring a variable as class/struct
 member I can't use auto but need the explicit type...
 
 I used typeof() but that doesn't help a lot:
 
 gText = [Grapheme(53, 0, 0, 72057594037927936, [83, ...., 1)]Result!string
 
 I want to iterate a string byGrapheme so that I can add, delete,
 change graphemes.


[...]

Since graphemes are variable-length in terms of code points, you can't
exactly *edit* a range of graphemes -- you can't replace a 1-codepoint
grapheme with a 6-codepoint grapheme, for example, since there's no
space in the underlying string to store that.

If you want to add/delete/change graphemes, what you *really* want is to
use an array of Graphemes:

	Grapheme[] editableGraphs;

You can then splice it, insert stuff, delete stuff, whatever.

When you're done with it, convert it back to string with something like
this:

	string result = editableGraphs.map!(g => g[]).joiner.to!string;


T

-- 
The irony is that Bill Gates claims to be making a stable operating system and
Linus Torvalds claims to be trying to take over the world. -- Anonymous
Dec 27 2019
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2019-12-27 19:44:59 +0000, H. S. Teoh said:


 Since graphemes are variable-length in terms of code points, you can't
 exactly *edit* a range of graphemes -- you can't replace a 1-codepoint
 grapheme with a 6-codepoint grapheme, for example, since there's no
 space in the underlying string to store that.


Hi, my idea was that when I use a grapheme range, it will abstract away 
that graphemes consist of different sized code-points. And the docs at 
https://dlang.org/phobos/std_uni.html#byGrapheme show an example using 
this kind of range:

auto gText = text.byGrapheme;

gText.take(3);
gText.drop(3);

But maybe I need to get a better understanding of the ranges stuff too...


 If you want to add/delete/change graphemes, what you *really* want is to
 use an array of Graphemes:
 
 	Grapheme[] editableGraphs;
 
 You can then splice it, insert stuff, delete stuff, whatever.
 
 When you're done with it, convert it back to string with something like
 this:
 
 	string result = editableGraphs.map!(g => g[]).joiner.to!string;


I played around with this approach...

string r1 = "Robert M. Münch";
	// Code-Units  = 16
	// Code-Points = 15
	// Graphemes   = 15

Grapheme[] gr1 = r1.byGrapheme.array;
writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string);
	//  Text = obert M. Münch
writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring);
	//  wText = obert M. Münch
writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring);
	//  dText = obert M. Münch

Why is the first letter missing? Is this a bug?

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Dec 29 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Dec 29, 2019 at 01:19:09PM +0100, Robert M. Münch via
Digitalmars-d-learn wrote:

 On 2019-12-27 19:44:59 +0000, H. S. Teoh said:


[...]

 If you want to add/delete/change graphemes, what you *really* want
 is to use an array of Graphemes:
 
 	Grapheme[] editableGraphs;
 
 You can then splice it, insert stuff, delete stuff, whatever.
 
 When you're done with it, convert it back to string with something
 like this:
 
 	string result = editableGraphs.map!(g => g[]).joiner.to!string;


 
 I played around with this approach...
 
 string r1 = "Robert M. Münch";
 	// Code-Units  = 16
 	// Code-Points = 15
 	// Graphemes   = 15
 
 Grapheme[] gr1 = r1.byGrapheme.array;
 writeln(" Text = ", gr1.map!(g => g[]).joiner.to!string);
 	//  Text = obert M. Münch
 writeln("wText = ", gr1.map!(g => g[]).joiner.to!wstring);
 	//  wText = obert M. Münch
 writeln("dText = ", gr1.map!(g => g[]).joiner.to!dstring);
 	//  dText = obert M. Münch
 
 Why is the first letter missing? Is this a bug?


[...]

I suspect there's a scope-related bug/issue somewhere here.  I did some
experiments and discovered that using foreach to iterate over a
Grapheme[] is OK, but somehow when using Grapheme[] with .map to slice
over each one, I get random UTF-8 encoding errors and missing
characters.

I suspect the cause is that whatever Grapheme.opSlice returns is going
out-of-scope when used with .map, that's why it's malfunctioning. The
last time I looked at the Grapheme code, there's a bunch of
memory-related stuff involving dtors that's *probably* the cause of this
problem.

Please file a bug for this.


T

-- 
Life is complex. It consists of real and imaginary parts. -- YHL
Dec 30 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:
[...]

 I suspect the cause is that whatever Grapheme.opSlice returns is going
 out-of-scope when used with .map, that's why it's malfunctioning.


[...]

Haha, it's actually right there in the Grapheme docs for the opSlice
overloads:

        Random-access range over Grapheme's $(CHARACTERS).

        Warning: Invalidates when this Grapheme leaves the scope,
        attempts to use it then would lead to memory corruption.

Looks like when you use .map over the Grapheme, it gets copied into a
temporary, which gets invalidated when map.front returns.  Somewhere
we're missing a 'scope' qualifier...


T

-- 
War doesn't prove who's right, just who's left. -- BSD Games' Fortune
Dec 30 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/30/19 6:31 PM, H. S. Teoh wrote:

 On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:
 [...]

 I suspect the cause is that whatever Grapheme.opSlice returns is going
 out-of-scope when used with .map, that's why it's malfunctioning.


 [...]
 
 Haha, it's actually right there in the Grapheme docs for the opSlice
 overloads:
 
          Random-access range over Grapheme's $(CHARACTERS).
 
          Warning: Invalidates when this Grapheme leaves the scope,
          attempts to use it then would lead to memory corruption.
 
 Looks like when you use .map over the Grapheme, it gets copied into a
 temporary, which gets invalidated when map.front returns.  Somewhere
 we're missing a 'scope' qualifier...
 


Then the original example should be fixable by putting "ref" in for all 
the lambdas.

But this is kind of disturbing. Why does the grapheme do this? The 
original data is not scoped.

e.g.:

writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);

-Steve
Dec 31 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 On 12/30/19 6:31 PM, H. S. Teoh wrote:

 On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:




[...]

 Haha, it's actually right there in the Grapheme docs for the opSlice
 overloads:
 
          Random-access range over Grapheme's $(CHARACTERS).
 
          Warning: Invalidates when this Grapheme leaves the scope,
          attempts to use it then would lead to memory corruption.
 
 Looks like when you use .map over the Grapheme, it gets copied into
 a temporary, which gets invalidated when map.front returns.
 Somewhere we're missing a 'scope' qualifier...




[...]

 Then the original example should be fixable by putting "ref" in for
 all the lambdas.
 
 But this is kind of disturbing. Why does the grapheme do this? The
 original data is not scoped.


Honestly I have no idea, but glancing at the code in std.uni reveals
that the returned slice is actually a wrapper object that contains a
pointer to the parent Grapheme object.  So if the parent was a
temporary and goes out of scope before the wrapper does, we're left with
a dangling pointer.



 e.g.:
 
 writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);


[...]

Unfortunately this doesn't work. Somehow the ref parameter doesn't match
whatever it is std.algorithm.map is trying to pass to it.


T

-- 
They say that "guns don't kill people, people kill people." Well I think the
gun helps. If you just stood there and yelled BANG, I don't think you'd kill
too many people. -- Eddie Izzard, Dressed to Kill
Dec 31 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/31/19 2:58 PM, H. S. Teoh wrote:

 On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 e.g.:

 writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);


 [...]
 
 Unfortunately this doesn't work. Somehow the ref parameter doesn't match
 whatever it is std.algorithm.map is trying to pass to it.


Huh, it seemed to work for me. Got the full "Robert" with an R. map does 
support ref-ness. Maybe you didn't put ref in the right place?

-Steve
Dec 31 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 On 12/31/19 2:58 PM, H. S. Teoh wrote:

 On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 e.g.:
 
 writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);


 [...]
 
 Unfortunately this doesn't work. Somehow the ref parameter doesn't
 match whatever it is std.algorithm.map is trying to pass to it.


 
 Huh, it seemed to work for me. Got the full "Robert" with an R. map
 does support ref-ness. Maybe you didn't put ref in the right place?


Here's my full non-working code:

	import std;
	void main() {
		auto x = "Bla\u0301hbla\u0310h\u0309!";
		auto r = x.byGrapheme;
		writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
	}

The compiler says:

	/usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D
main.__lambda1 cannot deduce function from argument types !()(Grapheme),
candidates are:
	test.d(5):        __lambda1
	/usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template instance
test.main.MapResult!(__lambda1, Result!string) error instantiating
	test.d(5):        instantiated from here: map!(Result!string)

What did I do wrong?


T

-- 
What do you get if you drop a piano down a mineshaft? A flat minor.
Dec 31 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/31/19 4:22 PM, H. S. Teoh wrote:

 On Tue, Dec 31, 2019 at 04:02:47PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 On 12/31/19 2:58 PM, H. S. Teoh wrote:

 On Tue, Dec 31, 2019 at 09:33:14AM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 e.g.:

 writeln(" Text = ", gr1.map!((ref g) => g[]).joiner.to!string);


 [...]

 Unfortunately this doesn't work. Somehow the ref parameter doesn't
 match whatever it is std.algorithm.map is trying to pass to it.


 Huh, it seemed to work for me. Got the full "Robert" with an R. map
 does support ref-ness. Maybe you didn't put ref in the right place?


 
 Here's my full non-working code:
 
 	import std;
 	void main() {
 		auto x = "Bla\u0301hbla\u0310h\u0309!";
 		auto r = x.byGrapheme;
 		writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
 	}
 
 The compiler says:
 
 	/usr/src/d/phobos/std/algorithm/iteration.d(604): Error: template D
main.__lambda1 cannot deduce function from argument types !()(Grapheme),
candidates are:
 	test.d(5):        __lambda1
 	/usr/src/d/phobos/std/algorithm/iteration.d(499): Error: template instance
test.main.MapResult!(__lambda1, Result!string) error instantiating
 	test.d(5):        instantiated from here: map!(Result!string)
 
 What did I do wrong?


auto r = x.byGrapheme.array;

This is how Robert originally had it if you look a few messages up.

Otherwise, it's not an lvalue.

The fact that a Grapheme's return requires you keep the grapheme in 
scope for operations seems completely incorrect and dangerous IMO (note 
that operators are going to always have a ref this, even when called on 
an rvalue). So even though using ref works, I think the underlying issue 
here really is the lifetime problem.

-Steve
Dec 31 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2019 at 04:36:56PM -0500, Steven Schveighoffer via
Digitalmars-d-learn wrote:

 On 12/31/19 4:22 PM, H. S. Teoh wrote:


[...]

 	import std;
 	void main() {
 		auto x = "Bla\u0301hbla\u0310h\u0309!";
 		auto r = x.byGrapheme;
 		writefln("%s", r.map!((ref g) => g[]).joiner.to!string);
 	}




[...]

 What did I do wrong?


 
 auto r = x.byGrapheme.array;


Haha, in my hurry I totally forgot about the .array. Mea culpa.


[...]

 The fact that a Grapheme's return requires you keep the grapheme in
 scope for operations seems completely incorrect and dangerous IMO
 (note that operators are going to always have a ref this, even when
 called on an rvalue). So even though using ref works, I think the
 underlying issue here really is the lifetime problem.


[...]

After my wrong recollection of the history surrounding indexOf vs.
countUntil, I'm not sure I can rely on my memory anymore, :-P but AIUI
Dmitri implemented it this way because he wanted to avoid allocations
(GC or otherwise) in the most common case of Grapheme containing just a
small number of code points (usually 1 or 2). When the number of
combining diacritics exceed the size of the Grapheme struct, then it
would quietly switch to malloc or some such for holding the data.  My
guess is that this is the reason for passing &this to the wrapper range
returned by opSlice(). And possibly it's also to allow mutation of the
Grapheme via the returned slice?

Perhaps this whole approach should be looked at again. Certainly, unless
I'm missing something, it *ought* to be possible to implement Grapheme
in a way that doesn't require this scoped reference business.


T

-- 
The diminished 7th chord is the most flexible and fear-instilling chord. Use it
often, use it unsparingly, to subdue your listeners into submission!
Dec 31 2019
prev sibling parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2019-12-31 21:36:56 +0000, Steven Schveighoffer said:


 The fact that a Grapheme's return requires you keep the grapheme in 
 scope for operations seems completely incorrect and dangerous IMO (note 
 that operators are going to always have a ref this, even when called on 
 an rvalue). So even though using ref works, I think the underlying 
 issue here really is the lifetime problem.


Thanks for all the answers, pretty enlighting even I'm not sure I get 
everything 100%.

So, what to do for now? File a bug-report? What needs to be fixed?

I'm using the ref approach for now, in hope it will be OK for my 
use-case, which is converting a wstring to a grapheme[], alter the 
array, and map it back to a wstring. Sounds like a lot of porcessing 
for handling unicode text, but I don't think it gets a lot simpler than 
that.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Jan 04 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 04, 2020 at 08:19:14PM +0100, Robert M. Münch via
Digitalmars-d-learn wrote:

 On 2019-12-31 21:36:56 +0000, Steven Schveighoffer said:
 

 The fact that a Grapheme's return requires you keep the grapheme in
 scope for operations seems completely incorrect and dangerous IMO
 (note that operators are going to always have a ref this, even when
 called on an rvalue). So even though using ref works, I think the
 underlying issue here really is the lifetime problem.


 
 Thanks for all the answers, pretty enlighting even I'm not sure I get
 everything 100%.
 
 So, what to do for now? File a bug-report? What needs to be fixed?


At a minimum, I think we should file a bug report to investigate whether
Grapheme.opSlice can be implemented differently, such that we avoid this
obscure referential behaviour that makes it hard to work with in complex
code. I'm not sure if this is possible, but IMO we should at least
investigate the possibilities.



 I'm using the ref approach for now, in hope it will be OK for my
 use-case, which is converting a wstring to a grapheme[], alter the
 array, and map it back to a wstring. Sounds like a lot of porcessing
 for handling unicode text, but I don't think it gets a lot simpler
 than that.


[...]

Unicode is a beast. Be glad that you can even do this in the first
place.  If I were writing this in C, I wouldn't even know where to
begin!


T

-- 
No! I'm not in denial!
Jan 04 2020
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2020-01-05 04:18:34 +0000, H. S. Teoh said:


 At a minimum, I think we should file a bug report to investigate whether
 Grapheme.opSlice can be implemented differently, such that we avoid this
 obscure referential behaviour that makes it hard to work with in complex
 code. I'm not sure if this is possible, but IMO we should at least
 investigate the possibilities.


Done... my first bug report :-) I copied togehter all the findings from 
this thread.

-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Jan 06 2020
parent Alex <sascha.orlov gmail.com> writes:
On Monday, 6 January 2020 at 08:39:19 UTC, Robert M. Münch wrote:

 On 2020-01-05 04:18:34 +0000, H. S. Teoh said:


 At a minimum, I think we should file a bug report to 
 investigate whether
 Grapheme.opSlice can be implemented differently, such that we 
 avoid this
 obscure referential behaviour that makes it hard to work with 
 in complex
 code. I'm not sure if this is possible, but IMO we should at 
 least
 investigate the possibilities.


 Done... my first bug report :-) I copied togehter all the 
 findings from this thread.


For the sake of completeness
https://issues.dlang.org/show_bug.cgi?id=20483
Jan 06 2020
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Dec 30, 2019 at 03:31:31PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:

 On Mon, Dec 30, 2019 at 03:09:58PM -0800, H. S. Teoh via Digitalmars-d-learn
wrote:
 [...]

 I suspect the cause is that whatever Grapheme.opSlice returns is
 going out-of-scope when used with .map, that's why it's
 malfunctioning.


 [...]
 
 Haha, it's actually right there in the Grapheme docs for the opSlice
 overloads:
 
         Random-access range over Grapheme's $(CHARACTERS).
 
         Warning: Invalidates when this Grapheme leaves the scope,
         attempts to use it then would lead to memory corruption.
 
 Looks like when you use .map over the Grapheme, it gets copied into a
 temporary, which gets invalidated when map.front returns.  Somewhere
 we're missing a 'scope' qualifier...


[...]

Indeed, compiling with dmd -dip1000 produces this error message:

	test.d(15): Error: returning g.opSlice() escapes a reference to parameter g,
perhaps annotate with return
	/usr/src/d/phobos/std/algorithm/iteration.d(499):        instantiated from
here: MapResult!(__lambda1, Grapheme[])
	test.d(15):        instantiated from here: map!(Grapheme[])

Not the most helpful message (the annotation has to go in Phobos code,
not in user code), but it does at least point to the cause of the
problem.


T

-- 
What doesn't kill me makes me stranger.
Dec 30 2019
prev sibling parent Alexandru Ermicioi <alexandru.ermicioi gmail.com> writes:
On Friday, 27 December 2019 at 17:26:58 UTC, Robert M. Münch 
wrote:

 ...


There are set of range interfaces that can be used to mask range 
type. Check for 
https://dlang.org/library/std/range/interfaces/input_range.html 
for starting point, and for 
https://dlang.org/library/std/range/interfaces/input_range_object.html for
wrapping any range to those interfaces.

Note: resulting wrapped range is an object and has reference 
semantics, beware of using it directly with other range 
algorithms as they can consume your range.

Best regards,
Alexandru.
Dec 30 2019