www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - UTF-8 Unicode operators vs digraphs

reply Dlighted <test example.com> writes:
The site told me I should make a new thread rather than replying 
to a years old one. Forgive me if I made a completely useless 
offtopic thread that wasted everyone's time.
I thought about Unicode operators multiple times over the years 
and it's been mentioned on the D forum more than once:
https://forum.dlang.org/thread/nujfnoneihykkkfzeeaj forum.dlang.org
https://forum.dlang.org/post/ahxrzlvznpdutwvvmlfv forum.dlang.org

I think especially digraphs like >= <= != are a relict of the 
past.
They are stand-ins for ≥ ≤ ≠ just like C has for many other now 
common symbols. I'm supposed to do mental gymnastics and visually 
overlap the symbols to see the actual symbol.

Until C23:

??/	\\
??'	^
??(	[
??)	]
??!	|
??<	{
??>	}
??-	~
Since C 95:
<:	[
:>	]
<%	{
%>	}

https://en.wikipedia.org/wiki/Digraphs_and_trigraphs_(programming)

Keeping the old for compatibility and ease of typing makes sense 
but it doesn't really make sense to force the old ones.
Is != /= ~= -ne <> really better than ≠? (Okay, I haven't used 
Haskell yet. I'm strawmanning.)

The Julia programming language supports all the Unicode math 
symbols. Maybe a step to far since programming and math are 
different.

Personally I can in fact type some math symbols on my keyboard 
layout natively on higher layers without using compose. ¬ ∨ ∧
And every Linux user (like me) can use xcompose by pressing the 
compose key or shortcut. My xcompose also has some already 
predefined. >= <= /= turn into ≥ ≤ ≠.
There are also third party keyboard layouts and IMEs for Windows. 
I have used one and it worked fine.
Maybe code formatters could convert to them too.

Implementation:
Many languages now standardize on UTF-8. UTF-8 has conquered the 
world and everything else is legacy baggage.
There is exactly **one representation** for these old 1993 
Unicode symbols in UTF-8.
It's a sequence of three bytes forming one codepoint.
Even if you don't want to deal with code points or graphemes, 
**it is no different than implementing a C trigraph** to my 
understanding. It should not matter if the byte holds an ASCII 
value or a value outside that range.
So in UTF-8 ≤ is a trigraph of 0xE2, 0x89, 0xA4. It's easier than 
actual digraphs or trigraphs because ≤ can't be part of something 
else due to the way UTF-8 was designed. The only question I have 
is why we haven't done it yet.

Personally I find && quite ugly too. ∧ ∨ for && || are more 
readable. I know Walter doesn't like the keywords _and or not_ 
because they've been used as variable names. I read it somewhere 
on the forum. But I guess that would be too controversial for 
people who don't know math well. I actually like the words but I 
use a Germanic language, so I'm naturally biased whereas math is 
universal.

However the symbols ≥ ≤ ≠ would be an undeniable upgrade. So, uh 
... **Why not?**

---

https://forum.dlang.org/post/u571i2$1bv3$1 digitalmars.com
On Wednesday, 31 May 2023 at 08:47:04 UTC, Walter Bright wrote:

 I use putty a lot to access computers remotely in text mode. 
 With some experimentation, some Unicode characters are 
 rendered, but some aren't, like the 69 quotes. Maybe the 
 programming world isn't quite ready for them yet.


He probably didn't use Cygwin. Console applications (especially 
cmd.exe) have loads of problems on Windows without 
https://cygwin.com/
With Cygwin everything always ran fine for me without Unicode 
issues.
The new Windows Terminal might also work.
https://apps.microsoft.com/detail/9n0dx20hk701

The old Windows parts will always be like this. It's a frozen 
museum. You gotta let go. Change habits.
However all modern parts of it shouldn't have any problems at 
all. And those 1993 math symbols are covered by fonts on every 
system today.

Those math symbols are only one code point. They work even in 
relatively ancient software that has problems with emojis such as 
Notepad++ and other scintilla engine based text editors.
Aug 11
next sibling parent reply monkyyy <crazymonkyyy gmail.com> writes:
On Tuesday, 11 August 2026 at 17:34:00 UTC, Dlighted wrote:

 The site told me I should make a new thread rather than 
 replying to a years old one. Forgive me if I made a completely 
 useless offtopic thread that wasted everyone's time.
 I thought about Unicode operators multiple times over the years 
 and it's been mentioned on the D forum more than once:
 https://forum.dlang.org/thread/nujfnoneihykkkfzeeaj forum.dlang.org
 https://forum.dlang.org/post/ahxrzlvznpdutwvvmlfv forum.dlang.org

 [...]


D started life as a c compiler 30 years ago and maintains 
compatibility with c in many bad ways; mentioning modern c++ 
versions probably doesn't help you convince walter, mentioning a 
formatter wont help you convince adr

D doesnt have a standard text editor so I bet for the foreseeable 
future walter will want to maintain ascii just assuming that some 
15 year old code only he runs will be mildly inconvenienced by 
him checking if it even does support it

Possibly good arguments would be if they are part of the top 10 
monospaced fonts and if gnu-c supports it; id check that.
Aug 11
parent Dlighted <test example.com> writes:
On Tuesday, 11 August 2026 at 17:57:32 UTC, monkyyy wrote:

 D started life as a c compiler 30 years ago and maintains 
 compatibility with c in many bad ways; mentioning modern c++ 
 versions probably doesn't help you convince walter, mentioning 
 a formatter wont help you convince adr


I haven't mentioned C++ at all.


 D doesnt have a standard text editor so I bet for the 
 foreseeable future walter will want to maintain ascii just 
 assuming that some 15 year old code only he runs will be mildly 
 inconvenienced by him checking if it even does support it


UTF-8 works with every text editor that came out in the last 25 
years.


 Possibly good arguments would be if they are part of the top 10 
 monospaced fonts and if gnu-c supports it; id check that.


Pretty sure the top 10 monospace fonts all support that.
D does not support the digraphs or trigraphs from C in D code. It 
is already putting accuracy over total keyboard layout 
compatibility. And who says the old digraphs need to go 
immediately?
Aug 11
prev sibling next sibling parent reply user1234 <user1234 12.de> writes:
On Tuesday, 11 August 2026 at 17:34:00 UTC, Dlighted wrote:

 [...]
 I think especially digraphs like >= <= != are a relict of the 
 past.
 They are stand-ins for ≥ ≤ ≠ just like C has for many other now 
 common symbols. I'm supposed to do mental gymnastics and 
 visually overlap the symbols to see the actual symbol.


This part of the problem is solved if you use a programming font 
with ligatures (FiraCode, JetBrainsMono, etc.).
Aug 11
next sibling parent reply H. S. Teoh <hsteoh qfbox.info> writes:
On Wednesday, 12 August 2026 at 02:12:31 UTC, user1234 wrote:

 On Tuesday, 11 August 2026 at 17:34:00 UTC, Dlighted wrote:

 [...]
 I think especially digraphs like >= <= != are a relict of the 
 past.
 They are stand-ins for ≥ ≤ ≠ just like C has for many other 
 now common symbols. I'm supposed to do mental gymnastics and 
 visually overlap the symbols to see the actual symbol.


 This part of the problem is solved if you use a programming 
 font with ligatures (FiraCode, JetBrainsMono, etc.).


Now *this* is what I call the "Walter mode of thinking". I.e., 
solving a problem with minimal effort instead of over-engineering 
a complex solution that introduces more issues than it solves.

Proposed solution: add digraphs/trigraphs to the language.
Cost: extensive changes to the parser, language spec, testsuite, 
and potentially affecting existing code unintentionally. 
Significant effort.

The "Walter-style" solution: just install a font with ligatures.
Cost: download a couple of font files.  No code changes required, 
zero code breakage.  Zero effort.

I approve of the second option. ;-)
Aug 11
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/11/2026 7:30 PM, H. S. Teoh wrote:

 Now *this* is what I call the "Walter mode of thinking". I.e., solving a
problem 
 with minimal effort instead of over-engineering a complex solution that 
 introduces more issues than it solves.


I like the cut of your jib.
Aug 12
prev sibling parent Dlighted <test example.com> writes:
On Wednesday, 12 August 2026 at 02:12:31 UTC, user1234 wrote:

 On Tuesday, 11 August 2026 at 17:34:00 UTC, Dlighted wrote:

 [...]
 I think especially digraphs like >= <= != are a relict of the 
 past.
 They are stand-ins for ≥ ≤ ≠ just like C has for many other 
 now common symbols. I'm supposed to do mental gymnastics and 
 visually overlap the symbols to see the actual symbol.


 This part of the problem is solved if you use a programming 
 font with ligatures (FiraCode, JetBrainsMono, etc.).


That doesn't actually work across github or similar, terminal 
(git) and text editor. I bet a lot of older text editors and 
terminals don't support ligatures in the first place. Simply 
using the right symbol will always display right.

The ligatures for && and || look almost the same as no ligatures 
in Fira Code and Jetbrains Mono. I doubt most developers use 
these fonts anyway. That's not drastically going to improve 
readability.

Your reasoning can be used to justify anything blocking progress. 
HTML entities for every non-ascii character in HTML for example.
Aug 11
prev sibling next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, August 11, 2026 11:34:00 AM Mountain Daylight Time Dlighted via
Digitalmars-d wrote:

 I think especially digraphs like >= <= != are a relict of the
 past.
 They are stand-ins for ≥ ≤ ≠ just like C has for many other now
 common symbols. I'm supposed to do mental gymnastics and visually
 overlap the symbols to see the actual symbol.

 Until C23:

 ??/   \\
 ??'   ^
 ??(   [
 ??)   ]
 ??!   |
 ??<   {
 ??>   }
 ??-   ~
 Since C 95:
 <:    [
 :>    ]
 <%    {
 %>    }

 https://en.wikipedia.org/wiki/Digraphs_and_trigraphs_(programming)

 Keeping the old for compatibility and ease of typing makes sense
 but it doesn't really make sense to force the old ones.
 Is != /= ~= -ne <> really better than ≠? (Okay, I haven't used
 Haskell yet. I'm strawmanning.)


Those symbols are not on a typical keyboard, which would make them a royal
pain for most people to deal with. It's far easier to program when all of
the relevant symbols can be trivially typed without having to figure out how
to access symbols which are not natively on the keyboard.

And honestly, I don't see any problem with the symbols that D currently uses
for comparisons. They're quite standard and well understood. I don't see how
using Unicode symbols instead actually adds any value when the current
symbols work perfectly fine and are easy to type.

Personally, while I agree that it's good that D natively supports Unicode,
as a general rule, I don't think that even non-ASCII identifiers should be
used, because it makes the code harder to maintain. Maybe it makes sense to
use Unicode identifiers with a proprietary codebase where everyone involved
is using something like a Japanese or Russian keyboard, but for open source,
it generally causes problems to use identifiers which are not ASCII. Having
operators which aren't ASCII would be even worse.

- Jonathan M Davis
Aug 12
next sibling parent Dlighted <test example.com> writes:
On Wednesday, 12 August 2026 at 09:29:51 UTC, Jonathan M Davis 
wrote:

 On Tuesday, August 11, 2026 11:34:00 AM Mountain Daylight Time 
 Those symbols are not on a typical keyboard, which would make 
 them a royal pain for most people to deal with. It's far easier 
 to program when all of the relevant symbols can be trivially 
 typed without having to figure out how to access symbols which 
 are not natively on the keyboard.


The same was true for {} []. Actually ≠ is easier to deal with, 
even if you can't type it, than != because it is one unique byte 
sequence/code point/grapheme. You always select the whole symbol, 
moving the caret with arrow keys moves it over the whole operator 
and pressing backspace deletes the whole operator.
It's more of a 'qwerty is the best' argument. Indians and Chinese 
use IME for their languages.
Anyway, it was supposed to be a laid-back thread. Not forcing 
immediate inclusion into D.


 it generally causes problems to use identifiers which are not 
 ASCII. Having operators which aren't ASCII would be even worse.


I doubt it. As long as there are alternative digraphs why would 
it cause problems for anyone? UTF-8 is the new ASCII. It's like 
worrying about compatibility with EBCDIC. It's not a big deal. 
Especially not with symbols that have exactly one representation 
and no lookalikes.
Aug 12
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/12/2026 2:29 AM, Jonathan M Davis wrote:

 Personally, while I agree that it's good that D natively supports Unicode,
 as a general rule, I don't think that even non-ASCII identifiers should be used


D supports non-ASCII identifiers only for C/C++ compatibility. Personally, they 
are an abomination. Even more wretched is the support for "normalized"
identifiers.

Putting the notion of normalized Unicode into Unicode was one of the most 
execrable decisions ever.
Aug 12
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 8/11/26 19:34, Dlighted wrote:

 The site told me I should make a new thread rather than replying to a 
 years old one. Forgive me if I made a completely useless offtopic thread 
 that wasted everyone's time.
 I thought about Unicode operators multiple times over the years and it's 
 been mentioned on the D forum more than once:
 https://forum.dlang.org/thread/nujfnoneihykkkfzeeaj forum.dlang.org
 https://forum.dlang.org/post/ahxrzlvznpdutwvvmlfv forum.dlang.org
 ...
 
 Personally I can in fact type some math symbols on my keyboard layout 
 natively on higher layers without using compose. ¬ ∨ ∧
 And every Linux user (like me) can use xcompose by pressing the compose 
 key or shortcut. My xcompose also has some already predefined. >= <= /= 
 turn into ≥ ≤ ≠.
 There are also third party keyboard layouts and IMEs for Windows. I have 
 used one and it worked fine.
 Maybe code formatters could convert to them too.
 ...


I am just using an emacs input mode. Plugins also exist for other editors.

My own language has Unicode operator support (though it's fully a 
formatting choice, everything has an ASCII alternative). It has been an 
uphill battle. E.g., see:

https://www.reddit.com/r/programming/comments/i4uavw/meet_silq_the_first_intuitive_highlevel_language/
https://www.reddit.com/r/programming/comments/ii4t0r/meet_silq_the_first_intuitive_highlevel_language/

Instead of discussing what is interesting about the language, a good 
chunk of the comments are just low-effort anti-Unicode-syntax posts. 
It's not considered sufficiently "normal" and people like writing snarky 
comments, so you always get a lot of spam whenever you are trying to 
share something.

One reddit user put it best:
"I feel dump. Can't understand what's going one how the syntaxt work."

Also, one drawback of the Unicode syntax design has been that e.g. 
OpenAI ignores 𝔹 when reading HTML pages, so GPT is actually not able 
to read documentation if it is written with Unicode. It is very adamant 
though that the syntax is `x := 0:;` and `x := vector(n, 0):^n` instead 
of `x := 0:𝔹;` and `x := vector(n, 0):𝔹^n`. Arguably this part would 
probably not happen with symbols D is likely to use.


 ...
 Personally I find && quite ugly too. ∧ ∨ for && || are more readable. I 
 know Walter doesn't like the keywords _and or not_ because they've been 
 used as variable names.


I think `∧` and `∨` I think are not great choices for `&&` and `||`, 
because `&&` and `||` are short-circuiting operators. Then `and` and 
`or` don't match either, for the same reason (though I guess Python does 
this anyway). One alternative I have seen that actually makes sense is 
`and then` and `or else`, but it's a bit verbose.


 I read it somewhere on the forum. But I guess 
 that would be too controversial for people who don't know math well. I 
 actually like the words but I use a Germanic language, so I'm naturally 
 biased whereas math is universal.
 ...


Short-circuiting is not universal.


 However the symbols ≥ ≤ ≠ would be an undeniable upgrade. So, uh ... 
 **Why not?**
 ...


In my experience: Humanity is not ready for it, it will blow people's 
minds and break people's pet software, even if it was written in 2026.

Not a reason not to do it, just a reason why I predict it will not 
happen anytime soon in D.
Aug 12
next sibling parent reply Dlighted <test example.com> writes:
On Wednesday, 12 August 2026 at 11:52:44 UTC, Timon Gehr wrote:

 Instead of discussing what is interesting about the language, a 
 good chunk of the comments are just low-effort 
 anti-Unicode-syntax posts. It's not considered sufficiently 
 "normal" and people like writing snarky comments, so you always 
 get a lot of spam whenever you are trying to share something.


Read this post and reflect.


 I think `∧` and `∨` I think are not great choices for `&&` and 
 `||`, because `&&` and `||` are short-circuiting operators. 
 Then `and` and `or` don't match either, for the same reason 
 (though I guess Python does this anyway). One alternative I 
 have seen that actually makes sense is `and then` and `or 
 else`, but it's a bit verbose.


There's a long list of languages that use "and" and "or".
Pascal, Ada, Fortran, Lua, Python, PHP, R and probably others.
Ruby and Perl have them but with different precedence.


OpenAI ignores 𝔹 when reading HTML pages, so GPT is actually not 
able to read documentation if it is written with Unicode.


Gemini says its a problem with the letter you chose and not with 
Unicode in general:

Modern AI models (like ChatGPT) can read both symbols, but they 
process them through a tokenizer that breaks text down into bytes 
or smaller subword chunks.
- ∨ (Logical OR / Down Tack): This is a standard 
mathematical/logical symbol present in Unicode (U+2228). Because 
it is frequently used in math, logic, and programming datasets 
online, tokenizers often handle it gracefully, sometimes mapping 
it directly or splitting it efficiently.
- 𝔹 (Blackboard Bold Capital B): This belongs to the Unicode 
Mathematical Alphanumeric Symbols block (U+1D539). These symbols 
are notorious for causing tokenization inefficiencies. Because 
they look visually similar to a standard Latin "B" to a human, 
but have completely different underlying byte codes, tokenizers 
often fragment them into multiple awkward tokens.
Aug 12
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/12/2026 5:30 AM, Dlighted wrote:

 - 𝔹 (Blackboard Bold Capital B): This belongs to the Unicode Mathematical 
 Alphanumeric Symbols block (U+1D539). These symbols are notorious for causing 
 tokenization inefficiencies. Because they look visually similar to a standard 
 Latin "B" to a human, but have completely different underlying byte codes, 
 tokenizers often fragment them into multiple awkward tokens.


Another one of my major gripes about Unicode - characters that render 
identically but are completely different.
Aug 12
parent reply Dlighted <test example.com> writes:
On Wednesday, 12 August 2026 at 19:50:18 UTC, Walter Bright wrote:

 I suppose one could adapt their code editor to display != as ≠, 
 but then aligned columns may not work well.
 So they stretch out ≠ to be two characters wide? Code editors 
 tend to use monospace fonts.


Today a monospace font is often not actually monospaced. The font 
determines the width of each letter. And it can use an opentype 
ligature glyph that is simply two columns wide. It also means 
that it does not break up the ligature when it reaches the end of 
the screen but puts the whole operator >= on the next line when 
visual line wrapping is enabled. I checked in the Kate text 
editor with Fira Code just now.


 Another one of my major gripes about Unicode - characters that 
 render identically but are completely different.



 Putting the notion of normalized Unicode into Unicode was one 
 of the most execrable decisions ever.


I hope I don't misunderstand or sound rude.
It is unavoidable for letters from different scripts to look the 
same in some fonts when supporting many scripts.
However the problem is far older than Unicode. I (i) and l (L) in 
English fonts of the last centuries are often stylized to look 
the same.
Historically capital I (i) and capital J (j) were the same letter 
in handwriting and old fonts because j was merely a calligraphic 
variant of i (and often only for lowercase) used in front of 
vocals just like "r rotunda" (ꝛ) and r. It wasn't supposed to 
change pronunciation but indicate it. It was not supposed to be 
an own letter.
In fact some alphabetic enumerations in German books still skip 
the letter j! a) b) c) d) e) f) g) h) i) k)
Here's a picture of it:
https://www.praxis-foerderdiagnostik.de/wp-content/uploads/2021/02/neu_Klammerreg_Ue1-scaled.jpg
There are also many names with I or J variants that are actually 
the same name! Spoken the same way.
I guess you'd have to _normalize_ them to find out.

 From my understanding normalization does not apply to ≥. It 
doesn't change when normalized to any representation because it 
has a fixed unique representation.
Aug 13
next sibling parent Dlighted <test example.com> writes:
On Thursday, 13 August 2026 at 07:20:08 UTC, Dlighted wrote:

 Today a monospace font is often not actually monospaced. The 
 font determines the width of each letter. And it can use an 
 opentype ligature glyph that is simply two columns wide. It 
 also means that it does not break up the ligature when it 
 reaches the end of the screen but puts the whole operator >= on 
 the next line when visual line wrapping is enabled. I checked 
 in the Kate text editor with Fira Code just now.


I have to correct myself. Wrapping of ligatures is broken in Kate.
I tried a string like _________________>= and it displayed 
nothing at the end of the line and only the right half of the 
ligature in the next line.
Aug 13
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/13/2026 12:20 AM, Dlighted wrote:

 It is unavoidable for letters from different scripts to look the same in some 
 fonts when supporting many scripts.


What Unicode has done wrong is attach semantic meaning to code points. This is 
all wrong. When you read a printed page, the semantic meaning depends on the 
context, not the glyph.

For example, there is the letter 'a' and the mathematical variable 'a'. They
are 
different code points, but they are identical. Have you ever read a math book 
and had trouble distinguishing the two? I haven't.

'a' can have endless semantic meanings depending on the context. Unicode trying 
to assign different code points to them is hopeless and makes Unicode more or 
less unusable.


 I guess you'd have to _normalize_ them to find out.


Normalizing in Unicode means two different code points represent the same code 
point.

Bottom line: you should be able to print a page written in Unicode, then OCR it 
and get back the same Unicode text (at least for the non-blank points).
Instead, 
we have Unicode that means one thing to the human reader and something totally 
different to a computer reading it.
Aug 13
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 14 August 2026 at 00:17:36 UTC, Walter Bright wrote:

 Bottom line: you should be able to print a page written in 
 Unicode, then OCR it and get back the same Unicode text (at 
 least for the non-blank points).


Plenty of fonts make I and l look pretty much the same. Sometimes 
1 too. And 0 and O may be hard to see.

Maybe they shouldn't, but enough do that I'm glad there is 
semantic meaning attached independently of the appearance some 
design committee decided ought to work for everyone.
Aug 13
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/13/2026 5:41 PM, Adam D. Ruppe wrote:

 Plenty of fonts make I and l look pretty much the same. Sometimes 1 too. And 0 
 and O may be hard to see.


Unicode should not be a font. Although for more crazy reasons they did put some 
fonts in.


 Maybe they shouldn't, but enough do that I'm glad there is semantic meaning 
 attached independently of the appearance some design committee decided ought
to 
 work for everyone.


Fonts should be selected by style sheets, not Unicode.
Aug 13
prev sibling parent Mindy Batek (0xEAB) <desisma heidel.beer> writes:
On Friday, 14 August 2026 at 00:41:07 UTC, Adam D. Ruppe wrote:

 Maybe they shouldn't, but enough do that I'm glad there is 
 semantic meaning attached independently of the appearance some 
 design committee decided ought to work for everyone.


Yeah, also depending on the context, «Ⅳ» and «IV» may or may not 
be the same.
Good luck having your screenreader do the differentiation…
Aug 14
prev sibling next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 14/08/2026 12:17 PM, Walter Bright wrote:

     I guess you'd have to /normalize/ them to find out.
 
 Normalizing in Unicode means two different code points represent the 
 same code point.
 
 Bottom line: you should be able to print a page written in Unicode, then 
 OCR it and get back the same Unicode text (at least for the non-blank 
 points). Instead, we have Unicode that means one thing to the human 
 reader and something totally different to a computer reading it.


This is not correct. To explain why:
A glyph is the image that gets rendered to screen, print wherever.

A code point is a value between 0 to 0x10FFFF, that represents a character.

Code points are divided between starters, and non-starters. This is a 
property of each code point that is provided by a table.

One or more starters and zero or more non-starters make up a glyph.
When one starter is used and zero or more non-starters its called a 
grapheme.
When more than one starter is used, its referred to as a ligature.

Unicode defines the relationship to a glyph for one starter and zero or 
more non-starters. It does not define multiple starters.

Within Unicode when a starter is equivalent to a starter with 
non-starters, the former starter is called a composed character. These 
were historical additions from earlier character sets that probably 
shouldn't have been added (but ya know 1991 was a very time hardware 
wise and not all requirements were known).

And yes they are considered equivalent.
To make processing easier you normalize them so you only have to deal 
with a single form. Usually NFC (composed) to save memory.

Non-starters are sadly required, you can't go round defining all the 
combinations between them and starters. It would be an explosion well 
beyond 0x10FFFF. Instead font rendering engines compose the glyph 
dynamically.

There is no distinction between human and computer, specifically because 
normalization, normalizes it.

It is a shame that they introduced composed characters. That was a 
mighty big mistake.
Aug 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/13/2026 5:59 PM, Richard (Rikki) Andrew Cattermole wrote:

 It is a shame that they introduced composed characters. That was a mighty big 
 mistake.


Sounds like you are agreeing with me.
Aug 13
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 14/08/2026 5:18 PM, Walter Bright wrote:

 On 8/13/2026 5:59 PM, Richard (Rikki) Andrew Cattermole wrote:

 It is a shame that they introduced composed characters. That was a 
 mighty big mistake.


 
 Sounds like you are agreeing with me.


Normalization does a lot more than just handle composition characters.
Aug 14
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Aug 13, 2026 at 05:17:36PM -0700, Walter Bright via Digitalmars-d wrote:

 On 8/13/2026 12:20 AM, Dlighted wrote:

 It is unavoidable for letters from different scripts to look the
 same in some fonts when supporting many scripts.


 
 What Unicode has done wrong is attach semantic meaning to code points.
 This is all wrong. When you read a printed page, the semantic meaning
 depends on the context, not the glyph.


[...]

 Bottom line: you should be able to print a page written in Unicode,
 then OCR it and get back the same Unicode text (at least for the
 non-blank points).  Instead, we have Unicode that means one thing to
 the human reader and something totally different to a computer reading
 it.


The problem isn't as simple as you imagine.  Some scripts have the
*same* logical glyph look differently in different contexts, a *subset*
of which looks like a subset of a different glyph in another language.

Take for example, the Cyrillic т.  The cursive form looks like this: m.
To a native speaker, they are EXACTLY THE SAME LETTER, just differently
styled.  The same way an English speaker sees the same letter whether
the glyph is an upright M or a cursive M.  This is a variation that a
different font face would give you.

Here's the problem: suppose Unicode does as you propose: represent the
glyph m as a single codepoint.  Now you have a problem: т and m now must
be represented as two different codepoints, even though they are EXACTLY
THE SAME LETTER.  Because in English, t and m are different letters with
different glyphs, so you can't have just a single codepoint for т and m.
It makes no sense for an English word like "mama" to be represented the
same way as Cyrillic "тата".  The only sane solution is to code the
Cyrillic т/m and the English M/m differently.  Otherwise you still have
the same problem that multiple codepoints map to the same logical
character.

Another example is the Cyrillic п which in some fonts look like n.  You
can't seriously be proposing that we represent the Cyrillic п with the
same codepoint as the English n.  In Cyrillic, that's a stylistic
difference, in English, that's a completely different glyph.  If you
insist that the same glyph shapes must share the same codepoint, then
the Greek π should be represented as the English n too, since п and π
are equivalent, and in some fonts look identical.  If you argue that
since п and n are visually distinct shapes, they must be separate
codepoints, then I could also argue that English upright letters should
be coded differently from italics, since they *are* visually distinct,
and in many fonts involve actually different glyph shapes.

This doesn't even begin to scratch the surface of the issues that arise
when you're dealing with languages like Arabic, where the same letter
can appear in completely different shapes depending on where it occurs
in a word, and in what context it's used.

tl;dr: the issues involved are much more complex than what an English
speaker may imagine.  Unicode isn't without its silly points and
arguably backwards decisions, but given the complexity of the issue,
it's doing not too bad.


T

-- 
BREAKFAST.COM halted...Cereal Port Not Responding. -- YHL
Aug 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
P.S. I am well aware that nobody agrees with me on this. It's me against the 
universe! Tally ho!

On 8/13/2026 6:17 PM, H. S. Teoh wrote:

 The problem isn't as simple as you imagine.  Some scripts have the
 *same* logical glyph look differently in different contexts, a *subset*
 of which looks like a subset of a different glyph in another language.


The mistake stems from thinking that a code point has semantic meaning. It does 
not. It gets its meaning from the context in which it is used. Like the text in 
a book. I have no trouble discerning a German letter a from an English letter
a, 
for example.


 Take for example, the Cyrillic т.  The cursive form looks like this: m.
 To a native speaker, they are EXACTLY THE SAME LETTER, just differently
 styled.  The same way an English speaker sees the same letter whether
 the glyph is an upright M or a cursive M.  This is a variation that a
 different font face would give you.


Unicode should never have gotten into the font business. Fonts are selected by 
style sheets.



 Here's the problem: suppose Unicode does as you propose: represent the
 glyph m as a single codepoint.  Now you have a problem: т and m now must
 be represented as two different codepoints, even though they are EXACTLY
 THE SAME LETTER.  Because in English, t and m are different letters with
 different glyphs, so you can't have just a single codepoint for т and m.
 It makes no sense for an English word like "mama" to be represented the
 same way as Cyrillic "тата".  The only sane solution is to code the
 Cyrillic т/m and the English M/m differently.  Otherwise you still have
 the same problem that multiple codepoints map to the same logical
 character.


These are all problems resulting from assigning semantic information. People
who 
print books do not have these problems, because individual letters have no 
semantic meaning.



 Another example is the Cyrillic п which in some fonts look like n.  You
 can't seriously be proposing that we represent the Cyrillic п with the
 same codepoint as the English n.  In Cyrillic, that's a stylistic
 difference, in English, that's a completely different glyph.


Yes, I am serious. There is no such thing as a Cyrillic n and an English n. 
There is only n and n.


 If you
 insist that the same glyph shapes must share the same codepoint, then
 the Greek π should be represented as the English n too, since п and π
 are equivalent,


This only becomes a problem when semantic meanings are assigned to glyphs.


 and in some fonts look identical.


The font selected is (or should be) irrelevant to Unicode.


 If you argue that
 since п and n are visually distinct shapes, they must be separate
 codepoints,


Yes.


 then I could also argue that English upright letters should
 be coded differently from italics, since they *are* visually distinct,
 and in many fonts involve actually different glyph shapes.


Italics are a creature of style sheets and should not be in Unicode. After all, 
are we talking a doubling of the number of code points in order to support
italics?



 This doesn't even begin to scratch the surface of the issues that arise
 when you're dealing with languages like Arabic, where the same letter
 can appear in completely different shapes depending on where it occurs
 in a word, and in what context it's used.


I know nothing about Arabic, but it looks like this could be handled with the 
code that renders the Unicode code points. Don't get me started with the insane 
code points that reverse the direction of the text! (It's a fine way to inject 
malicious code into source code.)


 tl;dr: the issues involved are much more complex than what an English
 speaker may imagine.  Unicode isn't without its silly points and
 arguably backwards decisions, but given the complexity of the issue,
 it's doing not too bad.


It's incomprehensible, and grows worse every time the committee meets and adds 
more trinkets to it. Rikki added something like 5000 lines of code just to deal 
with C Unicode identifiers.
Aug 13
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Aug 13, 2026 at 11:13:34PM -0700, Walter Bright via Digitalmars-d wrote:
[...]

 On 8/13/2026 6:17 PM, H. S. Teoh wrote:


[...]

 Take for example, the Cyrillic т.  The cursive form looks like this:
 m.  To a native speaker, they are EXACTLY THE SAME LETTER, just
 differently styled.  The same way an English speaker sees the same
 letter whether the glyph is an upright M or a cursive M.  This is a
 variation that a different font face would give you.


 
 Unicode should never have gotten into the font business. Fonts are
 selected by style sheets.


It didn't.  Whatever made you think it did?

 
[...]

 Another example is the Cyrillic п which in some fonts look like n.
 You can't seriously be proposing that we represent the Cyrillic п
 with the same codepoint as the English n.  In Cyrillic, that's a
 stylistic difference, in English, that's a completely different
 glyph.


 
 Yes, I am serious. There is no such thing as a Cyrillic n and an
 English n.  There is only n and n.


You totally missed my point.  My point is that n and п are the SAME
THING in Cyrillic.  Just as upright L and cursive L are the same thing
in English.  If "there is only n and n", then what you're actually
saying is that Cyrillic п is the same thing as English n, even though
they have different font shapes.  You're contradicting yourself.



 If you insist that the same glyph shapes must share the same
 codepoint, then the Greek π should be represented as the English n
 too, since п and π are equivalent,


 
 This only becomes a problem when semantic meanings are assigned to
 glyphs.


I'm not assigning any semantic meaning to anything.  I'm just going by
what you propose: the same glyph shape should be assigned the same
codepoint.

The problem is, glyph shapes are font-dependent.  And different fonts
don't agree with each other.



 and in some fonts look identical.


 
 The font selected is (or should be) irrelevant to Unicode.


Exactly, and that is why basing codepoints on glyph shapes is a wrong
idea.



 If you argue that since п and n are visually distinct shapes, they
 must be separate codepoints,


 
 Yes.


Then it follows, by your argument, that italic A and upright A must have
different codepoints.  Because they have different shapes.



 then I could also argue that English upright letters should be coded
 differently from italics, since they *are* visually distinct, and in
 many fonts involve actually different glyph shapes.


 
 Italics are a creature of style sheets and should not be in Unicode.
 After all, are we talking a doubling of the number of code points in
 order to support italics?


See, here you contradict yourself.  You just said that if two glyphs
have different shapes, they must be different codepoints.  It's obvious
that upright A and italic A are different shapes.  Therefore they must
be different codepoints.

"No," you say. "They are the same letter!"

You just assigned semantic meaning to different glyph shapes.


T

-- 
MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a
16-bit patch to an 8-bit operating system originally coded for a 4-bit
microprocessor, written by a 2-bit company that can't stand 1-bit of
competition.
Aug 14
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/14/2026 7:38 AM, H. S. Teoh wrote:

 On Thu, Aug 13, 2026 at 11:13:34PM -0700, Walter Bright via Digitalmars-d
wrote:

 Unicode should never have gotten into the font business.


 
 It didn't.  Whatever made you think it did?


https://www.w3.org/TR/xml-entity-names/1D6.html

https://www.w3.org/TR/xml-entity-names/fraktur.html


 You totally missed my point.  My point is that n and п are the SAME
 THING in Cyrillic.  Just as upright L and cursive L are the same thing
 in English.  If "there is only n and n", then what you're actually
 saying is that Cyrillic п is the same thing as English n, even though
 they have different font shapes.  You're contradicting yourself.


If two code points have the same glyph, they should be the same code point.


 The problem is, glyph shapes are font-dependent.  And different fonts
 don't agree with each other.


The Unicode standard shows a picture of each glyph. If there are more than one 
with the same glyph, they should have the same code point. It's as simple as
that.


 See, here you contradict yourself.  You just said that if two glyphs
 have different shapes, they must be different codepoints.  It's obvious
 that upright A and italic A are different shapes.  Therefore they must
 be different codepoints.


Italics are a font choice.
Aug 14
next sibling parent Dlighted <test example.com> writes:
On Saturday, 15 August 2026 at 06:54:59 UTC, Walter Bright wrote:

 On 8/14/2026 7:38 AM, H. S. Teoh wrote:

 You totally missed my point.  My point is that n and п are the 
 SAME
 THING in Cyrillic.


 The Unicode standard shows a picture of each glyph. If there 
 are more than one with the same glyph, they should have the 
 same code point. It's as simple as that.


Папа = papa
It's also pronounced that way.
If every glyph, that has a lookalike, had the same grapheme, 
you'd constantly have to switch fonts to not break the other 
language or use script indicators. In other words you'd be 
reinventing codepages.

Historically Cyrillic has been changed to look more like Latin 
script.
https://linguistics.stackexchange.com/questions/28009/when-did-other-slavic-nations-adopt-the-latin-inspired-look-of-printed-cyrillic
However other languages have lookalikes too:
Armenian Small Letter Vo ո
Armenian Small Letter Seh ս
Greek Small Letter Kappa κ
Aug 15
prev sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Aug 14, 2026 at 11:54:59PM -0700, Walter Bright via Digitalmars-d wrote:
[...]

 The Unicode standard shows a picture of each glyph. If there are more
 than one with the same glyph, they should have the same code point.
 It's as simple as that.
 

 See, here you contradict yourself.  You just said that if two glyphs
 have different shapes, they must be different codepoints.  It's
 obvious that upright A and italic A are different shapes.  Therefore
 they must be different codepoints.


 
 Italics are a font choice.


This is where you're still not getting the point.

In English, the difference between italic A and upright A is a font
choice.

In Cyrillic the difference between т and m is also a font choice.

You're trying to say that т and m must be different codepoints because
they have different shapes.  That's no different from saying that italic
A and upright A must have different codepoints because they are
different shapes.

If you say that upright A and italic A are a font choice, then you must
also admit that the difference between т and m are also a font choice,
and therefore т and m must share the same codepoint.

So, which one is it?  You can't have it both ways.


T

-- 
War doesn't prove who's right, just who's left. -- BSD Games' Fortune
Aug 15
next sibling parent reply Dlighted <test example.com> writes:
On Saturday, 15 August 2026 at 15:11:10 UTC, H. S. Teoh wrote:

 This is where you're still not getting the point.

 In English, the difference between italic A and upright A is a 
 font choice.

 In Cyrillic the difference between т and m is also a font 
 choice.

 You're trying to say that т and m must be different codepoints 
 because they have different shapes.  That's no different from 
 saying that italic A and upright A must have different 
 codepoints because they are different shapes.

 If you say that upright A and italic A are a font choice, then 
 you must also admit that the difference between т and m are 
 also a font choice, and therefore т and m must share the same 
 codepoint.

 So, which one is it?  You can't have it both ways.


I bet you're confusing him even more. I'll put it this way:

- For writing to humans freedom is generally desirable. Even if 
you hate it, other people aren't all programmers. For them 
typography and culture may matter. Fail to cover it and you just 
have Unicode codepages.

- In the context of identifiers, if you wanted to prevent 
confusion of _potential_ lookalikes, you'd have to discriminate 
against specific languages/script/ranges/code-points and 
combination marks or whitelist certain ranges/code-points. 
However by choosing ASCII you are doing the same thing, since 
ASCII barely covers English. (Yes, I get it. Walter Bright would 
make O and 0 the same character. Just don't start words with O 
and it won't be a problem, right?)

Unicode is extremely large and ASCII is extremely small. If I go 
into the supermarket, most items are trash I don't even look at 
or not interesting to me for various reasons. However there are 
still interesting things that aren't bad which I don't have in my 
fridge.
Aug 15
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/15/2026 12:39 PM, Dlighted wrote:

 Walter Bright would make O and 0 the same character.


https://github.com/ioccc-src/winner/blob/master/1986/bright/bright.c

#include <stdio.h>
#define O1O printf
#define OlO putchar
#define O10 aexit
#define Ol0 strlen
#define QLQ fopen
#define OlQ fgetc
#define O1Q abs
#define QO0 for
typedef char lOL;

lOL*QI[] = {"Use:\012\011dump file\012","Unable to open file '\x25s'\012",
  "\012","   ",""};

int aexit(int x) { exit(x); return x; }

main(I,Il)
lOL*Il[];
{	FILE *L;
	unsigned lO;
	int Q,OL[' '^'0'],llO = EOF,

	O=1,l=0,lll=O+O+O+l,OQ=056;
	lOL*llL="%2x ";
	(I != 1<<1&&(O1O(QI[0]),O10(1011-1010))),
	((L = QLQ(Il[O],"r"))==0&&(O1O(QI[O],Il[O]),O10(O)));
	lO = I-(O<<l<<O);
	while (L-l,1)
	{	QO0(Q = 0L;((Q &~(0x10-O))== l);
			OL[Q++] = OlQ(L));
		if (OL[0]==llO) break;
		O1O("\0454x: ",lO);
		if (I == (1<<1))
		{	QO0(Q=Ol0(QI[O<<O<<1]);Q<Ol0(QI[0]);
			Q++)O1O((OL[Q]!=llO)?llL:QI[lll],OL[Q]);/*"
			O10(QI[1O])*/
			O1O(QI[lll]);{}
		}
		QO0 (Q=0L;Q<1<<1<<1<<1<<1;Q+=Q<0100)
		{	(OL[Q]!=llO)? /* 0010 10lOQ 000LQL */
			((D(OL[Q])==0&&(*(OL+O1Q(Q-l))=OQ)),
			OlO(OL[Q])):
			OlO(1<<(1<<1<<1)<<1);
		}
		O1O(QI[01^10^9]);
		lO+=Q+0+l;}
	}
	D(l) { return l>=' '&&l<='\~';
}
Aug 16
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/15/2026 8:11 AM, H. S. Teoh wrote:

 So, which one is it?  You can't have it both ways.


The answer is forget about semantic meaning for glyphs, and just have glyphs.

Like a printer setting out type for a book. Printed books don't have any of 
these problems and conundrums Unicode has.

Italics is simply bending the glyph to the right a bit.
Aug 16
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Sun, Aug 16, 2026 at 06:48:40PM -0700, Walter Bright via Digitalmars-d wrote:

 On 8/15/2026 8:11 AM, H. S. Teoh wrote:

 So, which one is it?  You can't have it both ways.


 
 The answer is forget about semantic meaning for glyphs, and just have
 glyphs.


What is a "glyph"?

Telling a Cyrillic user that "т" and "m" are different glyphs is exactly
the same thing as telling an English speaker that upright A and cursive
A are different glyphs.


--T
Aug 16
prev sibling next sibling parent reply Dlighted <test example.com> writes:
On Friday, 14 August 2026 at 00:17:36 UTC, Walter Bright wrote:

 On 8/13/2026 12:20 AM, Dlighted wrote:
 What Unicode has done wrong is attach semantic meaning to code 
 points. This is all wrong. When you read a printed page, the 
 semantic meaning depends on the context, not the glyph.

 For example, there is the letter 'a' and the mathematical 
 variable 'a'. They are different code points, but they are 
 identical. Have you ever read a math book and had trouble 
 distinguishing the two? I haven't.


Yes, I agree. However you can simply choose not to use 
mathemathical letters. There are other similar small missteps too 
like the prebaked fi ligature (fi) that you're not supposed to use 
and only exists for compatibility with some old codepage. I 
haven't seen my Tesseract OCR produce math letters though. It's 
not a major problem. We could talk about the pregnant man emoji 
(🫃, 🫃🏻, 🫃🏼, 🫃🏽, 🫃🏾, 🫃🏿), some widely used
symbols missing for 
political reasons, the handgun becoming a watergun and so on but 
overall I think it's not that bad.


 I guess you'd have to _normalize_ them to find out.


 Normalizing in Unicode means two different code points 
 represent the same code point.


Depends on normalization form.
Anyway, I know you mean stuff like é, è, ê having multiple 
representations but it's not much different of a problem.
Unicode allows adding all kinds of combining marks to letters 
allowing the combinations not standardized by Unicode.
 From what I've seen most modern software outputs precomposed 
forms for simple letters if available.
I guess it could have been avoided by having no precomposed forms 
but that would increase the size of common european letters and 
make them all two code points. Imagine how much software would 
give me troubles for using two code points. In the decomposed 
form, you can accidently match the uncomposed character before 
the combining mark you're combining with if you're just matching 
bytes without any awareness of graphemes.
Aug 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Some good observations!

If there is a specific code point for é, then the combining character 
alternative should be illegal.
Aug 13
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, August 14, 2026 12:41:45 AM Mountain Daylight Time Walter Bright
via Digitalmars-d wrote:

 Some good observations!

 If there is a specific code point for é, then the combining character
 alternative should be illegal.


It should have been illegal to have multiple ways to represent the same
grapheme, but some of the requirements that the Unicode guys had to deal
with made that impossible (e.g. being able to convert from older encodings
to Unicode and back again losslessly). The stuff that came before Unicode
infected Unicode pretty badly. Unfortunately, if it hadn't, then Unicode
likely would never have been accepted in the first place. So, I'm not sure
that there was ever much chance of Unicode actually being clean, sadly.

Reading up on the Unicode standard and why it is the way that it is has been
one of the most frustrating experiences of my career.

- Jonathan M Davis
Aug 14
prev sibling parent reply Dlighted <test example.com> writes:
On Friday, 14 August 2026 at 06:41:45 UTC, Walter Bright wrote:

 Some good observations!

 If there is a specific code point for é, then the combining 
 character alternative should be illegal.


Don't quote me on this but it (NFC) already seems to be 
convention. I don't know how many input methode there are that 
produce decomposed forms (NFD) with separate accent and how many 
of those decomposed letters are actually in circulation today 
outside documents explicitely talking about them. I wish I'd see 
some hard data on this.
Exceptions often named on the internet is Apple filesystem 
metadata and filenames and copy pasting from it using Mac Finder 
file manager. In theory anything can make them but in practice 
little does.
However if I copy paste an ö I see in the wild anywhere else, 
it's almost certainly precomposed (NFC).

I pasted some German Wikipedia articles in this converter to 
check I'm not talking bullshit and it told me they were already 
in NFC.
https://onlineminitools.com/normalize-unicode-text
Aug 14
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 14/08/2026 8:09 PM, Dlighted wrote:

 On Friday, 14 August 2026 at 06:41:45 UTC, Walter Bright wrote:

 Some good observations!

 If there is a specific code point for é, then the combining character 
 alternative should be illegal.


 
 Don't quote me on this but it (NFC) already seems to be convention. I 
 don't know how many input methode there are that produce decomposed 
 forms (NFD) with separate accent and how many of those decomposed 
 letters are actually in circulation today outside documents explicitely 
 talking about them. I wish I'd see some hard data on this.
 Exceptions often named on the internet is Apple filesystem metadata and 
 filenames and copy pasting from it using Mac Finder file manager. In 
 theory anything can make them but in practice little does.
 However if I copy paste an ö I see in the wild anywhere else, it's 
 almost certainly precomposed (NFC).
 
 I pasted some German Wikipedia articles in this converter to check I'm 
 not talking bullshit and it told me they were already in NFC.
 https://onlineminitools.com/normalize-unicode-text


Strictly speaking html should be NFC. 
https://www.w3.org/TR/charmod-norm/#unicodeNormalization
Aug 14
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, August 13, 2026 7:17:03 PM Mountain Daylight Time H. S. Teoh via
Digitalmars-d wrote:

 tl;dr: the issues involved are much more complex than what an English
 speaker may imagine.  Unicode isn't without its silly points and
 arguably backwards decisions, but given the complexity of the issue,
 it's doing not too bad.


Yeah...

There are definitely a number of bad decisions in the Unicode standard, and
some of the idiocy comes from a variety of external requirements that make
things _way_ messier than they'd be if you could design Unicode in a vacuum
(e.g. requiring that it be possible to convert from a different character
set to Unicode and back again with the result being identical forces Unicode
to include graphemes with the quirks of a bunch of different character sets
whose design philosophies go against what Unicode is trying to do, which is
part of why we have multiple ways to represent the same grapheme).

However, a _lot_ of the stupidy of Unicode comes from the insanity of
natural languages. English has its problems, but when it comes to encoding
characters, it's downright trivial in comparison to a number of other
languages and writing systems (e.g. Cyrillic and Arabic as you mentioned).
Unicode has a number of goals along the lines of what Walter is looking for
which got completely screwed over when dealing with real languages (e.g.
characters being treated differently depending on where in the word or
sentence they are). Even correctly determining whether code points are part
of a single grapheme or not can actually be language-dependent! It's
absolutely insane.

So, on the whole, I'd say that the main problems with Unicode generally fall
into two categories:

1. Natural languages are messy, making handling them in a way that's sane
for a computer essentially impossible. It may be possible to do a better job
than Unicode has, but to a degree, the problem simply cannot be solved in a
sane fashion.

2. Requirements that were placed on Unicode in order to be compatible with
older encodings make following even the sane rules which would otherwise
have been followable impossible to follow.

Both of those mean that pretty much every rule that Unicode has for itself
gets broken somewhere in order to work in some special case. I honestly
question whether there's a single piece of software on the planet which
fully handles Unicode correctly. It's just too much of a disgusting mess.
It's still arguably better than what we had before, but it's by no means
good.

- Jonathan M Davis
Aug 14
next sibling parent reply Dlighted <test example.com> writes:
On Friday, 14 August 2026 at 07:54:51 UTC, Jonathan M Davis wrote:

 However, a _lot_ of the stupidy of Unicode comes from the 
 insanity of natural languages. English has its problems, but 
 when it comes to encoding characters, it's downright trivial in 
 comparison to a number of other languages and writing systems 
 (e.g. Cyrillic and Arabic as you mentioned).


Cyrillic has identical complexity as the Latin alphabet. Not any 
more or less.
Arabic and Hebrew on the other hand uniquely stand out in 
complexity among all languages in the world by constantly 
demanding bidirectional text support in everything.
Aug 14
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Aug 14, 2026 at 08:27:43AM +0000, Dlighted via Digitalmars-d wrote:

 On Friday, 14 August 2026 at 07:54:51 UTC, Jonathan M Davis wrote:

 However, a _lot_ of the stupidy of Unicode comes from the insanity
 of natural languages. English has its problems, but when it comes to
 encoding characters, it's downright trivial in comparison to a
 number of other languages and writing systems (e.g. Cyrillic and
 Arabic as you mentioned).


 Cyrillic has identical complexity as the Latin alphabet. Not any more
 or less.
 Arabic and Hebrew on the other hand uniquely stand out in complexity
 among all languages in the world by constantly demanding bidirectional
 text support in everything.


You forgot Chinese.  Each character is composed of any number of
radicals, including other characters, in completely arbitrary 2D layouts
with completely arbitrary distortions and simplifications to shoehorn
everything into a square block. (Look up "Biangbiang noodles" in
Wikipedia sometime, for laughs.)  Unicode literally threw up its hands
and embraced the combinatorial explosion instead of trying to reason
with it. :-D

Natural languages are totally insane.  For all its warts and
infelicities Unicode is doing not too bad of a job keeping things
*relatively* internationalizable.  At least, it's doing a LOT better
than the bad ole days of per-language codepages, where everybody and his
mother's dog could invent their own codepage, and text-processing
software broke in spectacular ways because you never knew which codepage
a string snippet belonged to.  It was the wild, wild west.  That Unicode
managed to emerge from that mess and unify everything is no mean feat,
warts and wrong decisions notwithstanding.


T

-- 
No! I'm not in denial!
Aug 14
parent reply Dlighted <test example.com> writes:
On Friday, 14 August 2026 at 14:56:06 UTC, H. S. Teoh wrote:

 You forgot Chinese.  Each character is composed of any number 
 of radicals, including other characters, in completely 
 arbitrary 2D layouts with completely arbitrary distortions and 
 simplifications to shoehorn everything into a square block. 
 (Look up "Biangbiang noodles" in Wikipedia sometime, for 
 laughs.)  Unicode literally threw up its hands and embraced the 
 combinatorial explosion instead of trying to reason with it. :-D


To humans but to the computer chinese is also simple. It's just a 
bunch of squares lined up in a row. Even easier than Latin. Just 
more glyphs. They write left to right on computers today and 
don't need bidi.
Aug 14
parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Aug 14, 2026 at 05:48:02PM +0000, Dlighted via Digitalmars-d wrote:

 On Friday, 14 August 2026 at 14:56:06 UTC, H. S. Teoh wrote:

 You forgot Chinese.  Each character is composed of any number of
 radicals, including other characters, in completely arbitrary 2D
 layouts with completely arbitrary distortions and simplifications to
 shoehorn everything into a square block. (Look up "Biangbiang
 noodles" in Wikipedia sometime, for laughs.)  Unicode literally
 threw up its hands and embraced the combinatorial explosion instead
 of trying to reason with it. :-D


 To humans but to the computer chinese is also simple. It's just a
 bunch of squares lined up in a row.


That's because Unicode gave up trying to figure it out, and sacrificed a
huge block of codepoints (> 20,000 codepoints) to cover the
combinatorial explosion of radicals, thus giving you a 1-to-1
correspondence for the most commonly-used combinations.

Unfortunately, that belies the actual nature of the writing -- those
radicals inside characters can behave like independent entities, and
combine with each other in very complex 2D ways (including such lovely
things like wrapping around other radicals, curling around them, nesting
one within another, etc).  Fortunately, we no longer live in the age of
those old insane Chinese scribes who invented this crazy system -- most
radical combinations have been standardized and so Unicode could get
away with just spending lots of codepoints for each combination.

(And the proof that this is only a compromise on Unicode's part is the
various CJK supplemental blocks that have since been allocated.  Because
the combinatorial explosion is theoretically infinite, you see -- and
over time new combinations are discovered to be in use, so Unicode has
to add them to supplemental blocks.  We're lucky that most combinations
have largely fallen out of use.  And that modern-day Chinese no longer
have the same obsession as the old scribes in creating new combinations
on the fly.  Otherwise CJK codepoints would easily take over the entire
codepoint space.  :-D  Or worse, you'd have multi-codepoint
diacritic-style 2D compositions for Chinese characters... the people
designing text rendering engines would faint trying to implement such a
monstrosity.)


T

-- 
A bend in the road is not the end of the road unless you fail to make the turn.
-- Brian White
Aug 14
parent reply Kagamin <spam here.lot> writes:
On Friday, 14 August 2026 at 19:12:39 UTC, H. S. Teoh wrote:

 :-D  Or worse, you'd have multi-codepoint diacritic-style 2D 
 compositions for Chinese characters... the people designing 
 text rendering engines would faint trying to implement such a 
 monstrosity.)


Can't font customize rendering for different combinations, how 
it's done for ligatures? The problem I see maybe not all text 
rendering engines support this. If the font doesn't assign a 
ligature, it will become a bunch of radicals, hopefully it will 
still be understandable, which can be better than unknown 
character displayed today.
Aug 15
parent reply Dlighted <test example.com> writes:
On Saturday, 15 August 2026 at 18:46:26 UTC, Kagamin wrote:

 Can't font customize rendering for different combinations, how 
 it's done for ligatures? The problem I see maybe not all text 
 rendering engines support this. If the font doesn't assign a 
 ligature, it will become a bunch of radicals, hopefully it will 
 still be understandable, which can be better than unknown 
 character displayed today.


Yes, fonts can reuse glyphs as part of other glyphs.
He's suggesting that it could be encoded in a more complex way 
through combinations, so that the rendering engine dynamically 
composes the glyphs from smaller things without font creators 
implementing anything manually or code points being used 
specifically for one combination.
However that would add a lot of complexity to font rendering and 
in reality anything that has to do with language is just as much 
arbitrary as it follows rules. We just prefer to look at the 
patterns.
Aug 15
parent Kagamin <spam here.lot> writes:
On Saturday, 15 August 2026 at 19:09:42 UTC, Dlighted wrote:

 However that would add a lot of complexity to font rendering 
 and in reality anything that has to do with language is just as 
 much arbitrary as it follows rules. We just prefer to look at 
 the patterns.


I mean to render them as ligatures. The rendering engine doesn't 
create ligatures, but somehow finds them in the font.
Aug 17
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
There's no excuse for adding emojis to Unicode.
Aug 16
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 17/08/2026 1:36 PM, Walter Bright wrote:

 There's no excuse for adding emojis to Unicode.


https://www.youtube.com/watch?v=DlwXYY9Ncu4

[:smile:]

We are not the target audience for this.

And yes I recognize its not a nice set of features, but as a concept it 
is a fundamental part of internet based communication.
Aug 16
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 8/12/26 14:30, Dlighted wrote:

 On Wednesday, 12 August 2026 at 11:52:44 UTC, Timon Gehr wrote:

 Instead of discussing what is interesting about the language, a good 
 chunk of the comments are just low-effort anti-Unicode-syntax posts. 
 It's not considered sufficiently "normal" and people like writing 
 snarky comments, so you always get a lot of spam whenever you are 
 trying to share something.


 
 Read this post and reflect.
 ...


This is entirely redundant (how else do you think I engage with the 
forums) and also somewhat disrespectful.

I was answering your only question, hoping that the insight might be 
helpful. Recall that you wrote:

 The only question I have is why we haven't done it yet.


(This is responding to what you mean assuming I am reading your mind 
correctly, feel free to clarify by simply spelling it out.)


 I think `∧` and `∨` I think are not great choices for `&&` and `||`, 
 because `&&` and `||` are short-circuiting operators. Then `and` and 
 `or` don't match either, for the same reason (though I guess Python 
 does this anyway). One alternative I have seen that actually makes 
 sense is `and then` and `or else`, but it's a bit verbose.


 
 There's a long list of languages that use "and" and "or".


There is an even longer list of languages without Unicode operator 
symbols. ;)


 Pascal, Ada, Fortran, Lua, Python, PHP, R and probably others.
 Ruby and Perl have them but with different precedence.
 ...


Anyway, let's take a cursory glance at your list:

Pascal: Inconsistent, there are literally compiler directives to make it 
evaluate both operands in Free Pascal.
https://www.freepascal.org/docs-html/prog/progsu4.html

Ada: `and` and `or` evaluate both operands, `and then` and `or else` are 
the short-circuiting versions.
https://docs.adacore.com/live/wave/arm12/html/arm12/arm12-4-5-1.html

Fortran: AFAIU it's actually spelled .AND. and .OR.
Also, the Fortran standard actually does not require them to 
short-circuit at all:
https://gcc.gnu.org/onlinedocs/gcc-15.3.0/gfortran/Evaluation-of-logical-expressions.html

Furthermore, they are even allowed to short-circuit backwards.

Lua: Yes.

PHP: Yes. However, please do understand that citing PHP as a precedent 
is the opposite of persuasive.

R: No? I think it is spelled `&&`/`||` in R.
https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Conditional-execution_003a-if-statements

Ruby/Perl: Yes. (They have both `&&`/`||` and `and`/`or`, but `and`/`or` 
have much lower precedence.)


 OpenAI ignores 𝔹 when reading HTML pages, so GPT is actually not able 
 to read documentation if it is written with Unicode.


 
 Gemini says its a problem with the letter you chose and not with Unicode 
 in general:
 ...


Gemini is hallucinating. I guess this was not the flagship model.

This was quite obviously neither a problem with the letter I chose nor 
with Unicode in general, nor with tokenization. This was strictly an 
issue with the agentic harness that ChatGPT was using for online chats. 
However, because ChatGPT is more popular than Unicode syntax, it did 
become an issue for us, and we could have avoided it by not adopting 
Unicode in our language.

Anyway, I had reported the bug to OpenAI, and testing it again now, it 
appears they actually did recently fix the issue (after having remained 
open for a very long time). However, had we not used Unicode, we would 
never have had this problem in the first place.

It's not that this is my personal preference, it's just that there are 
external factors. Personally I much prefer formatting my code with 
Unicode symbols. (Much to the chagrin of some of the people I work with.)

It is an uphill battle. Another anecdote: see how my co-author chose to 
interface with my Unicode identifiers in D:
https://github.com/silq-lang/silq/blob/6e0c42bf78a5c924912d5adf3c471bd988d5320b/hqir.d#L34-L60

It would be easy to dismiss his instinct as entirely irrational, but I 
think it is not the case. There is just a lot of software out there that 
won't deal with Unicode correctly, and there is often an added 
complexity cost with a nontrivial additional bug risk when you do want 
to handle it correctly.

The ChatGPT harness was just another example of poorly written software 
that I don't control and that can't deal with Unicode correctly, and 
this is the sense in which the anecdote is meaningful in this context.


I also used to use a big file with rules like these in a LaTeX header:

\usepackage[utf8]{inputenc}
...
\DeclareUnicodeCharacter{2115}{\mathbb N}
\DeclareUnicodeCharacter{2124}{\mathbb Z}
\DeclareUnicodeCharacter{211A}{\mathbb Q}
\DeclareUnicodeCharacter{211D}{\mathbb R}
\DeclareUnicodeCharacter{2102}{\mathbb C}
\DeclareUnicodeCharacter{2119}{\mathbb P}
...
\DeclareUnicodeCharacter{2080}{_0}
\DeclareUnicodeCharacter{2081}{_1}
\DeclareUnicodeCharacter{2082}{_2}
\DeclareUnicodeCharacter{2083}{_3}
\DeclareUnicodeCharacter{2084}{_4}
\DeclareUnicodeCharacter{2085}{_5}
\DeclareUnicodeCharacter{2086}{_6}
\DeclareUnicodeCharacter{2087}{_7}
\DeclareUnicodeCharacter{2088}{_8}
\DeclareUnicodeCharacter{2089}{_9}
\DeclareUnicodeCharacter{208A}{_{+}}
\DeclareUnicodeCharacter{208B}{_{-}}
\DeclareUnicodeCharacter{208C}{_{=}}
...

This allowed me to write more readable LaTeX files with content like:

ℕ₀ (turns into \mathbb{N}_0)

Why did I have to abandon it? My boss was using a crappy editor on 
Windows and it did not support utf-8 properly, so he was unable to 
actually see, let alone edit, my formulas.

And this was all while pretending that there are no issues whatsoever 
with Unicode itself. (It does not even have subscripted versions of all 
small Latin letters; very frustrating, but this is because it was not 
actually even made to be useful for typesetting formulas in a plaintext 
file.)


 Modern AI models (like ChatGPT) can read both symbols, but they process 
 them through a tokenizer that breaks text down into bytes or smaller 
 subword chunks.
 - ∨ (Logical OR / Down Tack): This is a standard mathematical/logical 
 symbol present in Unicode (U+2228). Because it is frequently used in 
 math, logic, and programming datasets online, tokenizers often handle it 
 gracefully, sometimes mapping it directly or splitting it efficiently.
 - 𝔹 (Blackboard Bold Capital B): This belongs to the Unicode 
 Mathematical Alphanumeric Symbols block (U+1D539). These symbols are 
 notorious for causing tokenization inefficiencies. Because they look 
 visually similar to a standard Latin "B" to a human, but have completely 
 different underlying byte codes, tokenizers often fragment them into 
 multiple awkward tokens.


As I indicated above, this diagnosis is incorrect, it literally was not 
seeing the tokens _only when grabbing the HTML from an online search in 
the in-browser chat_. There was no problem at all when pasting the 
symbol directly into e.g. the chat window (or when using the API with a 
custom harness). There is no issue with tokenization. The model just got 
really adamant that the syntax with the token stripped is correct, 
because it assumed that it retrieved it that way from the authoritative 
source.
Aug 12
parent Dlighted <test example.com> writes:
Okay, I was unnecessarily rude. Sorry.

On Wednesday, 12 August 2026 at 22:43:01 UTC, Timon Gehr wrote:


 ℕ₀ (turns into \mathbb{N}_0)



 Why did I have to abandon it? My boss was using a crappy editor 
 on Windows and it did not support utf-8 properly, so he was 
 unable to actually see, let alone edit, my formulas.


I guess this is what it all comes back to. It's like the UTF-8 
BOM mark and notepad.exe.
Aug 12
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/12/2026 4:52 AM, Timon Gehr wrote:

 Not a reason not to do it, just a reason why I predict it will not happen 
 anytime soon in D.


I suppose one could adapt their code editor to display != as ≠, but then
aligned 
columns may not work well.
Aug 12
next sibling parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 13/08/2026 7:48 AM, Walter Bright wrote:

 On 8/12/2026 4:52 AM, Timon Gehr wrote:

 Not a reason not to do it, just a reason why I predict it will not 
 happen anytime soon in D.


 
 I suppose one could adapt their code editor to display != as ≠, but then 
 aligned columns may not work well.


You don't have to.

Change your font, they already offer this (I've used a popular one do that).
Aug 12
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/12/2026 1:31 PM, Richard (Rikki) Andrew Cattermole wrote:

 On 13/08/2026 7:48 AM, Walter Bright wrote:

 On 8/12/2026 4:52 AM, Timon Gehr wrote:

 Not a reason not to do it, just a reason why I predict it will not happen 
 anytime soon in D.


 I suppose one could adapt their code editor to display != as ≠, but then 
 aligned columns may not work well.


 
 You don't have to.
 
 Change your font, they already offer this (I've used a popular one do that).



So they stretch out ≠ to be two characters wide? Code editors tend to use 
monospace fonts.
Aug 12
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 13/08/2026 12:03 PM, Walter Bright wrote:

 On 8/12/2026 1:31 PM, Richard (Rikki) Andrew Cattermole wrote:

 On 13/08/2026 7:48 AM, Walter Bright wrote:

 On 8/12/2026 4:52 AM, Timon Gehr wrote:

 Not a reason not to do it, just a reason why I predict it will not 
 happen anytime soon in D.


 I suppose one could adapt their code editor to display != as ≠, but 
 then aligned columns may not work well.


 You don't have to.

 Change your font, they already offer this (I've used a popular one do 
 that).


 
 
 So they stretch out ≠ to be two characters wide? Code editors tend to 
 use monospace fonts.


JetBrains Mono, a MonoSpace font does make it two characters.

https://www.jetbrains.com/lp/mono/#ligatures
Aug 12
parent Mindy Batek (0xEAB) <desisma heidel.beer> writes:
On Thursday, 13 August 2026 at 00:07:16 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 JetBrains Mono


Highly recommended font family!
While its look and feel may appear a bit unusual at first, once 
one gets used to it, one will notice that it is a really 
well-designed font for engineering purposes.
Aug 12
prev sibling parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Wed, Aug 12, 2026 at 12:48:02PM -0700, Walter Bright via Digitalmars-d wrote:

 On 8/12/2026 4:52 AM, Timon Gehr wrote:

 Not a reason not to do it, just a reason why I predict it will not
 happen anytime soon in D.


 
 I suppose one could adapt their code editor to display != as ≠, but
 then aligned columns may not work well.


Coding fonts with ligatures for those things generally space the
combined glyph as double-width, so it doesn't throw off the alignment.


T

-- 
Why did the mathematician reinvent the square wheel?  Because he wanted to
drive smoothly over an inverted catenary road.
Aug 12
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/11/2026 10:34 AM, Dlighted wrote:

 Until C23:

 ??/    \\
 ??'    ^
 ??(    [
 ??)    ]
 ??!    |
 ??<    {
 ??>    }
 ??-    ~
 Since C 95:
 <:    [
 :>    ]
 <%    {
 %>    }

 https://en.wikipedia.org/wiki/Digraphs_and_trigraphs_(programming)


In my 40-some (cough cough) years of programming, I have never ever seen those 
trigraphs and digraphs appear anywhere other than a test suite. They were a 
complete failure.

I agree that UTF-16 and UCS-32 are essentially dead. I know that Windows runs
on 
UTF-16, but I still recommend using UTF-8 and then convert only when reading 
to/from a Windows API.

D was designed to be agnostic about UTF-8/16/32, as that was the sign of those 
times. But I asked Adam Wilson to abandon UTF-16/32 for the new Phobos, other 
than the adapter ranges.

As for ≥ ≤ ≠, they certainly look nice. But I don't really want to look
at a 
cheat card to remember how to type them in.
Aug 12
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 8/12/26 21:37, Walter Bright wrote:

 On 8/11/2026 10:34 AM, Dlighted wrote:

 Until C23:

 ??/    \\
 ??'    ^
 ??(    [
 ??)    ]
 ??!    |
 ??<    {
 ??>    }
 ??-    ~
 Since C 95:
 <:    [
 :>    ]
 <%    {
 %>    }

 https://en.wikipedia.org/wiki/Digraphs_and_trigraphs_(programming)


 
 In my 40-some (cough cough) years of programming, I have never ever seen 
 those trigraphs and digraphs appear anywhere other than a test suite. 
 They were a complete failure.
 ...


I think they were worth it just for helping facilitate this wholesome 
interaction:
https://www.quora.com/Are-there-really-programmers-with-computer-science-degrees-who-cannot-pass-the-FizzBuzz-test/answer/Gayle-Laakmann-McDowell?comment_id=9146415&comment_type=2

Anyway, OPs point is that e.g. `<=` is arguably just a digraph for `≤`. 
Except that `≤` is not even legal, and then they argue that that is the 
problem, because it requires people to keep using the digraph.


 I agree that UTF-16 and UCS-32 are essentially dead. I know that Windows 
 runs on UTF-16, but I still recommend using UTF-8 and then convert only 
 when reading to/from a Windows API.
 
 D was designed to be agnostic about UTF-8/16/32, as that was the sign of 
 those times. But I asked Adam Wilson to abandon UTF-16/32 for the new 
 Phobos, other than the adapter ranges.
 
 As for ≥ ≤ ≠, they certainly look nice. But I don't really want to look 
 at a cheat card to remember how to type them in.
 


For me it is literally just

\>= (or \le)

\<= (or \ge)

\neq (I am used to this anyway, but could be reconfigured to allow \!= 
as well easily).


I think the bigger issue with `≠` that it's clearly the negation of `=` 
and not `==`. Arguably `==` is the "digraph" for the "Unicode variant" 
`=`, but `=` is already parsed as assignment.

Therefore code would likely have to mix `≠` and `==`, or there would 
need to be a way to overload `=` to mean equality comparison sometimes 
and assignment other times. I think this would be somewhat annoying to 
try and pull off in D.
Aug 12
parent Walter Bright <newshound2 digitalmars.com> writes:
At least we don't have to deal with RADIX-50!
Aug 12
prev sibling parent reply Kagamin <spam here.lot> writes:
On Tuesday, 11 August 2026 at 17:34:00 UTC, Dlighted wrote:

 On Wednesday, 31 May 2023 at 08:47:04 UTC, Walter Bright wrote:

 I use putty a lot to access computers remotely in text mode. 
 With some experimentation, some Unicode characters are 
 rendered, but some aren't, like the 69 quotes. Maybe the 
 programming world isn't quite ready for them yet.


 He probably didn't use Cygwin.


It's probably because Lucida Console doesn't have reverse 9 
quotes.
“this will work”
‟this won't work”
And many fonts don't have `∨` `∧`.
Aug 13
parent Dlighted <test example.com> writes:
On Thursday, 13 August 2026 at 17:12:35 UTC, Kagamin wrote:

 It's probably because Lucida Console doesn't have reverse 9 
 quotes.
 “this will work”
 ‟this won't work”
 And many fonts don't have `∨` `∧`.


It's probably a Microsoft exclusive problem with some ancient 90s 
component, they can't touch anymore.
Any normal terminal has font fallback just like any other program.
Most font faces only contain a limited amount of coverage.
Aug 13