www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Trying to use Mir ion, its a perfect example of the failure of D's

reply A moo person <moo_mail fake.com> writes:
This is a semi rant born from frustration. Currently I'm trying 
to figure out Mir Ion so I can serialize some data, and oh boy is 
it frustrating.

The entire library is annotated out the wazoo with  safe and 
 nogc and pure and inout. Its a serialization library so as you 
might imagine it's doing reflection on types, calling member 
functions to try and serialize stuff, things like that. I have 
spent hours now adding attributes and casting attributes away. 
Writing  trusted wrapper types for std.algo ranges. Trying to 
decipher the vomit that dmd gives when one the types is not quite 
right. All I want to do is serialize a list of structs from a 
range and be able to deserialize them back but I can't do that 
because I still have not gotten it working.

It's actually maddening, I have spent 2 days on this. What value 
are all these attributes adding? Because it's a useability 
nightmare. Every time I encounter a D library that tries to be 
clever with these annotations, it's just absolutely made things 
more complex and less useable. Same story every time I have ever 
tried to use them in my own code.
Jan 18 2023
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
You can pretty much not use any attributes if you use none (including not using 
 safe).

We can help if you can present specific cases.
Jan 18 2023
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 19/01/2023 3:10 PM, Walter Bright wrote:

 You can pretty much not use any attributes if you use none (including 
 not using  safe).
 
 We can help if you can present specific cases.


If you call code that is annotated, that then calls your code this is 
not true.

Callbacks, hooks on types as used with serialization ext. don't really 
work here currently.

There is a reason why I've long wanted  localnogc, but yeah... pure and 
 safe also need local versions too.
Jan 18 2023
prev sibling next sibling parent reply A moo person <moo_mail fake.com> writes:
On Thursday, 19 January 2023 at 02:10:14 UTC, Walter Bright wrote:

 You can pretty much not use any attributes if you use none 
 (including not using  safe).

 We can help if you can present specific cases.


In this case you can't leave them unannotated as all the Mir 
functions are annotated and they try to call stuff in the type 
you are trying to serialize.

 safe and const are the one that is the most problematic in my 
case. (I mentioned  nogc before but I listed it by mistake but I 
have ran into similar issues with  nogc libs)

I am not really looking for help with Mir Ion specifically as its 
just some library, honestly at this point I'm probably not going 
to use it because it's too unwieldy and over engineered.

But here is what I have had to do so far to even try to get 
things compiling. I feel that Mir trying to be  safe is forcing 
me to write even less safe code than I would normally just to try 
and get it through the type system. At a certain point you just 
start casting and annotating randomly just to get it to compile. 
These sorts of workarounds seem to be what always happens when I 
encounter these annotations. In my own code I avoid them like the 
plague if I can't get by without them.


struct PageSerial {
	// this is just some GUI element
	// the contents of this gui element is what I am trying to 
ser/deser
	private Grid g;

	 serdeKeys("blocks")
	 serdeIgnoreIn
	// serdeLikeList
	auto blocks_out()  property  trusted const {
		// Had to put  trusted because the mir function to ser/deser is 
 safe
		// Had to put const because the mir function decides to cast my 
type to const when working with it?? no idea why

		import std.algorithm : filter, map;
		import std.range : ElementType;
		// have to cast away the const
		auto t = cast(PageSerial*)(&this);
		auto r =
			(t.g).contents.children()[]
			.filter!(c => (cast(GridBlock)c !is null))()
			.map!(c => GridBlockSerial(cast(GridBlock)c))();
		static assert(is(ElementType!(typeof(r)) == GridBlockSerial));

		// trustRange wraps all the range functions with  trusted
		// had to do this because somehow the map type's empty 
evaluated to  system
		// but the range is iterated in the  safe serializeValue so i 
have to force it
		return trustRange(r);
	}


	// this one doesnt even work so not even sure if its set up 
right for Mir
	// at least it compiles but trying to read the data back in 
causes exceptions
	 serdeKeys("blocks")
	 serdeIgnoreOut
	 serdeLikeList
	 serdeProxy!GridBlockSerial
	auto blocks_in()  property  trusted const  {
		struct R{
			private Grid x;
			void put(GridBlockSerial b){ x.addElement(b.makeElement()); }
		}
		return R(cast(Grid)g);
	}
}

GridBlockSerial is another serialized type but not as complex. 
The PageSerial type gets passed into Mir's serializeJson and 
deserializeJson. Each of which pass them down into some lower 
level templates (eg Mir's serializeValue) which is annotated 
 safe.

I try to go in and read Mir's code but its just a spaghetti mess 
of templates, its very hard to reason about and understand what 
is going on. It's like a ton of engineering effort was spent in 
an attempt to make the library "safer" in some type system sense, 
but the net result is a library that is borderline unusably 
cumbersome and actually incentivizes me to write less safe code.

If the argument is that the library should have been designed to 
infer the annotations, then why even have the annotations in the 
language at all if the best practice is to infer them.
Jan 18 2023
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Thursday, 19 January 2023 at 04:06:27 UTC, A moo person wrote:

 If the argument is that the library should have been designed 
 to infer the annotations, then why even have the annotations in 
 the language at all if the best practice is to infer them.


In principle, the annotations are there mainly for (a) cases 
where attributes can't be inferred (e.g., extern functions), and 
(b) cases where you want to override attribute inference (e.g., 
 trusted).

Unfortunately, the compiler does not do as much inference as it 
could (e.g., it does not do inference for most ordinary 
non-template functions), which leads some D programmers to adopt 
the habit of sprinkling 'pure  safe nothrow  nogc' (or similar) 
all over their code as a precautionary measure. It seems like 
this is more or less what the Mir developers have done.

This is, ultimately, a usability problem with the 
language--sometimes it's best practice to write out the 
attributes by hand, other times it's a code smell, and if a 
library author doesn't know which is which, their users will 
suffer for it.

In the long run, I hope that we can address this by allowing the 
compiler to infer attributes for more functions; in the short 
run, the best we can do is try to fix the affected libraries.
Jan 18 2023
prev sibling next sibling parent Kagamin <spam here.lot> writes:
On Thursday, 19 January 2023 at 04:06:27 UTC, A moo person wrote:

 struct PageSerial {
 	// this is just some GUI element
 	// the contents of this gui element is what I am trying to 
 ser/deser
 	private Grid g;

 	 serdeKeys("blocks")
 	 serdeIgnoreIn
 	// serdeLikeList
 	auto blocks_out()  property  trusted const {
 		// Had to put  trusted because the mir function to ser/deser 
 is  safe
 		// Had to put const because the mir function decides to cast 
 my type to const when working with it?? no idea why

 		import std.algorithm : filter, map;
 		import std.range : ElementType;
 		// have to cast away the const
 		auto t = cast(PageSerial*)(&this);
 		auto r =
 			(t.g).contents.children()[]
 			.filter!(c => (cast(GridBlock)c !is null))()
 			.map!(c => GridBlockSerial(cast(GridBlock)c))();
 		static assert(is(ElementType!(typeof(r)) == GridBlockSerial));


Not every type can be serialized, normally you serialize a DTO - 
a type designed for serialization and holds primitive or simple 
data, then it will be compatible with attributes.
Jan 23 2023
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/18/2023 8:06 PM, A moo person wrote:

 If the argument is that the library should have been designed to infer the 
 annotations, then why even have the annotations in the language at all if the 
 best practice is to infer them.


Great question! I'll start by saying generally speaking, templates should not
be 
annotated and should allow inference to do its thing.

As for why one would need annotations at all:

1. the inference isn't perfect, in particular if the templates are recursive.

2. if everything is inferred, there's no way to tell code is  safe or not. This 
would be indistinguishable from having no safety checks at all.

3. use an annotation when a guarantee is desired

4. inference isn't done for conventional functions, because as mentioned before 
that would cause problems with .di files

5. to enforce a desired coding paradigm
Jan 23 2023
parent Hipreme <msnmancini hotmail.com> writes:
On Tuesday, 24 January 2023 at 07:42:32 UTC, Walter Bright wrote:

 On 1/18/2023 8:06 PM, A moo person wrote:

 If the argument is that the library should have been designed 
 to infer the annotations, then why even have the annotations 
 in the language at all if the best practice is to infer them.


 Great question! I'll start by saying generally speaking, 
 templates should not be annotated and should allow inference to 
 do its thing.

 As for why one would need annotations at all:

 1. the inference isn't perfect, in particular if the templates 
 are recursive.

 2. if everything is inferred, there's no way to tell code is 
  safe or not. This would be indistinguishable from having no 
 safety checks at all.

 3. use an annotation when a guarantee is desired

 4. inference isn't done for conventional functions, because as 
 mentioned before that would cause problems with .di files

 5. to enforce a desired coding paradigm


2: If everything is inferred, maybe the person doing the code 
doesn't actually care if it is  safe or not. The problem on  safe 
is how attributes work right now, super shitty to work now with 
it (const included). Most of D code does not care with  safe. 90% 
of my code could be inferred as  safe , but no, I need to 
explicitly say on every function I need to define because when 
creating a  safe function, I would not be able to call it.

4: It would not cause any problems with di files. Inference can 
output annotation to .di files, they can be regenerated whenever 
you wish to distribute.
Jan 24 2023
prev sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Thursday, 19 January 2023 at 02:10:14 UTC, Walter Bright wrote:

 You can pretty much not use any attributes if you use none 
 (including not using  safe).


If a function takes a delegate, there is no way (outside of 
changing the type, e.g., templating on the type of the delegate) 
to indicate conditional attributes.

So if someone in a library wrote

void process(void delegate() userData)  nogc {}

it is going to force userData to be nogc regardless of their own 
desires.
Jan 19 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2023 5:14 AM, Adam D Ruppe wrote:

 So if someone in a library wrote
 
 void process(void delegate() userData)  nogc {}
 
 it is going to force userData to be nogc regardless of their own desires.


If process calls userData, then that is as it should be. If userData is just 
being "passed through" and not called, then the easiest workaround is to cast
it 
to something innocuous, or use a union.
Jan 19 2023
next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 20 January 2023 at 01:45:31 UTC, Walter Bright wrote:

 void process(void delegate() userData)  nogc {}


 If process calls userData, then that is as it should be.


Ideally, you'd be able to say `process` inherits the nogc-ness of 
`userData`.


 userData is just being "passed through" and not called, then 
 the easiest workaround is to cast it to something innocuous, or 
 use a union.


But this is also a major pain. D's attributes are one of the 
biggest misses of the language.
Jan 19 2023
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2023 6:18 PM, Adam D Ruppe wrote:

 Ideally, you'd be able to say `process` inherits the nogc-ness of `userData`.


That's exactly what template attribute inference does.
Jan 19 2023
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/19/23 10:38 PM, Walter Bright wrote:

 On 1/19/2023 6:18 PM, Adam D Ruppe wrote:

 Ideally, you'd be able to say `process` inherits the nogc-ness of 
 `userData`.


 
 That's exactly what template attribute inference does.
 


Not exactly what is being asked (though the quote implies it).

What is being asked is for process to inherit the nogc-ness of *the 
argument passed to userData*.

That is, you know process is calling `userData`, and you want that to 
figure into the inference (this doesn't technically require a template).

What we want is the effective attributes of the function to be the most 
restrictive possible of both the code inside the function, and the 
argument to the function.

The compiler would need a way to specify this for non-inferred 
functions, but it could be inferred for templates and auto functions.

-Steve
Jan 19 2023
next sibling parent reply Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Friday, 20 January 2023 at 05:48:26 UTC, Steven Schveighoffer 
wrote:

 On 1/19/23 10:38 PM, Walter Bright wrote:

 On 1/19/2023 6:18 PM, Adam D Ruppe wrote:

 Ideally, you'd be able to say `process` inherits the 
 nogc-ness of `userData`.


 
 That's exactly what template attribute inference does.
 


 […]

 What we want is the effective attributes of the function to be 
 the most restrictive possible of both the code inside the 
 function, and the argument to the function.

 The compiler would need a way to specify this for non-inferred 
 functions, but it could be inferred for templates and auto 
 functions.


I wrote a DIP (_Attributes for Higher-Order Functions,_ AfHOF) 
for that, but didn’t make it. I tried to be pragmatic and squeeze 
it into the current syntax via doubling-down on a bug in the type 
system (i.e. making it a ‘feature’). Timon Gehr made me aware of 
the bug. Writing the DIP made me aware that higher-order 
functions are a blocker for making e.g. ` safe` the default: If 
you have `void f(void delegate() dg)`, there is no consistent way 
to add ` safe` to `f` and/or `dg` that won’t break properly 
annotated ` system` code.

Mathias Lang has a work-in-progress DIP called _Argument 
dependent attributes_ (AdA). It introduces syntax to specify that 
the attributes of higher-order function (i.e. a function with 
callbacks) depend on the attributes of the callback arguments. 
It’s a half-way solution that only targets parameters of delegate 
or function pointer type. A full solution would include 
parameters of composed types that include delegate or function 
pointer types somewhere: Slices of delegate or function pointer 
type could be considered reasonably common. Attribute variables 
would do that.

* D has runtime value variables, e.g. your good old `x` in `int 
x;`.
* D has type variables, e.g. you good old `T` in `template 
t(T){..}`
* D has symbol variables, e.g. template `alias` parameters.
* D has 1 type constructor variable: `inout`. (D were more 
consistent if it allowed for two or more independent type 
constructor variables, but that’s rarely needed practically.)
* D has no attribute variables.

If D added 1 fixed-name attribute variable per attribute (cf. 
`inout`), they could be spelled ` safety`, `purity`, ` gcness`, 
and `throwiness`.

```d
void callInSequence(
     void delegate()  safety purity  gcness throwiness[] dgs
)  safety purity  gcness throwiness
{
     foreach (dg; dgs) dg();
}
```
When called, the argument is a slice of delegates that are 
` safe` or ` system` – ` safety` is replaced by that; that are 
`pure` or not – purity is replaced by that; that are ` nogc` or 
not – ` gcness` is replaced by that; that may throw or not – 
`throwiness` by `throw` or `nothrow`.

This works the same way as when `inout(int)[] f() inout` is 
called on a mutable, `const` or `immutable` object, its return 
type will be `int[]`, `const(int)[]`, or `immutable(int)[]` after 
replacing `inout` with the call-side type constructor applied to 
the object’s type.
Jan 20 2023
next sibling parent Ruby The Roobster <rubytheroobster yandex.com> writes:
On Friday, 20 January 2023 at 13:08:46 UTC, Quirin Schroll wrote:

 On Friday, 20 January 2023 at 05:48:26 UTC, Steven
 ... [snip]

 If D added 1 fixed-name attribute variable per attribute (cf. 
 `inout`), they could be spelled ` safety`, `purity`, ` gcness`, 
 and `throwiness`.

 ```d
 void callInSequence(
     void delegate()  safety purity  gcness throwiness[] dgs
 )  safety purity  gcness throwiness
 {
     foreach (dg; dgs) dg();
 }
 ```
 When called, the argument is a slice of delegates that are 
 ` safe` or ` system` – ` safety` is replaced by that; that are 
 `pure` or not – purity is replaced by that; that are ` nogc` or 
 not – ` gcness` is replaced by that; that may throw or not – 
 `throwiness` by `throw` or `nothrow`.

 [snip]



I second this, or a less verbose version of this.  One who wishes 
to vary all of the attributes shouldn't have to explicitly write 
``` safety  gcness purity throwiness``` every time they want to 
vary all of the attributes.  Perhaps adding a keyword such as 
```attrib``` to vary all attributes not specifically set is the 
best course of action.  Then we can write:

```d
void callInSequence(void delegate() attrib  safe dgs[])
{
     foreach(dg; dgs)
         dg();
}
```

This will require that all delegates be  safe, but there are no 
other restrictions on their attributes.
Jan 20 2023
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/20/23 8:08 AM, Quirin Schroll wrote:


 
 If D added 1 fixed-name attribute variable per attribute (cf. `inout`), 
 they could be spelled ` safety`, `purity`, ` gcness`, and `throwiness`.
 
 ```d
 void callInSequence(
      void delegate()  safety purity  gcness throwiness[] dgs
 )  safety purity  gcness throwiness
 {
      foreach (dg; dgs) dg();
 }
 ```


My proposal was basically to have a ` called` attribute that indicates 
the function delegate may be called inside the function, meaning the 
attributes of the call should reflect the attributes of the parameter.


 This works the same way as when `inout(int)[] f() inout` is called on a 
 mutable, `const` or `immutable` object, its return type will be `int[]`, 
 `const(int)[]`, or `immutable(int)[]` after replacing `inout` with the 
 call-side type constructor applied to the object’s type.


Yes, inout is exactly the same type of mechanism

-Steve
Jan 20 2023
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 1/20/23 15:53, Steven Schveighoffer wrote:

 On 1/20/23 8:08 AM, Quirin Schroll wrote:
 

 If D added 1 fixed-name attribute variable per attribute (cf. 
 `inout`), they could be spelled ` safety`, `purity`, ` gcness`, and 
 `throwiness`.

 ```d
 void callInSequence(
      void delegate()  safety purity  gcness throwiness[] dgs
 )  safety purity  gcness throwiness
 {
      foreach (dg; dgs) dg();
 }
 ```


 
 My proposal was basically to have a ` called` attribute that indicates 
 the function delegate may be called inside the function, meaning the 
 attributes of the call should reflect the attributes of the parameter.


The issue with that is that it does not generalize/compose. E.g., you 
can't write a non-templated function that composes two delegates while 
preserving attributes.
Jan 21 2023
prev sibling parent reply A moo person <moo_mail fake.com> writes:
On Friday, 20 January 2023 at 14:53:49 UTC, Steven Schveighoffer 
wrote:

 My proposal was basically to have a ` called` attribute that 
 indicates the function delegate may be called inside the 
 function, meaning the attributes of the call should reflect the 
 attributes of the parameter.


I really hate that the proposal to the problem of too many 
attributes in the language is "hey lets add another attribute"... 
like really?
Jan 22 2023
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/22/23 3:43 AM, A moo person wrote:

 On Friday, 20 January 2023 at 14:53:49 UTC, Steven Schveighoffer wrote:

 My proposal was basically to have a ` called` attribute that indicates 
 the function delegate may be called inside the function, meaning the 
 attributes of the call should reflect the attributes of the parameter.


 
 I really hate that the proposal to the problem of too many attributes in 
 the language is "hey lets add another attribute"... like really?


The problem of "too many attributes" is not what is causing your project 
to fail to build. It is a lack of expressiveness in the language that 
forces mir ion to make a choice -- enforce the most restrictive 
attributes for templates and delegates, or remove all of them and force 
the user not to care about purity/safety/gc.

It shouldn't have to make that choice. A library should only put 
restrictions on its users that it requires. Otherwise, the user should 
be free to pick unsafety, safety, gc or nogc, etc.

Ideally, D should be able to do inference of everything, but separate 
compilation makes this impossible. So we have attributes to designate 
what the declaration has.

-Steve
Jan 22 2023
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
In general, yes, moving towards attribute inference for ordinary functions 
rather than just templates is where we want to go.

The reason it isn't done currently is because it makes .di declarations 
incompatible with the .d definitions.
Jan 20 2023
next sibling parent reply Hipreme <msnmancini hotmail.com> writes:
On Friday, 20 January 2023 at 19:18:23 UTC, Walter Bright wrote:

 In general, yes, moving towards attribute inference for 
 ordinary functions rather than just templates is where we want 
 to go.

 The reason it isn't done currently is because it makes .di 
 declarations incompatible with the .d definitions.


.di needs to become a lot better before impacting some kind of 
decision:

1. It needs a specific coding style to improve its output, such 
as writing your import inside the functions itself, though it 
should only expose what is really needed. Read that as classes, 
interfaces, types. Unless the user don't public import constants 
or functions, there's no reason to it be included in output, 
unless present in a template.

2. It could be used to do CTFE caching. As far as I understand, 
it is not possible to do that right now. But it would make a huge 
gain to the D community and would make it a lot more appealing.

3. The inference could be passed to the .di files at generation 
time. That would mean:
```d
int add(int a, int b){return a+b;}
```
Could easily be output to the .di as:
```d
 safe pure nothrow  nogc int add(int a, int b);
```

This could even save time in the .di usage as it would save the 
work of needing to do any kind of inference in library code.

This is things I could think out of the box. I would say that we 
are lacking tooling support for .di too. I don't think dub 
integrates nicely with .di files, which were one of the reasons 
I'm not using it.

Plus, as Adam said many times, the .di generation is pretty raw 
right now, so, there's a lot of handwork to maintain one of 
those, I believe that solving those items could be improved.


Plus, we could create an attribute specifically used for 
templates which are meant to only be instantiated once, such as 
` noexport`, which would mean that in the .di output, it would 
not need to keep the template itself. This is useful for not 
exporting code that was only done for inline ctfe'd.
Jan 20 2023
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 21/01/2023 10:23 AM, Hipreme wrote:

 This is things I could think out of the box. I would say that we are 
 lacking tooling support for .di too. I don't think dub integrates nicely 
 with .di files, which were one of the reasons I'm not using it.


Dub doesn't abstract the .di generator. However it is very easy to pass 
in the flags in to make it do what you want. This is not a limiting factor.

Fact is, the .di generator doesn't produce usable D code, nor does it 
produce it in a way that -I can use.

It is useless in its current state and is a major issue that will need 
to be solved once DLL's work properly in dmd.
Jan 20 2023
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 20 January 2023 at 21:33:11 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 Fact is, the .di generator doesn't produce usable D code, nor 
 does it produce it in a way that -I can use.

 It is useless in its current state and is a major issue that 
 will need to be solved once DLL's work properly in dmd.


It doesn't even work for static libraries? That's concerning, if 
true.
Jan 20 2023
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 21/01/2023 11:03 AM, Paul Backus wrote:

 On Friday, 20 January 2023 at 21:33:11 UTC, Richard (Rikki) Andrew 
 Cattermole wrote:

 Fact is, the .di generator doesn't produce usable D code, nor does it 
 produce it in a way that -I can use.

 It is useless in its current state and is a major issue that will need 
 to be solved once DLL's work properly in dmd.


 
 It doesn't even work for static libraries? That's concerning, if true.


The only thing that differentiates static from shared libraries is 
export. That was never an issue when I last tried it.
Jan 20 2023
prev sibling parent A moo person <moo_mail fake.com> writes:
On Friday, 20 January 2023 at 19:18:23 UTC, Walter Bright wrote:

 In general, yes, moving towards attribute inference for 
 ordinary functions rather than just templates is where we want 
 to go.

 The reason it isn't done currently is because it makes .di 
 declarations incompatible with the .d definitions.


Do people even use .di's?
Jan 20 2023
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Consider:

  void delegate() dg;

  // no error
  void process(void delegate() userData)  nogc { dg = userData; }

  // Error: ` nogc` function `process2` cannot call non- nogc delegate
`userData`
  void process2(void delegate() userData)  nogc { userData(); }

If process() is made into a template, it will "inherit"  nogc based on whether 
the delegate is called or not.
Jan 19 2023
prev sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Jan 19, 2023 at 05:45:31PM -0800, Walter Bright via Digitalmars-d wrote:

 On 1/19/2023 5:14 AM, Adam D Ruppe wrote:

 So if someone in a library wrote
 
 void process(void delegate() userData)  nogc {}
 
 it is going to force userData to be nogc regardless of their own
 desires.


 
 If process calls userData, then that is as it should be. If userData
 is just being "passed through" and not called, then the easiest
 workaround is to cast it to something innocuous, or use a union.


This is a flaw in the language: there is no way to express that
`process`'s attributes inherit from the passed-in delegate.  This has
been a pain point for years.

The basic idea is that we want to ensure that the body of `process`,
aside from the call(s) to userData, is pure/ nogc/ safe/etc., but we
don't want to prevent the caller from passing an impure/GC/ system/etc.
delegate (because it doesn't matter to the internal workings of
`process`).  This makes `process` usable from both GC and  nogc code: if
the caller is  nogc, then it must pass in a  nogc delegate.  But if the
caller is GC allocating, then an allocating delegate is permitted, since
the caller is already non- nogc so a non- nogc delegate will not change
anything.

If we don't do it this way, there will always be a case that doesn't
work.  If we make the delegate parameter  nogc, then non- nogc code will
not be able to pass in a delegate that GC-allocates.  If OTOH we try to
be inclusive and make the delegate parameter unmarked, then  nogc code
will not be able to call `process`.  The only workaround in this case is
to invent `processWithGC` and `processWithNoGC` (because we cannot
overload on  nogc/non- nogc) with identical function bodies.  This is a
waste, since the function bodies can be merged without changing any
semantics at all.

The same argument applies to the other attributes: pure,  safe, nothrow,
etc..


T

-- 
He who sacrifices functionality for ease of use, loses both and deserves
neither. -- Slashdotter
Jan 19 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2023 7:19 PM, H. S. Teoh wrote:

 This is a flaw in the language: there is no way to express that
 `process`'s attributes inherit from the passed-in delegate.  This has
 been a pain point for years.


void process()(void delegate() userData) {}
Jan 19 2023
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Jan 19, 2023 at 07:41:02PM -0800, Walter Bright via Digitalmars-d wrote:

 On 1/19/2023 7:19 PM, H. S. Teoh wrote:

 This is a flaw in the language: there is no way to express that
 `process`'s attributes inherit from the passed-in delegate.  This
 has been a pain point for years.


 
 void process()(void delegate() userData) {}


Code:

````
void process()(void delegate() cb) { cb(); }

void gcFunc() {
	int[] a;
	process({ a = new int[10]; });
}

void nogcFunc(void delegate()  nogc cb)  nogc {
	process(cb);
}
````

Compiler output:

````
/tmp/test.d(9): Error: ` nogc` function `test.nogcFunc` cannot call non- nogc
function `test.process!().process`
````

Nope, doesn't work.


T

-- 
Obviously, some things aren't very obvious.
Jan 19 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2023 8:28 PM, H. S. Teoh wrote:

 void process()(void delegate() cb) { cb(); }


void process(T)(T cb) { cb(); }
Jan 20 2023
parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 20 January 2023 at 19:15:09 UTC, Walter Bright wrote:

 On 1/19/2023 8:28 PM, H. S. Teoh wrote:

 void process()(void delegate() cb) { cb(); }


 void process(T)(T cb) { cb(); }


Quirin Schroll addresses the drawbacks of that
https://forum.dlang.org/post/fzujvfmtckvcusdelzqj forum.dlang.org

HS Teoh adds another issue here:
https://forum.dlang.org/post/mailman.8202.1674234814.31357.digitalmars-d puremagic.com
Jan 20 2023
prev sibling parent reply Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Friday, 20 January 2023 at 03:41:02 UTC, Walter Bright wrote:

 On 1/19/2023 7:19 PM, H. S. Teoh wrote:

 This is a flaw in the language: there is no way to express that
 `process`'s attributes inherit from the passed-in delegate.  
 This has
 been a pain point for years.


 void process()(void delegate() userData) {}


Er, no. The template `process` will be inferred ` system thorw 
 gc impure` because the delegate is annotated ` system thorw  gc 
impure` implicitly – unless you don’t actually call it. The 
correct version is this:
```D
void process(DG : void delegate())(DG callback) { callback(); }
```

This has two drawbacks:
1. `process` cannot be virtual.
2. the argument bound to `callback` cannot have its parameter 
types inferred.

By 2. I mean the little more interesting example when the 
delegate takes parameters, say `int`:
```D
/*A*/ void process(void delegate(int) callback) { callback(); }
/*B*/ void process(DG : void delegate(int))(DG callback) { 
callback(); }
```
(The constraint is optional, but useful.)

For version A, when `process` is called, like `process((x){})`, 
the compiler can statically infer that `x` is of type `int`. For 
version B, it cannot, but it does not matter practically for 
functions like `process`, however it does matter for `opApply` 
because programmers using a `foreach` loop really want the type 
of the iteration variable inferred (especially in meta 
programming); this inference only works when `opApply` is not a 
template. `opApply` may still be a template instance (or an alias 
to a template instance), which we can put to use. The shortest I 
could get to:
```d
import std.meta : AliasSeq;
import std.traits : SetFunctionAttributes, functionLinkage, 
functionAttributes, FunctionAttribute;

template WithAnyCombinationOfAttributes(DG)
{
     alias WithAnyCombinationOfAttributes = AliasSeq!();
     static foreach (safety; [ FunctionAttribute.system, 
FunctionAttribute.safe ])
	static foreach (purity; [ FunctionAttribute.none, 
FunctionAttribute.pure_ ])
	static foreach (gcness; [ FunctionAttribute.none, 
FunctionAttribute.nogc ])
	static foreach (exness; [ FunctionAttribute.none, 
FunctionAttribute.nothrow_ ])
     {
         WithAnyCombinationOfAttributes = 
AliasSeq!(WithAnyCombinationOfAttributes, 
SetFunctionAttributes!(DG, functionLinkage!DG, 
(functionAttributes!DG & ~ FunctionAttribute.system) | safety | 
purity | gcness | exness));
     }
}

mixin template opApplyFromImpl(alias impl, protoDG, alias 
_WithAnyCombinationOfAttributes = WithAnyCombinationOfAttributes)
{
     static foreach (DG; _WithAnyCombinationOfAttributes!protoDG)
	alias opApply = impl!DG;
}
```

You can find the code in action 
[here](https://wandbox.org/permlink/rqasKlZTaspJ9gnI). In my 
opinion, something like this should be in Phobos.

The drawbacks are lots of template instantiations and that it 
always generates 16 overloads.
Jan 20 2023
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Jan 20, 2023 at 11:49:42AM +0000, Quirin Schroll via Digitalmars-d
wrote:
[...]

 The correct version is this:
 ```D
 void process(DG : void delegate())(DG callback) { callback(); }
 ```
 
 This has two drawbacks:
 1. `process` cannot be virtual.
 2. the argument bound to `callback` cannot have its parameter types
 inferred.


And 3.: it causes the body of `process` to be duplicated across multiple
instantiations, even if the only difference in DG is in attributes,
i.e., the generated code is 100% identical.  This leads to template
bloat.


[...]

 The drawbacks are lots of template instantiations and that it always
 generates 16 overloads.


Exactly. The language needs to support this natively, not leave it up to
workarounds that bring in lots of template bloat.


T

-- 
Famous last words: I *think* this will work...
Jan 20 2023
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2023 9:13 AM, H. S. Teoh wrote:

 And 3.: it causes the body of `process` to be duplicated across multiple
 instantiations, even if the only difference in DG is in attributes,
 i.e., the generated code is 100% identical.  This leads to template
 bloat.


This is a general issue with templates. Some linkers are able to remove 
duplicates. The compiler could be enhanced for it, too. It's not a language
problem.
Jan 20 2023
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/20/2023 3:49 AM, Quirin Schroll wrote:

 1. `process` cannot be virtual.


Virtual functions are meant to be overridden, meaning their attributes are 
inherited with covariant and contravariant rules. This is incompatible with 
attribute inference.


 2. the argument bound to `callback` cannot have its parameter types inferred.


The version of this I posted can.
Jan 20 2023
parent reply Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Friday, 20 January 2023 at 20:28:52 UTC, Walter Bright wrote:

 On 1/20/2023 3:49 AM, Quirin Schroll wrote:

 1. `process` cannot be virtual.


 Virtual functions are meant to be overridden, meaning their 
 attributes are inherited with covariant and contravariant 
 rules. This is incompatible with attribute inference.


 2. the argument bound to `callback` cannot have its parameter 
 types inferred.


 The version of this I posted can.


TL;DR: You can infer
_either_ parameter types of callbacks
_or_ attributes of callbacks,
but you can never infer _both._

Maybe you misunderstood what I was trying to convey.

I’m guessing you mean this version:
```d
void process()(void delegate() userData) {/*impl*/}
```

Trying 2.099 (dmd-nightly on run.dlang.org), it cannot infer 
types. Only non-templates can.



I’m using `opApply` because it’s (as far as I know) the only 
callback pattern mentioned in the spec and it’s by far the most 
restricted. If `opApply` works, it works with simpler cases, too.

```d
struct S1
{
     int opApply     (scope int delegate(string)       callback)
     { return callback(""); }
}
struct S2
{
     int opApply     (scope int delegate(string)  safe callback) 
 safe
     { return callback(""); }
}
struct S3
{
     int opApply()   (scope int delegate(string)       callback)
     { return callback(""); }
}
struct S4
{
     int opApply(DG :       int delegate(string))(DG   callback)
     { return callback(""); }
}
struct S5
{
     int opApply(DG                       )(DG         callback)
     { return callback(""); }
}

void main()  safe
{
     foreach (x; S1()) { } // Error: ` safe` function `D main` 
cannot call
                           //        ` system` function 
`onlineapp.S1.opApply`
     foreach (x; S2()) { } // OK
     foreach (x; S3()) { } // Error: cannot infer type for 
`foreach` variable `x`
     foreach (x; S4()) { } // Error: cannot infer type for 
`foreach` variable `x`
     foreach (x; S5()) { } // Error: cannot infer type for 
`foreach` variable `x`


     foreach (string x; S1()) { } // Error: ` safe` function `D 
main` cannot call
                                  //` system` function 
`onlineapp.S1.opApply`
     foreach (string x; S2()) { } // OK
     foreach (string x; S3()) { } // Error: none of the overloads 
of template
                                  // `onlineapp.S3.opApply` are 
callable using
                                  // argument types (**)
     foreach (string x; S4()) { } // Error: none of the overloads 
of template
                                  // `onlineapp.S4.opApply` are 
callable using
                                  // argument types (**)
     foreach (string x; S5()) { } // Error: template instance (++) 
error instantiating
     // (**) = `!()(int delegate(ref string __applyArg0) pure 
nothrow  nogc  safe)`
     // (++) = `onlineapp.S5.opApply!(int delegate(ref string) 
pure nothrow  nogc  safe)`

     foreach (string x; &S4().opApply!(int delegate(string) 
 safe)) { } // OK
     foreach (string x; &S5().opApply!(int delegate(string) 
 safe)) { } // OK
}
```


Failure on `S1` is due to the lack of “` safe` relative to 
callable argument”; `S2` proves that by annotating `opApply` and 
restricting `callback` to ` safe` arguments.

Failure on `S3`, `S4`, and `S5` are due to lack of inference in 
the type-inferred cases; `S3` failing is unreasonable to some 
degree because the callback’s type does not depend on template 
arguments. The pattern in `S4` (type parameter used for the only 
argument is restricted to a delegate type) could be recognized by 
the compiler, but that would clearly be an enhancement and not a 
bug. That `S5` fails is to be expected. Information about the 
types (even arity) of delegate would have to be inferred from the 
body of `opApply`.



(No change by providing a type: Failure on `S1` is due to the 
lack of “` safe` relative to callable argument”; `S2` proves that 
by annotating `opApply` and restricting `callback` to ` safe` 
arguments.)

I’d have expected `S3` to fail because the function won’t ever be 
inferred ` safe` because the callback is not restricted to 
` safe`.
`S4` and `S5` should work, but don’t. The compiler attempts to 
instantiate them with an inappropriate delegate type: `int 
delegate(ref string)  …`. This is a bug. For every argument, 
unless the call-site uses `ref` on the argument, the compiler 
must try a delegate type without `ref`; for *n* arguments, this 
can lead to 2*ⁿ* attempts; this is by nature, it cannot be 
helped, however, if non-`ref` ones are tried first, it’s likely 
they succeed on the first attempt. This can be seen when you 
instantiate `opApply` explicitly.
Jan 23 2023
parent reply Walter Bright <newshound2 digitalmars.com> writes:
Help us help you by posting the bug reports to bugzilla!
Jan 23 2023
parent Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Tuesday, 24 January 2023 at 07:45:00 UTC, Walter Bright wrote:

 Help us help you by posting the bug reports to bugzilla!


I filed issues about `opApply` in the past. An example is 
[this](https://issues.dlang.org/show_bug.cgi?id=19597).

Recognizing `ref`-ness is already filed if I remember correctly; 
apart from that, technically speaking, those aren’t bugs, but 
various degrees of potential enhancements.

The empty template parameter case is the easiest. Not supporting 
just makes the spec a little easier (function templates don’t 
infer vs. function templates don’t infer unless instantiating the 
template with empty parentheses compiles). It’s still rather 
useless. It cannot infer attributes based on the caller-side 
delegate supplied: The delegate type – and with it its attributes 
– is fixed.

Recognizing the restriction pattern `int opApply(DG : int 
delegate(string)(DG dg)` would save a lot of typing (enhancement 
filed as [issue 
23666](https://issues.dlang.org/show_bug.cgi?id=23666)). The 
repetition of up to 16 overloads can be automated using a mixin 
template ([I posted one before in this 
thread](https://forum.dlang.org/post/fzujvfmtckvcusdelzqj forum.dlang.org))
that takes a implementation template and instantiates it with the given
delegate type varied with all combinations of attributes. I tried to improve
this so that the mixin template needs not be given the delegate type, but finds
the template parameter’s constraint type and uses that. This didn’t work
for two reasons: 1. For some reason, the string extracted cannot be mixed-in as
a type, even if comprised of built-in types only, e.g. `int delegate(int)`. 2.
I know that even if one got the first problem solved, the mixed-in string need
not designate a valid type because of various ways scoping works; it could
still be a good best-effort solution. The compiler quite likely has information
about the parameter constraint, the language just lacks `__traits` to expose
it. (Enhancement filed as [issue
23665](https://issues.dlang.org/show_bug.cgi?id=23665); it’s only long
because it’s inherently complicated.)
Feb 02 2023
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 19 January 2023 at 01:51:41 UTC, A moo person wrote:

 This is a semi rant born from frustration. Currently I'm trying 
 to figure out Mir Ion so I can serialize some data, and oh boy 
 is it frustrating.

 [snip]


Have you tried filing an issue?

Some mir functionality could have better documentation or 
tutorials. I don't know a lot about mir ion, but that was my 
sense of it.
Jan 19 2023
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/18/23 8:51 PM, A moo person wrote:

 This is a semi rant born from frustration. Currently I'm trying to 
 figure out Mir Ion so I can serialize some data, and oh boy is it 
 frustrating.
 
 The entire library is annotated out the wazoo with  safe and  nogc and 
 pure and inout. Its a serialization library so as you might imagine it's 
 doing reflection on types, calling member functions to try and serialize 
 stuff, things like that. I have spent hours now adding attributes and 
 casting attributes away. Writing  trusted wrapper types for std.algo 
 ranges. Trying to decipher the vomit that dmd gives when one the types 
 is not quite right. All I want to do is serialize a list of structs from 
 a range and be able to deserialize them back but I can't do that because 
 I still have not gotten it working.
 
 It's actually maddening, I have spent 2 days on this. What value are all 
 these attributes adding? Because it's a useability nightmare. Every time 
 I encounter a D library that tries to be clever with these annotations, 
 it's just absolutely made things more complex and less useable. Same 
 story every time I have ever tried to use them in my own code.


Something like serialization/deserialization (which requires templates) 
should use attribute inference, and use unittests to prove memory 
safety/nogc/etc.

Now, this *can* break down if not explicit since the compiler sometimes 
gives up on inference, so maybe that's what happened here.

-Steve
Jan 19 2023
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 19 January 2023 at 15:22:08 UTC, Steven 
Schveighoffer wrote:

 Something like serialization/deserialization (which requires 
 templates) should use attribute inference, and use unittests to 
 prove memory safety/nogc/etc.

 Now, this *can* break down if not explicit since the compiler 
 sometimes gives up on inference, so maybe that's what happened 
 here.

 -Steve


That is exactly what happened here. It *should* just work with 
inference but it was impossible. Even reducing to a simple case 
wasn't possible, because it only appears in the large.

Note I wasn't personally involved, just 2nd hand experience.
Jan 20 2023
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Thu, Jan 19, 2023 at 01:51:41AM +0000, A moo person via Digitalmars-d wrote:
[...]

 The entire library is annotated out the wazoo with  safe and  nogc and
 pure and inout. Its a serialization library so as you might imagine
 it's doing reflection on types, calling member functions to try and
 serialize stuff, things like that.


IMO, library code, esp. templated library code, should never have any
explicit attributes.  It should be left to compiler inference so that it
will work with user types of any kind, and attributes within the library
code itself should be enforced by appropriately attributed unittests.



 I have spent hours now adding attributes and casting attributes away.
 Writing  trusted wrapper types for std.algo ranges.


Something has gone deeply wrong when one has to resort to casts.  I'd
say file a bug against the library.



 Trying to decipher the vomit that dmd gives when one the types is not
 quite right.  All I want to do is serialize a list of structs from a
 range and be able to deserialize them back but I can't do that because
 I still have not gotten it working.


I'd just write serialization code myself. Thanks to D's introspection
abilities, this kind of code is not hard to write.  Use
std.traits.FieldNameTuple to get a list of fields in the struct, then
use std.conv.to to convert them to string.  On deserialization, do the
same thing except use std.conv.to to convert from the string to the
field type.  Probably some amount of customization will be needed if
your fields need special handling (e.g. string literals that should be
escaped in the serialized form / unescaped when you deserialize, etc.);
you can just use `static if (is(typeof(field) == MyCustomType))` to
handle those on a case-by-case basis.



 It's actually maddening, I have spent 2 days on this. What value are
 all these attributes adding? Because it's a useability nightmare.
 Every time I encounter a D library that tries to be clever with these
 annotations, it's just absolutely made things more complex and less
 useable. Same story every time I have ever tried to use them in my own
 code.


My policy is to let the compiler infer attributes as much as possible,
and only write them manually when I really, really, *really* need to.
Then use attributed unittests to make sure the compiler doesn't suddenly
change its mind about how something is attributed.

In general, I think we should move in the direction of expanding
attribute inference as far as possible.  Nobody wants to deal with
attribute soup; it should be the compiler's job to automate this tedium
as much as possible and only leave it up to the human when there's no
other way around it.


T

-- 
Knowledge is that area of ignorance that we arrange and classify. -- Ambrose
Bierce
Jan 19 2023
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/19/2023 4:31 PM, H. S. Teoh wrote:

 Nobody wants to deal with attribute soup


<wry>
ImportC has to deal with the attribute soup of extensions added to various C 
compilers, all different. D is not so bad!
</wry>
Jan 19 2023
prev sibling next sibling parent Quirin Schroll <qs.il.paperinik gmail.com> writes:
On Friday, 20 January 2023 at 00:31:54 UTC, H. S. Teoh wrote:

 […]

 In general, I think we should move in the direction of 
 expanding attribute inference as far as possible.  Nobody wants 
 to deal with attribute soup; it should be the compiler's job to 
 automate this tedium as much as possible and only leave it up 
 to the human when there's no other way around it.


I understand why Walter generally does not want inferred 
attributes for non-template functions. It increases build times 
and a non-template function simply *is* e.g. `pure` or not – it 
does not depend on anything outside the function or how it’s 
used. It would help if the compiler can be asked to inform you 
about sub-optimally annotated functions. It could be a 
compiler-flag (to check all functions compiled) and/or a `pragma` 
for checking on a per-module or per-function basis. It would list 
functions and the annotations they lack, but could have.
Jan 20 2023
prev sibling parent A moo person <moo_mail fake.com> writes:
On Friday, 20 January 2023 at 00:31:54 UTC, H. S. Teoh wrote:

 I'd just write serialization code myself.


Yep this is what I am doing now. Not the first time I have 
written a serializer in D using traits(allMembers) but its old 
and not well maintained. I was attracted to Mir because it can do 
text and binary. I was hoping to use text for development and 
switch to binary for release. But I will just write my own, it 
might be a little more work but it's not so bad if I don't try to 
handle indirection and allocation. Just structs, getter/setters 
that return structs, and input/output ranges for iterables.
Jan 20 2023
prev sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 19 January 2023 at 01:51:41 UTC, A moo person wrote:

 This is a semi rant born from frustration. Currently I'm trying 
 to figure out Mir Ion so I can serialize some data, and oh boy 
 is it frustrating.


We use sbin for [de]serialization, which just works (in our case).

https://code.dlang.org/packages/sbin

-- Bastiaan.
Jan 23 2023