www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Good News: Almost all druntime supported on arsd webassembly

reply Hipreme <msnmancini hotmail.com> writes:
Hello people. I have tried working again with adam's wasm minimal 
runtime, and yesterday I was able to make a great progress on it.

Currently, the only feature that I did not bother to implement 
support for was the try/catch/finally/throw friends. Pretty much 
only because I don't use in my engine. I would like to ask you 
guys some feedback on its current usage, I have written a file 
with only tests to the wasm runtime:

```d
// ldc2 -i=. --d-version=CarelessAlocation -i=std 
-Iarsd-webassembly/ -L-allow-undefined -ofserver/omg.wasm 
-mtriple=wasm32-unknown-unknown-wasm 
arsd-webassembly/core/arsd/aa 
arsd-webassembly/core/arsd/objectutils 
arsd-webassembly/core/internal/utf 
arsd-webassembly/core/arsd/utf_decoding hello 
arsd-webassembly/object.d

import arsd.webassembly;
import std.stdio;

class A {
	int _b = 200;
	int a() { return 123; }
}

interface C {
	void test();
}
interface D {
	void check();
}

class B : A, C
{
	int val;
	override int a() { return 455 + val; }

	void test()
	{
		rawlog(a());
		int[] a;
		a~= 1;
	}
}

void rawlog(Args...)(Args a, string file = __FILE__, size_t line 
= __LINE__)
{
	writeln(a, " at "~ file~ ":", line);
}


struct Tester
{
	int b = 50;
	string a = "hello";
}
void main()
{
	float[] f = new float[4];
	assert(f[0] is float.init);
	f~= 5.5; //Append
	f~= [3, 4];
	int[] inlineConcatTest = [1, 2] ~ [3, 4];

	auto dg = delegate()
	{
		writeln(inlineConcatTest[0], f[1]);
	};
	dg();
	B b = new B;
	b.val = 5;
	A a = b;
	a.a();
	C c = b;
	c.test();
	assert(cast(D)c is null);
	Tester[] t = new Tester[10];
	assert(t[0] == Tester.init);
	assert(t.length == 10);

	switch("hello")
	{
		case "test":
			writeln("broken");
			break;
		case "hello":
			writeln("Working switch string");
			break;
		default: writeln("What happenned here?");
	}
	string strTest = "test"[0..$];
	assert(strTest == "test");

	
	Tester* structObj = new Tester(50_000, "Inline Allocation");
	writeln(structObj is null, structObj.a, structObj.b);

	int[string] hello = ["hello": 500];
	assert(("hello" in hello) !is null, "No key hello yet...");
	assert(hello["hello"] == 500, "Not 500");
	hello["hello"] = 1200;
	assert(hello["hello"] == 1200, "Reassign didn't work");
	hello["h2o"] = 250;
	assert(hello["h2o"] == 250, "New member");


	int[] appendTest;
	appendTest~= 50;
	appendTest~= 500;
	appendTest~= 5000;
	foreach(v; appendTest)
		writeln(v);
	string strConcatTest;
	strConcatTest~= "Hello";
	strConcatTest~= "World";
	writeln(strConcatTest);
	int[] intConcatTest = cast(int[2])[1, 2];
	intConcatTest~= 50;
	string decInput = "a";
	decInput~= "こんいちは";
	foreach(dchar ch; "こんいちは")
	{
		decInput~= ch;
		writeln(ch);
	}
	writeln(decInput);
	int[] arrCastTest = [int.max];

	foreach(v; cast(ubyte[])arrCastTest)
		writeln(v);

}
```

All those tests are currently passing. That means we almost got 
all the common features from the D Runtime into Arsd custom 
runtime. Meaning that the only thing that would be missing right 
now being the WASI libc. But my engine is not going to use it 
entirely, only a subset of it. So, I would like to say that 
whoever wants to play with it now is able to do it so.


That being said, I would carefully advice that while I bring 
those implementations, I didn't care about memory leaks, so, it 
is a runtime without GC: careless allocations. But! It is 
possible to port some programs specially if you're already doing 
house clearing yourself. As my engine does not leak memory in 
loop (as that would make it trigger the GC and thus make it 
slow), it is totally possible to use it.

"But why didn't you continued Skoppe's WASM work?", I literally 
am not able to build LDC runtime no matter how hard I tried. 
Doing that work on a minimal runtime was a lot easier.

If you do find any use in the work I've done, please do test it 
as I'll benefit from both you guys test. If you find any 
performance improvement on it, it'll be gladly be accepted.

I do not intend to replace the druntime with that. I've done the 
minimal subset of features that makes my engine work for the web. 
A real druntime is still expected and awaited by me.

The link to the PR to be tested can be found there: 
https://github.com/adamdruppe/webassembly/pull/9
Jan 06 2023
next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Fri, Jan 06, 2023 at 12:52:43PM +0000, Hipreme via Digitalmars-d-announce
wrote:

 Hello people. I have tried working again with adam's wasm minimal
 runtime, and yesterday I was able to make a great progress on it.


[...]

 All those tests are currently passing. That means we almost got all
 the common features from the D Runtime into Arsd custom runtime.
 Meaning that the only thing that would be missing right now being the
 WASI libc. But my engine is not going to use it entirely, only a
 subset of it. So, I would like to say that whoever wants to play with
 it now is able to do it so.
 
 
 That being said, I would carefully advice that while I bring those
 implementations, I didn't care about memory leaks, so, it is a runtime
 without GC: careless allocations. But! It is possible to port some
 programs specially if you're already doing house clearing yourself. As
 my engine does not leak memory in loop (as that would make it trigger
 the GC and thus make it slow), it is totally possible to use it.


[...]

This is awesome stuff, thanks for pushing ahead with it!!  Keep this up,
and I might actually decide to use your game engine for my projects. ;-)

The big question I have right now is, what's the status of interfacing
with web APIs such as WebGL?  How much JS glue is needed for that to
work?  My dream is for all of the JS boilerplate to be automated away,
so that I don't have to write a single line of JS for my D project to
work in WASM.


T

-- 
Those who don't understand Unix are condemned to reinvent it, poorly.
Jan 06 2023
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 6 January 2023 at 22:13:15 UTC, H. S. Teoh wrote:

 The big question I have right now is, what's the status of 
 interfacing with web APIs such as WebGL?


This part is really easy, you can call it from D with the 
opDispatch or pass it through as eval strings.
Jan 06 2023
prev sibling next sibling parent reply Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Friday, 6 January 2023 at 12:52:43 UTC, Hipreme wrote:

 Hello people. I have tried working again with adam's wasm 
 minimal runtime, and yesterday I was able to make a great 
 progress on it.

 [...]


This sounds great. Thank you for your efforts. I will play around 
with it someday. I have not touched my pet game a while [1]. I 
used emscripten for sdl, and skoppe's druntime fork.

One question. Does GC work with Adam's druntime for wasm? If yes, 
how? IMHO, D's GC need a second thread, which is a problem with 
wasm.

1: https://github.com/aferust/drawee
Jan 06 2023
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 6 January 2023 at 22:14:23 UTC, Ferhat Kurtulmuş wrote:

 One question. Does GC work with Adam's druntime for wasm?


I haven't actually written one yet, so it leaks if you don't pay 
attention yourself. But I have a plan that should work: you do 
the setTimeout(collect, 0) so it runs when the event loop is 
idle. Then the wasm stack is empty so you can scan pure memory.


 IMHO, D's GC need a second thread


This isn't true, it never needs a second thread, even on normal 
desktop.

The problem with GC on wasm is the webasm stack is opaque. You 
can have the compiler output a shadow stack or my plan of 
scanning when it is empty. Either should work but I haven't had 
time to implement anything.
Jan 06 2023
next sibling parent Ferhat =?UTF-8?B?S3VydHVsbXXFnw==?= <aferust gmail.com> writes:
On Friday, 6 January 2023 at 22:39:36 UTC, Adam D Ruppe wrote:

 On Friday, 6 January 2023 at 22:14:23 UTC, Ferhat Kurtulmuş 
 wrote:

 One question. Does GC work with Adam's druntime for wasm?


 I haven't actually written one yet, so it leaks if you don't 
 pay attention yourself. But I have a plan that should work: you 
 do the setTimeout(collect, 0) so it runs when the event loop is 
 idle. Then the wasm stack is empty so you can scan pure memory.


 IMHO, D's GC need a second thread


 This isn't true, it never needs a second thread, even on normal 
 desktop.

 The problem with GC on wasm is the webasm stack is opaque. You 
 can have the compiler output a shadow stack or my plan of 
 scanning when it is empty. Either should work but I haven't had 
 time to implement anything.


thank you for the clarification, Adam.
Jan 09 2023
prev sibling parent reply Lingo Chen <lingo ms1.hinet> writes:
On Friday, 6 January 2023 at 22:39:36 UTC, Adam D Ruppe wrote:

 On Friday, 6 January 2023 at 22:14:23 UTC, Ferhat Kurtulmuş 
 wrote:

 One question. Does GC work with Adam's druntime for wasm?


 I haven't actually written one yet, so it leaks if you don't 
 pay attention yourself. But I have a plan that should work: you 
 do the setTimeout(collect, 0) so it runs when the event loop is 
 idle. Then the wasm stack is empty so you can scan pure memory.


 IMHO, D's GC need a second thread


 This isn't true, it never needs a second thread, even on normal 
 desktop.

 The problem with GC on wasm is the webasm stack is opaque. You 
 can have the compiler output a shadow stack or my plan of 
 scanning when it is empty. Either should work but I haven't had 
 time to implement anything.


https://v8.dev/blog/wasm-gc-porting

wasmgc is shipping. any plan for porting?
Nov 04 2023
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 5 November 2023 at 01:28:25 UTC, Lingo Chen wrote:

 https://v8.dev/blog/wasm-gc-porting

 wasmgc is shipping. any plan for porting?


My understanding is that it requires emitting dedicated 
instructions for allocating such objects, as well as dedicated 
instructions for accessing them. Since LLVM doesn't support them, 
I am not sure what a possible path to support should look like, 
aside from writing a complete fresh backend.
Nov 05 2023
next sibling parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 06/11/2023 5:18 AM, Sebastiaan Koppe wrote:

 I am not sure what a possible path to support should look like, aside 
 from writing a complete fresh backend.


I don't think it would be a whole new backend. The output is the LLVM IR 
essentially, so its just the glue code that would need to be implemented.

However after that has occurred, who knows how much you'd get working 
other than structs newing.
Nov 05 2023
prev sibling parent reply Lingo Chen <lingo ms1.hinet> writes:
On Sunday, 5 November 2023 at 16:18:53 UTC, Sebastiaan Koppe 
wrote:

 On Sunday, 5 November 2023 at 01:28:25 UTC, Lingo Chen wrote:

 https://v8.dev/blog/wasm-gc-porting

 wasmgc is shipping. any plan for porting?


 My understanding is that it requires emitting dedicated 
 instructions for allocating such objects, as well as dedicated 
 instructions for accessing them. Since LLVM doesn't support 
 them, I am not sure what a possible path to support should look 
 like, aside from writing a complete fresh backend.


https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

llvm ir seems to have support for the reftype(gc). so it a 
compiler problem?
Nov 05 2023
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 6 November 2023 at 00:47:43 UTC, Lingo Chen wrote:

 On Sunday, 5 November 2023 at 16:18:53 UTC, Sebastiaan Koppe 
 wrote:

 On Sunday, 5 November 2023 at 01:28:25 UTC, Lingo Chen wrote:

 https://v8.dev/blog/wasm-gc-porting

 wasmgc is shipping. any plan for porting?


 My understanding is that it requires emitting dedicated 
 instructions for allocating such objects, as well as dedicated 
 instructions for accessing them. Since LLVM doesn't support 
 them, I am not sure what a possible path to support should 
 look like, aside from writing a complete fresh backend.


 https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

 llvm ir seems to have support for the reftype(gc). so it a 
 compiler problem?


Unsure how it's progressed since but as can be seen on the last 
slide of that pdf, it's work in progress.
Nov 06 2023
next sibling parent Hipreme <msnmancini hotmail.com> writes:
On Monday, 6 November 2023 at 11:05:45 UTC, Sebastiaan Koppe 
wrote:

 On Monday, 6 November 2023 at 00:47:43 UTC, Lingo Chen wrote:

 On Sunday, 5 November 2023 at 16:18:53 UTC, Sebastiaan Koppe 
 wrote:

 On Sunday, 5 November 2023 at 01:28:25 UTC, Lingo Chen wrote:

 https://v8.dev/blog/wasm-gc-porting

 wasmgc is shipping. any plan for porting?


 My understanding is that it requires emitting dedicated 
 instructions for allocating such objects, as well as 
 dedicated instructions for accessing them. Since LLVM doesn't 
 support them, I am not sure what a possible path to support 
 should look like, aside from writing a complete fresh backend.


 https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

 llvm ir seems to have support for the reftype(gc). so it a 
 compiler problem?


 Unsure how it's progressed since but as can be seen on the last 
 slide of that pdf, it's work in progress.


As shown in my DConf presentation, yes, the GC can be a problem 
but it is a problem in a multitude of problems on the big 
picture. Having a complete libc working with D, for not needing 
to do a massive refactor in code is a really big problem. 
Asincify was used to fix that when using through EMScripten, 
while in my engine I took a completely different approach to fix 
that, but for common projects, it is not possible to have that if 
you want to port your project to WASM.

Even then, there's the communication as shown in the 
presentation, which currently I implement as a hack in my engine 
(which I discovered accidentally from the delegates generating an 
incrementing counter), having proper support to that would be 
totally essential.

So, having a working D Runtime and Phobos is a priority, while 
the GC problem is a thing the developer should not even care and 
is an eventual fix, a simple version upgrade that would make the 
projects simply better. Done is better than perfect, if we have 
no projects in webassembly, I don't believe anyone will get 
interested enough to implement it.
Nov 06 2023
prev sibling parent reply Lingo Chen <lingo ms1.hinet> writes:
On Monday, 6 November 2023 at 11:05:45 UTC, Sebastiaan Koppe 
wrote:

 https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

 llvm ir seems to have support for the reftype(gc). so it a 
 compiler problem?


 Unsure how it's progressed since but as can be seen on the last 
 slide of that pdf, it's work in progress.


Kotlin abandoned llvm and implemented a new wasm backend. So 
waiting for llvm probably is not a good idea?

Is taking D's frontend + binaryen's backend a feasible solution? 
If wasmgc's architecture is a good fit for D. Can it be done 
quickly?
Nov 06 2023
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 6 November 2023 at 23:44:11 UTC, Lingo Chen wrote:

 On Monday, 6 November 2023 at 11:05:45 UTC, Sebastiaan Koppe 
 wrote:

 https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

 llvm ir seems to have support for the reftype(gc). so it a 
 compiler problem?


 Unsure how it's progressed since but as can be seen on the 
 last slide of that pdf, it's work in progress.


 Kotlin abandoned llvm and implemented a new wasm backend. So 
 waiting for llvm probably is not a good idea?

 Is taking D's frontend + binaryen's backend a feasible 
 solution? If wasmgc's architecture is a good fit for D. Can it 
 be done quickly?


I have only looked approx 5 minutes at it, so its really hard to 
say. I expect it to be quite some effort.

There are some things we can do today that are smaller in scope 
and will help any future effort, such as reviving my druntime 
port. You are going to need that anyway, unless you go with the 
custom druntime or betterC route.
Nov 08 2023
parent reply Lingo Chen <lingo ms1.hinet> writes:
On Wednesday, 8 November 2023 at 09:06:31 UTC, Sebastiaan Koppe 
wrote:

 On Monday, 6 November 2023 at 23:44:11 UTC, Lingo Chen wrote:

 On Monday, 6 November 2023 at 11:05:45 UTC, Sebastiaan Koppe 
 wrote:

 https://llvm.org/devmtg/2022-11/slides/TechTalk3-ClangClang-WebAssembly.pdf

 llvm ir seems to have support for the reftype(gc). so it a 
 compiler problem?


 Unsure how it's progressed since but as can be seen on the 
 last slide of that pdf, it's work in progress.


 Kotlin abandoned llvm and implemented a new wasm backend. So 
 waiting for llvm probably is not a good idea?

 Is taking D's frontend + binaryen's backend a feasible 
 solution? If wasmgc's architecture is a good fit for D. Can it 
 be done quickly?


 I have only looked approx 5 minutes at it, so its really hard 
 to say. I expect it to be quite some effort.

 There are some things we can do today that are smaller in scope 
 and will help any future effort, such as reviving my druntime 
 port. You are going to need that anyway, unless you go with the 
 custom druntime or betterC route.


Yes, would love you to revived the druntime port, but I think 
implemented the new wasmgc backend is necessary too.

https://www.pdl.cmu.edu/SDI/2022/slides/Wasm-Basis-of-Next-Gen%20Runtime-Systems.pdf

after reading through the slide, I think wasmgc is the only way 
forward for good Javascript interoperability, and that is 
critical for my use case(handling 3d data, which needs to be gc).

As Rikki stated, it just swapping llvm ir for wasmgc ir. Could be 
non-workable, but I think it worth a try.

No experience in writing compiler, but I can handle 
assembly/forth syntax just fine. Is anyone interested?
Nov 08 2023
parent reply Hipreme <msnmancini hotmail.com> writes:
On Wednesday, 8 November 2023 at 13:29:46 UTC, Lingo Chen wrote:

 On Wednesday, 8 November 2023 at 09:06:31 UTC, Sebastiaan Koppe 
 wrote:

 On Monday, 6 November 2023 at 23:44:11 UTC, Lingo Chen wrote:

 On Monday, 6 November 2023 at 11:05:45 UTC, Sebastiaan Koppe 
 wrote:

 [...]


 Unsure how it's progressed since but as can be seen on the 
 last slide of that pdf, it's work in progress.


 Kotlin abandoned llvm and implemented a new wasm backend. So 
 waiting for llvm probably is not a good idea?

 Is taking D's frontend + binaryen's backend a feasible 
 solution? If wasmgc's architecture is a good fit for D. Can 
 it be done quickly?


 I have only looked approx 5 minutes at it, so its really hard 
 to say. I expect it to be quite some effort.

 There are some things we can do today that are smaller in 
 scope and will help any future effort, such as reviving my 
 druntime port. You are going to need that anyway, unless you 
 go with the custom druntime or betterC route.


 Yes, would love you to revived the druntime port, but I think 
 implemented the new wasmgc backend is necessary too.

 https://www.pdl.cmu.edu/SDI/2022/slides/Wasm-Basis-of-Next-Gen%20Runtime-Systems.pdf

 after reading through the slide, I think wasmgc is the only way 
 forward for good Javascript interoperability, and that is 
 critical for my use case(handling 3d data, which needs to be 
 gc).

 As Rikki stated, it just swapping llvm ir for wasmgc ir. Could 
 be non-workable, but I think it worth a try.

 No experience in writing compiler, but I can handle 
 assembly/forth syntax just fine. Is anyone interested?




I think you didn't quite get, but reviving the runtime Sebastian 
done is way more critical than having the GC, and it is as he 
said, having a new backend is a far away dream, we should not 
jump steps. If the runtime is not revived, there is no point in 
having a GC at all, and it is impossible to use the GC without 
the runtime, but runtime without the GC, it is possible :)
Nov 08 2023
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 09/11/2023 2:42 AM, Hipreme wrote:

 If the runtime is not revived, there is no point in having a GC at all, 
 and it is impossible to use the GC without the runtime, but runtime 
 without the GC, it is possible :)


That's not true.

You wouldn't need anything in druntime to make the wasm GC work. 
Everything would be done in the glue layer of the compiler and would be 
specific to that target.

The druntime bits for GC or TypeInfo wouldn't be used.
Nov 08 2023
parent reply Lingo Chen <lingo ms1.hinet> writes:
On Wednesday, 8 November 2023 at 13:52:34 UTC, Richard (Rikki) 
Andrew Cattermole wrote:

 On 09/11/2023 2:42 AM, Hipreme wrote:

 If the runtime is not revived, there is no point in having a 
 GC at all, and it is impossible to use the GC without the 
 runtime, but runtime without the GC, it is possible :)


 That's not true.

 You wouldn't need anything in druntime to make the wasm GC 
 work. Everything would be done in the glue layer of the 
 compiler and would be specific to that target.

 The druntime bits for GC or TypeInfo wouldn't be used.


Yes, the wasmgc's main selling point is the sharing of GC and 
TypeInfo between Javascript and WASM.

Sharing of data between the JS/WASM are problematic. JS has no 
destructor, so it practically impossible to do manual 
memory/resource management in JS side; One has to do a lot of 
works to pass data around. c/c++/rust, non-gc system languages, 
are not a good fit for JS/WASM, but D if ported to wasmgc would 
be a great fit.

I think D is in a happy position here.
Nov 08 2023
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 9 November 2023 at 04:02:48 UTC, Lingo Chen wrote:

 Yes, the wasmgc's main selling point is the sharing of GC and 
 TypeInfo between Javascript and WASM.


Not just javascript, but any WASM runtime.


 Sharing of data between the JS/WASM are problematic. JS has no 
 destructor, so it practically impossible to do manual 
 memory/resource management in JS side; One has to do a lot of 
 works to pass data around.


With spasm I was auto-generating all glue code based off of 
WebIDL files. The post build step was annoying but it worked 
pretty well.

I even had a prototype that would do it for any typescript 
library.

Of course using WASM ref types is way more ergonomic.

One thing to note about WASMGC, it is MVP and missing a lot of 
things like support for interior pointers etc. There are 
workarounds but it's not so simple.
Nov 08 2023
parent Lingo Chen <lingo ms1.hinet> writes:
On Thursday, 9 November 2023 at 07:01:23 UTC, Sebastiaan Koppe 
wrote:

 On Thursday, 9 November 2023 at 04:02:48 UTC, Lingo Chen wrote:

 Yes, the wasmgc's main selling point is the sharing of GC and 
 TypeInfo between Javascript and WASM.


 Not just javascript, but any WASM runtime.


 Sharing of data between the JS/WASM are problematic. JS has no 
 destructor, so it practically impossible to do manual 
 memory/resource management in JS side; One has to do a lot of 
 works to pass data around.


 With spasm I was auto-generating all glue code based off of 
 WebIDL files. The post build step was annoying but it worked 
 pretty well.

 I even had a prototype that would do it for any typescript 
 library.

 Of course using WASM ref types is way more ergonomic.

 One thing to note about WASMGC, it is MVP and missing a lot of 
 things like support for interior pointers etc. There are 
 workarounds but it's not so simple.


You really should lead a wasmgc porting effort. I am willing to 
contribute too.

WASM is not an easy platform, so adoption rate is slow. But 
WASMGC is a much nicer platform to work with and it going to get 
better too. It a natural platform for D to shine.

Kotlin is all in on WASMGC, and there is a lot of implementation 
detail that can be learned.
https://seb.deleuze.fr/introducing-kotlin-wasm/
Nov 09 2023
prev sibling parent Guillaume Piolat <first.last spam.org> writes:
On Friday, 6 January 2023 at 12:52:43 UTC, Hipreme wrote:

 Hello people. I have tried working again with adam's wasm 
 minimal runtime, and yesterday I was able to make a great 
 progress on it.


Awesome! To think that custom druntime can get you out of 
platform situations is great risk reduction.
Jan 07 2023