www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Reading IDX Files in D, an introduction to compile time programming

reply data pulverizer <data.pulverizer gmail.com> writes:
I have written an article targeted at people new to D on 
compile-time programming: 
https://www.active-analytics.com/blog/reading-idx-files-in-d/ and 
tweeted it here: 
https://twitter.com/chibisi/status/1296824381088440320?s=20

Comments welcome.

Thanks in advance.
Aug 21
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 21, 2020 at 03:04:30PM +0000, data pulverizer via
Digitalmars-d-announce wrote:
 I have written an article targeted at people new to D on compile-time
 programming:
 https://www.active-analytics.com/blog/reading-idx-files-in-d/
[...] CSS leakage into text in 2nd bullet point under "Introduction": "uspadding: 0.5em;s" should be "uses". T -- Shin: (n.) A device for finding furniture in the dark.
Aug 21
parent data pulverizer <data.pulverizer gmail.com> writes:
On Friday, 21 August 2020 at 15:54:14 UTC, H. S. Teoh wrote:
 CSS leakage into text in 2nd bullet point under "Introduction": 
 "uspadding: 0.5em;s" should be "uses".
Thanks, just fixed it.
Aug 21
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 21, 2020 at 08:54:14AM -0700, H. S. Teoh via Digitalmars-d-announce
wrote:
 On Fri, Aug 21, 2020 at 03:04:30PM +0000, data pulverizer via
Digitalmars-d-announce wrote:
 I have written an article targeted at people new to D on
 compile-time programming:
 https://www.active-analytics.com/blog/reading-idx-files-in-d/
[...] CSS leakage into text in 2nd bullet point under "Introduction": "uspadding: 0.5em;s" should be "uses".
[...] Anyway, besides that typo / formatting error, this is a very interesting idea to generate type declarations at compile-time based on external files. I'm gonna hafta "steal" this idea for my own projects. ;-) T -- Dogs have owners ... cats have staff. -- Krista Casada
Aug 21
prev sibling next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 8/21/20 8:04 AM, data pulverizer wrote:
 I have written an article targeted at people new to D on compile-time 
 programming: 
 https://www.active-analytics.com/blog/reading-idx-files-in-d/ and 
 tweeted it here: 
 https://twitter.com/chibisi/status/1296824381088440320?s=20
 
 Comments welcome.
 
 Thanks in advance.
Great article. (There are other typos in there; I would be happy to review your next articles. :) ) I use "import expressions" for parsing files at compile time as well but your article gave me some ideas where I may be able to parse types of fields very similar to your example. In my case I found a limitation: I cannot "iterate a directory" and import all file contents in there (the limitation is related to a C library function not having source code so it cannot be evaluated). So, I use a build step to generate the file that contains all files in my directory. So, I first import the file list then 'static foreach' that list to import and parse contents of other files. For the record, here is an example of code that does not work at compile time: import std.file; void main() { static foreach (entry; dirEntries(".", SpanMode.shallow)) { } } Error: `fakePureCalloc` cannot be interpreted at compile time, because it has no available source code (I think the error message is different in dmd 2.084 where my project currently uses.) Ali
Aug 21
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali «ehreli via
Digitalmars-d-announce wrote:
[...]
 In my case I found a limitation: I cannot "iterate a directory" and
 import all file contents in there (the limitation is related to a C
 library function not having source code so it cannot be evaluated).
The actual limitation is that string imports do not allow reading directory contents (the C function can be replaced if such were allowed). Generally, I don't expect directory traversal to ever be allowed at compile-time, since it opens the door to a huge can o' security worms. :-P
 So, I use a build step to generate the file that contains all files in
 my directory. So, I first import the file list then 'static foreach'
 that list to import and parse contents of other files.
Yeah, that's probably the simplest solution. [...]
 Error: `fakePureCalloc` cannot be interpreted at compile time, because
 it has no available source code
 
 (I think the error message is different in dmd 2.084 where my project
 currently uses.)
[...] fakePureCalloc is a red herring; even if it weren't a problem, you'd eventually run into the problem that you cannot do directory traversal at compile-time. T -- Time flies like an arrow. Fruit flies like a banana.
Aug 21
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 8/21/20 1:33 PM, H. S. Teoh wrote:

 On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali =C3=87ehreli via=20
Digitalmars-d-announce wrote:
 Generally, I don't expect directory traversal to ever be
 allowed at compile-time, since it opens the door to a huge can o'
 security worms. :-P
I've heard that before. So, if directory traversal is only for import=20 expressions then it should be fine because importing two files cannot be = less safe than importing one file. So, we need to let the compiler know=20 that we will import, nothing fancy: // Mapping from file name -> content string[string] contents =3D static import("my/dir"); Note the clever :o) use of 'static' which means "directory" in this=20 case. :o) Now we have all contents as an AA, ready to be parsed,=20 mixed-in, etc. But I this is not important at all. Build step is just fin= e. Ali
Aug 21
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 21 August 2020 at 20:33:51 UTC, H. S. Teoh wrote:
 On Fri, Aug 21, 2020 at 01:18:30PM -0700, Ali Çehreli via 
 Digitalmars-d-announce wrote: [...]
 In my case I found a limitation: I cannot "iterate a 
 directory" and import all file contents in there (the 
 limitation is related to a C library function not having 
 source code so it cannot be evaluated).
The actual limitation is that string imports do not allow reading directory contents (the C function can be replaced if such were allowed). Generally, I don't expect directory traversal to ever be allowed at compile-time, since it opens the door to a huge can o' security worms. :-P
I feel like limiting CTFE just gives a false sense of security and destroys many interesting use cases. If a part of my build system will do directory traversal to build the list of files to import, what difference would it make to not have this as a single build step. The argument that somehow dmd -run gen_code.d | dmd - Is more secure than just: makes no sense to me. See Jai for example. You can run absolutely *any* code at compile time. 5 years ago Jai's creator made a demo of running an OpenGL game at CT [1]. In the same demo he also used CTFE to validate calls to printf. He made the case that while many compilers go the route of hard-coding checks for printf style functions in the compiler, he thinks that users should be able to implement arbitrary checks in their code. And 5 years later, instead of D expanding the frontiers of what's possible via CTFE, printf checking was hard coded in the compiler [2]. [1]: https://www.youtube.com/watch?v=UTqZNujQOlA [2]: https://github.com/dlang/dmd/pull/10812/files I don't need say that unlimited CTFE has been a huge success for Jai. What I wish is that we can learn from this stop bringing arguments that C people would bring for D's CTFE ("My Makefile calls a Python script to generate C code and it's doing just fine, so I don't think one should be allowed to run code at compile time, as it will make the code just harder to follow"). As another example, types in Zig are first class citizens [3] and can be manipulated with CTFE just like any other value. "Type functions" in D should just be regular D functions taking types as parameters and returning types. [3]: https://ziglang.org/documentation/master/#Introducing-the-Compile-Time-Concept
Aug 22
next sibling parent aberba <karabutaworld gmail.com> writes:
On Saturday, 22 August 2020 at 07:04:19 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Friday, 21 August 2020 at 20:33:51 UTC, H. S. Teoh wrote:

 I don't need say that unlimited CTFE has been a huge success 
 for Jai.
Never heard of that success BTW. Probably a niche success. But that aside, do you acknowledge there are problems with allowing such read access or not?
 What I wish is that we can learn from this stop bringing 
 arguments that C people would bring for D's CTFE
Back in the days, I used to think if D had everything under the sun it would be a sole change factor until I saw (thought through) how C (and some technically meh ones) are still gaining some adoption despite not being even close to any what we have already. By the way, aren't we already successful? Its about time we acknowledge how successful we already are.
Aug 23
prev sibling parent Carl Sturtivant <sturtivant gmail.com> writes:
On Saturday, 22 August 2020 at 07:04:19 UTC, Petar Kirov 
[ZombineDev] wrote:
 I feel like limiting CTFE just gives a false sense of security 
 and destroys many interesting use cases. If a part of my build 
 system will do directory traversal to build the list of files 
 to import, what difference would it make to not have this as a 
 single build step. The argument that somehow

     dmd -run gen_code.d | dmd -

 Is more secure than just:



 makes no sense to me.

 See Jai for example. You can run absolutely *any* code at 
 compile time. 5 years ago Jai's creator made a demo of running 
 an OpenGL game at CT [1]. In the same demo he also used CTFE to 
 validate calls to printf. He made the case that while many 
 compilers go the route of hard-coding checks for printf style 
 functions in the compiler, he thinks that users should be able 
 to implement arbitrary checks in their code. And 5 years later, 
 instead of D expanding the frontiers of what's possible via 
 CTFE, printf checking was hard coded in the compiler [2].

 [1]: https://www.youtube.com/watch?v=UTqZNujQOlA
 [2]: https://github.com/dlang/dmd/pull/10812/files

 I don't need say that unlimited CTFE has been a huge success 
 for Jai. What I wish is that we can learn from this stop 
 bringing arguments that C people would bring for D's CTFE ("My 
 Makefile calls a Python script to generate C code and it's 
 doing just fine, so I don't think one should be allowed to run 
 code at compile time, as it will make the code just harder to 
 follow").

 As another example, types in Zig are first class citizens [3] 
 and can be manipulated with CTFE just like any other value. 
 "Type functions" in D should just be regular D functions taking 
 types as parameters and returning types.

 [3]: 
 https://ziglang.org/documentation/master/#Introducing-the-Compile-Time-Concept
Strongly Agreed!
Aug 23
prev sibling parent Martin Tschierschke <mt smartdolphin.de> writes:
On Friday, 21 August 2020 at 15:04:30 UTC, data pulverizer wrote:
 I have written an article targeted at people new to D on 
 compile-time programming: 
 https://www.active-analytics.com/blog/reading-idx-files-in-d/ 
 and tweeted it here: 
 https://twitter.com/chibisi/status/1296824381088440320?s=20

 Comments welcome.
.
 Thanks in advance.
Very interesting. +1 ! "we extract that an use it to" big error! D is missing :-)
Aug 23