digitalmars.D.learn - How do I use CTFE to generate an immutable associative array at
- Chad Joan (58/58) Feb 21 2017 Hello all,
- Stefan Koch (4/7) Feb 21 2017 You cannot create AA's at ctfe and carry them over to runtime use.
- Chad Joan (5/15) Feb 21 2017 I wondered if this might be the case.
- Chad Joan (32/48) Feb 21 2017 For the sake of the rest of the internet, here is what I'm
- H. S. Teoh via Digitalmars-d-learn (27/82) Feb 21 2017 [...]
- Chad Joan (9/35) Feb 21 2017 Hi,
- Chad Joan (52/93) Feb 21 2017 OK I couldn't help it:
- H. S. Teoh via Digitalmars-d-learn (36/36) Feb 21 2017 On Wed, Feb 22, 2017 at 12:38:47AM +0000, Chad Joan via Digitalmars-d-le...
- Era Scarecrow (22/26) Feb 21 2017 I have a partial static AA implementation that seems like it
- Daniel Kozak via Digitalmars-d-learn (4/55) Feb 21 2017 I have similar issue and I beleive I was able to workaround that
- H. S. Teoh via Digitalmars-d-learn (10/13) Feb 21 2017 That may appear to work, but I would *strongly* recommend against it,
- Jacob Carlborg (12/18) Feb 22 2017 You can use an enum to declare the AA and then assign it to an immutable...
- Daniel Kozak via Digitalmars-d-learn (3/22) Feb 22 2017 Yes this is how I mean it.
- H. S. Teoh via Digitalmars-d-learn (11/23) Feb 22 2017 [...]
- Stefan Koch (8/11) Feb 22 2017 All the problems disappear if the AA's are compiler visible and
- H. S. Teoh via Digitalmars-d-learn (28/39) Feb 22 2017 I'm not sure it's that simple. Just because AA's become CTFEable
- Stefan Koch (5/24) Feb 22 2017 The CTFE allocated memory can be copied into the object file
- H. S. Teoh via Digitalmars-d-learn (6/32) Feb 22 2017 This will fail for cross-compilation. Unless you mean that the new CTFE
Hello all, I'm trying to make this work: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup = parseTwoColumnCsv(import("some_data.csv")); void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- But (with DMD 2.073.1) I am getting this error: main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x', 'y', 'z'], ['1', '2', '3']:['4', '5', '6']] The case with normal (non-associative) arrays seems to work fine, but is not what I needed: --- pure string[][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[][] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result ~= [record[0],record[1]]; return result; } immutable string[][] dataLookup = parseTwoColumnCsv(import("some_data.csv")); void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- Any idea how I can get this working? I have tried a couple other things, like having the parseTwoColumnCsv function return an immutable string[string] and casting 'result' to that in the return statement, and I also tried just casting the value returned from parseTwoColumnCsv as it appears in the declaration of 'dataLookup'. I tried std.exception's assumeUnique, but the function/template signature doesn't seem to support associative arrays. Thanks.
Feb 21 2017
On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:Hello all, I'm trying to make this work: [...]You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
Feb 21 2017
On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!Hello all, I'm trying to make this work: [...]You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
Feb 21 2017
On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:For the sake of the rest of the internet, here is what I'm probably going to stick with: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup; static this() { dataLookup = parseTwoColumnCsv(import("some_data.csv")); } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- In this case the AA isn't actually coded into the executable; but at least the configuration from some_data.csv will be in the executable as a string. The program will construct the AA at startup. It's not as "cool", but it should get the job done. HTH.On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!Hello all, I'm trying to make this work: [...]You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
Feb 21 2017
On Tue, Feb 21, 2017 at 10:34:57PM +0000, Chad Joan via Digitalmars-d-learn wrote:On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:[...] Parsing strings at program startup is ugly and slow. What about parsing at compile-time with CTFE into an array literal and transforming that into an AA at startup, which should be a lot faster? // Warning: untested code pure string[2][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[2][] result; foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) { result ~= [record[0], record[1]]; } return result; } immutable string[string] dataLookup; static this() { enum halfCookedData = parseTwoColumnCsv(import("some_data.csv")); foreach (p; halfCookedData) { dataLookup[p[0]] = p[1]; } } T -- Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at it. -- Pete BleackleyOn Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:For the sake of the rest of the internet, here is what I'm probably going to stick with: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup; static this() { dataLookup = parseTwoColumnCsv(import("some_data.csv")); } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- In this case the AA isn't actually coded into the executable; but at least the configuration from some_data.csv will be in the executable as a string. The program will construct the AA at startup. It's not as "cool", but it should get the job done.On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!Hello all, I'm trying to make this work: [...]You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
Feb 21 2017
On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:Parsing strings at program startup is ugly and slow. What about parsing at compile-time with CTFE into an array literal and transforming that into an AA at startup, which should be a lot faster? // Warning: untested code pure string[2][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[2][] result; foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) { result ~= [record[0], record[1]]; } return result; } immutable string[string] dataLookup; static this() { enum halfCookedData = parseTwoColumnCsv(import("some_data.csv")); foreach (p; halfCookedData) { dataLookup[p[0]] = p[1]; } } THi, That makes a lot of sense and it had crossed my mind. In my case the data sets are super small, so I'm probably going to be lazy/productive and leave it the way it is. Anyone else from the internet copying these examples: try to just do it H.S. Teoh's way from the start ;) Thanks for the suggestion. - Chad
Feb 21 2017
On Tuesday, 21 February 2017 at 23:30:52 UTC, Chad Joan wrote:On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:OK I couldn't help it: --- pure private string[][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[][] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result ~= [record[0],record[1]]; return result; } pure private string[string] twoColumnArrayToAA(const string[][] arr) { string[string] result; foreach ( pair; arr ) result[pair[0]] = pair[1]; return result; } pure private string[string] importTwoColumnCsv(string csvFile)() { // Force the parse to happen at compile time. immutable string[][] tempArray = import(csvFile).parseTwoColumnCsv(); // Convert the parsed array into a runtime Associative Array and return it. return tempArray.twoColumnArrayToAA(); } immutable string[string] dataLookup; immutable string[string] dataLookup1; immutable string[string] dataLookup2; static this() { import std.range; dataLookup1 = importTwoColumnCsv!"some_data1.csv"; dataLookup2 = importTwoColumnCsv!"some_data2.csv"; foreach( pair; chain(dataLookup1.byKeyValue, dataLookup2.byKeyValue) ) dataLookup[pair.key] = pair.value; } void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- This example also shows joining associative arrays.Parsing strings at program startup is ugly and slow. What about parsing at compile-time with CTFE into an array literal and transforming that into an AA at startup, which should be a lot faster? // Warning: untested code pure string[2][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[2][] result; foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) { result ~= [record[0], record[1]]; } return result; } immutable string[string] dataLookup; static this() { enum halfCookedData = parseTwoColumnCsv(import("some_data.csv")); foreach (p; halfCookedData) { dataLookup[p[0]] = p[1]; } } THi, That makes a lot of sense and it had crossed my mind. In my case the data sets are super small, so I'm probably going to be lazy/productive and leave it the way it is. Anyone else from the internet copying these examples: try to just do it H.S. Teoh's way from the start ;) Thanks for the suggestion. - Chad
Feb 21 2017
On Wed, Feb 22, 2017 at 12:38:47AM +0000, Chad Joan via Digitalmars-d-learn wrote: [...] Hmm. It's actually not necessary to manually write a foreach loop to convert an array to an AA. We could, instead, change parseTwoColumnCsv() to return an array of std.typecons.Tuple instead (which, incidentally, means you can elide that foreach loop too!), then there's this handy function std.array.assocArray that will do the transcription for us, as follows. (In fact, you can merge AA's without ever needing to write `foreach` at all! See below.) --- pure private auto parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; return csvReader!(Tuple!(string,string))(inputCsv); } immutable string[string] dataLookup; immutable string[string] dataLookup1; immutable string[string] dataLookup2; static this() { import std.array : assocArray; import std.range : chain; // Force CTFE immutable tuples1 = parseTwoColumnCsv("some_data1.csv"); immutable tuples2 = parseTwoColumnCsv("some_data2.csv"); dataLookup1 = tuples1.assocArray; // Bam! :-P dataLookup2 = tuples2.assocArray; // Bam! :-P // Bam, bam! - merge AA's in a single step! dataLookup3 = chain(tuples1, tuples2).assocArray; } --- T -- You have to expect the unexpected. -- RL
Feb 21 2017
On Tuesday, 21 February 2017 at 22:34:57 UTC, Chad Joan wrote:In this case the AA isn't actually coded into the executable; but at least the configuration from some_data.csv will be in the executable as a string. The program will construct the AA at startup. It's not as "cool", but it should get the job done.I have a partial static AA implementation that seems like it works, I mentioned this in a different thread. https://github.com/rtcvb32/Side-Projects/blob/master/staticaa.d Try it out, etc. Usage: Create your AA as an enum (for simplicity) StaticAA!(KeyType, ValueType, getAALen(EnumAssosiativeArray), EnumAssosiativeArray.length)(EnumAssosiativeArray); Afterwards use it as you normally would for the same thing. Unittest example: enum AA = ["one":1, "two":2, "three":3, "four":4, "five":5, "six":6, "seven":7, "eight":8, "nine":9, "zero":0]; auto SAA = StaticAA!(string, int, getAALen(AA), AA.length)(AA); //just verifies the keys/values match. foreach(k, v; AA) { assert(SAA[k] == v); } Note: getAALen basically tests the array expanding it out until none of the hashes overlap or causes problems. Trying to compile these into a single call I've had issues, so if anyone has a better solution I'd go for it.
Feb 21 2017
I have similar issue and I beleive I was able to workaround that somehow, but it is so many years now :(. Have you tried enum dataLookup instead of immutable string[string] dataLookup Dne 21.2.2017 v 22:53 Chad Joan via Digitalmars-d-learn napsal(a):Hello all, I'm trying to make this work: --- pure string[string] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[string] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result[record[0]] = record[1]; return result; } immutable string[string] dataLookup = parseTwoColumnCsv(import("some_data.csv")); void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- But (with DMD 2.073.1) I am getting this error: main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x', 'y', 'z'], ['1', '2', '3']:['4', '5', '6']] The case with normal (non-associative) arrays seems to work fine, but is not what I needed: --- pure string[][] parseTwoColumnCsv(string inputCsv) { import std.csv; import std.typecons; string[][] result; foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) ) result ~= [record[0],record[1]]; return result; } immutable string[][] dataLookup = parseTwoColumnCsv(import("some_data.csv")); void main() { import std.stdio; writefln("dataLookup = %s", dataLookup); } --- Any idea how I can get this working? I have tried a couple other things, like having the parseTwoColumnCsv function return an immutable string[string] and casting 'result' to that in the return statement, and I also tried just casting the value returned from parseTwoColumnCsv as it appears in the declaration of 'dataLookup'. I tried std.exception's assumeUnique, but the function/template signature doesn't seem to support associative arrays. Thanks.
Feb 21 2017
On Tue, Feb 21, 2017 at 11:50:02PM +0100, Daniel Kozak via Digitalmars-d-learn wrote:I have similar issue and I beleive I was able to workaround that somehow, but it is so many years now :(. Have you tried enum dataLookup instead of immutable string[string] dataLookupThat may appear to work, but I would *strongly* recommend against it, because what happens when you use enum with an AA, is that the AA will be created *at runtime*, *every single time* it is referenced. (It is as if you copy-n-pasted the entire AA into the code each time you reference the enum.) Which will introduce ridiculous amounts of redundant work at runtime and cause a big performance penalty. T -- What did the alien say to Schubert? "Take me to your lieder."
Feb 21 2017
On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote:That may appear to work, but I would *strongly* recommend against it, because what happens when you use enum with an AA, is that the AA will be created *at runtime*, *every single time* it is referenced. (It is as if you copy-n-pasted the entire AA into the code each time you reference the enum.) Which will introduce ridiculous amounts of redundant work at runtime and cause a big performance penalty.You can use an enum to declare the AA and then assign it to an immutable variable using "static this". The you would only use to the immutable variable and never the enum. enum aa = [1 : 2]; immutable int[int] iaa; static this() { iaa = aa; } -- /Jacob Carlborg
Feb 22 2017
Yes this is how I mean it. Dne 22. 2. 2017 9:05 napsal u=C5=BEivatel "Jacob Carlborg via Digitalmars-d-learn" <digitalmars-d-learn puremagic.com>:On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote: That may appear to work, but I would *strongly* recommend against it,because what happens when you use enum with an AA, is that the AA will be created *at runtime*, *every single time* it is referenced. (It is as if you copy-n-pasted the entire AA into the code each time you reference the enum.) Which will introduce ridiculous amounts of redundant work at runtime and cause a big performance penalty.You can use an enum to declare the AA and then assign it to an immutable variable using "static this". The you would only use to the immutable variable and never the enum. enum aa =3D [1 : 2]; immutable int[int] iaa; static this() { iaa =3D aa; } -- /Jacob Carlborg
Feb 22 2017
On Wed, Feb 22, 2017 at 09:00:36AM +0100, Jacob Carlborg via Digitalmars-d-learn wrote: [...]You can use an enum to declare the AA and then assign it to an immutable variable using "static this". The you would only use to the immutable variable and never the enum. enum aa = [1 : 2]; immutable int[int] iaa; static this() { iaa = aa; }[...] Wow, this is miles better than the hacks that I've come up with! Also makes the no-static-AA issue much less of a problem that I thought it was. (In fact, now I'm wondering if we could just hack dmd to emit the equivalent of this code as a lowering, whenever the user tries to declare a compile-time initialized AA.) T -- Computers are like a jungle: they have monitor lizards, rams, mice, c-moss, binary trees... and bugs.
Feb 22 2017
On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote: (In fact, now I'm wondering if we could justhack dmd to emit the equivalent of this code as a lowering, whenever the user tries to declare a compile-time initialized AA.)All the problems disappear if the AA's are compiler visible and CTFEable code. There is as far as I know a project by Martin Nowak that does that. And uses templates for AA's rather then whacky TypeInfos. So in the future this problem will disappear.
Feb 22 2017
On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via Digitalmars-d-learn wrote:On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote:I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you.(In fact, now I'm wondering if we could just hack dmd to emit the equivalent of this code as a lowering, whenever the user tries to declare a compile-time initialized AA.)All the problems disappear if the AA's are compiler visible and CTFEable code.There is as far as I know a project by Martin Nowak that does that. And uses templates for AA's rather then whacky TypeInfos. So in the future this problem will disappear.Oh I know, the problem is that this "future" has been taking a long time arriving. I was involved in an early (but unfortunately unsuccessful) effort to rewrite AA's as library code. But there was simply too much compiler magic involved with AA's that it couldn't work at the time. Much of this magic has been dispelled over the past few years, though, so we should be in better shape now for moving AA's completely into the library. I'm very much looking forward to Martin's implementation when it's ready. But in the meantime, lowering compile-time initialized AA's could have the above hack as a temporary workaround until we can get AA literals to be embeddable in object code. T -- GEEK = Gatherer of Extremely Enlightening Knowledge
Feb 22 2017
On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote:On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via Digitalmars-d-learn wrote:The CTFE allocated memory can be copied into the object file verbatim because the structure is the same. Pointers are easy to fix up as well, because by definition the have to point to static data that will be in the object file.[...]I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. [...]
Feb 22 2017
On Wed, Feb 22, 2017 at 06:17:14PM +0000, Stefan Koch via Digitalmars-d-learn wrote:On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote:This will fail for cross-compilation. Unless you mean that the new CTFE engine is emulating the target machine directly? T -- Computers shouldn't beep through the keyhole.On Wed, Feb 22, 2017 at 04:08:45PM +0000, Stefan Koch via Digitalmars-d-learn wrote:The CTFE allocated memory can be copied into the object file verbatim because the structure is the same. Pointers are easy to fix up as well, because by definition the have to point to static data that will be in the object file.[...]I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. [...]
Feb 22 2017