www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - vk.xml

reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
So I was going through the vulcan spec to try to create a better 
D bindings for it. (pointer /len pairs to arrays adhering to D 
naming conventions and prettying up the *Create functions 
functions like vkResult *Create( arg ,&arg, retptr) to a fancy 
throw on misuse struct with constructors and that kind of stuff.)

  the structure of the registry is as follows.


struct KHR_registry
{

     string comment; // Not as xml comment  but a <comment> ... 
</comment>

     struct vendorid
     {
         string  Tag name;
         string  Tag id;
         string  Tag comment;
     }
     vendorid[] vendorids;

     struct tag
     {
         string  Tag name;
         string  Tag author;
         string  Tag contact;
     }
     tag[] tags;

     struct validity_type    // does not appear at this level but 
is common to commands and types
     {
         struct usage_type
         {
             string content;
         }
         usage_type[] usage;
     }
     struct type
     {
         string name;            // inconsistanly in tag and 
content
         string requires;
         string parent;
         string content;
         enum category_tpye
         {
             include,
             define,
             basetype,
             bitmask,
             handle,
             enum_,
             funcpointer,
             struct_,

         }
         category_tpye category;
         bool returnedonly;

         validity_type validity;
         struct member
         {
             string typename;      //name of the members type 
variable tag type
             string name;            //name of the member variable
             string len;                //companion member length 
variable for array pointers
             bool optional;                      //should generate 
as a default value
             bool noautovalidity;

         }
         enums* enum;                            //pointer to 
corresponding type enum definition
         struct params
         {
             type* paramType;                    //for some reason 
any pointer qualifiactions occur after e.g.
                                                 // <type> void 
</type> * see coment below
             string name;
             size_t  inderections;               // number of 
pointer inderections before reaching the type pointed to
                                                 // by paramType
                                                 // functions 
taking no args will have parameters == []
             string content;

         }
         params[] parameters;
     }
     type[] types;

     struct enums
     {
         string name,comment, type, expand;
           // expand is the C enum's Prefix e.g. VK_QUERY_TYPE
         struct enum_
         {
             string value, name, comment;
         }
     }

     struct command
     {
         string successcodes;
         string errorcodes;
         string content;
         struct proto_type
         {
             string content;
             type* returnType;
             string name;


         }
         proto_type proto;
         struct pram
         {
             string content;
             bool optional;
             string externsync;
             string len;
             size_t inderections;
             immutable invalidLen = "null-terminated";
             // if len is this then it is not an array
             // although this is inconsistant. sometimes as a len
             // others in the validity
         }
         validity_type validity;
         string queues, renderpass, cmdbufferlevel;
     }

     command[] commands;
     struct feature
     {
         string api, name, number;

         /*
          <require comment="API version">
          <type name="VK_API_VERSION"/>
          <enum name="VK_VERSION_MAJOR"/>
          <command name="VK_VERSION_MINOR"/>
          <type name="VK_VERSION_PATCH"/>
          </require>*/
         struct require

         {
             string comment;
             string metaType; // type enum command see above 
variable tag type
             string name;
         }
         require[] requires;
     }
     struct extension
     {
         string name, number, supported;
         struct require
         {
             struct elem
             {
                 string name, content;
                 string metaType; //variable tag type
                 string offset, extends;
             }
         }
     }
     extension[] extensions;
}

where "string content" is the string between the tags (does this 
have a name?) and thing with  Tag are always in the tag (although 
it is also inconsistent). I tried looking at the docs for std.xml 
but they were not very helpful.

To further complicate things not all of the fields are present in 
all cases also I need to keep track of the number of '*' (see 
fields named indirections) after a tag has closed  for pointer 
type declarations in function signatures and structs.

Also is there a way to do sane cross referencing without having 
internal pointers everywhere.

I'm sure there should be a very elegant way to do this by 
recursing down through the sub-structs

Many thanks
Nic
Feb 21 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Sunday, 21 February 2016 at 12:52:33 UTC, Nicholas Wilson 
wrote:
 So I was going through the vulcan spec to try to create a 
 better D bindings for it. (pointer /len pairs to arrays 
 adhering to D naming conventions and prettying up the *Create 
 functions functions like vkResult *Create( arg ,&arg, retptr) 
 to a fancy throw on misuse struct with constructors and that 
 kind of stuff.)

  the structure of the registry is as follows.


 struct KHR_registry
 {
 <snip>
 }
I'm glad to see more people looking to create a D binding from vk.xml! I was also working on this (http://forum.dlang.org/post/ygylvtuwwiwyqtcnlejh forum.dlang.org), unfortunately I won't be able to continue my work until early March, so I hope you'll do a good job in the mean time ;) Unfortunately the spec is written in a very C specific way so we need to have a bunch of special cases for parsing the C code that's interleaved with the XML tags. I haven't used std.xml yet, but I hope I may be able to help you.
 where "string content" is the string between the tags (does 
 this have a name?) and thing with  Tag are always in the tag 
 (although it is also inconsistent). I tried looking at the docs 
 for std.xml but they were not very helpful.
May be you can use http://dlang.org/phobos/std_xml#.Element.text in combination with http://dlang.org/phobos/std_xml#.Element.elements
 To further complicate things not all of the fields are present 
 in all cases also I need to keep track of the number of '*' 
 (see fields named indirections) after a tag has closed  for 
 pointer type declarations in function signatures and structs.
You mean things like: <param><type>void</type>** <name>ppData</name></param> (from the vkMapMemory command) ? Yeah this is extremely ugly. One way to do it is to go to the "param" XML element and check if the size of it's elements matches it's text size. In the case above everything between <param> and </param> is 39 characters and the two tags are 17 and 19 character respectively (total 36) which leaves 3 chars for the indirections. If we make a pass to remove all whitespace between tags (between ">" and "<") we should get exactly 2 characters which is the number of indirection we're looking for. Another way is to just strip the tags and leave only their internal text. In the above example the result should be: <param>void** ppData</param> which is valid D code and we can leave it that way for now.
 Also is there a way to do sane cross referencing without having 
 internal pointers everywhere.
I am no XML expert (so there may be an easier way), but I think it would be easier if we use associative arrays, instead of plain arrays. That way we can check for example: if ("VkBuffer" in types && "VkDevice" in types) types["VkBuffer"].parent = types["VkDevice"];
 I'm sure there should be a very elegant way to do this by 
 recursing down through the sub-structs
The simplest solution, IMO, is to insert all the types, commands, enums, etc. in associative arrays first and then do cross-referencing. E.g. Type[string] types; struct Type { // ... other members // Set to string first, and later change // to the proper Type, after the whole `types` table has been read. Algebraic(string, Type*) parent; // ... more members } Otherwise the only sane way to do forward reference resolution is to use D Fibers. I have used fibers in some of my projects, but the extra complications are not worth it for this task as the xml is only 5100 lines and the performance benefits probably would not be very large.
 Many thanks
 Nic
BTW if you run into some other issues with std.xml, you can also check http://arsdnet.net/web.d/dom.html and https://github.com/burner/std.xml2 Also don't forget to use unittests extensively!
Feb 21 2016
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 21 February 2016 at 15:18:44 UTC, ZombineDev wrote:
 On Sunday, 21 February 2016 at 12:52:33 UTC, Nicholas Wilson 
 wrote:
 So I was going through the vulcan spec to try to create a 
 better D bindings for it. (pointer /len pairs to arrays 
 adhering to D naming conventions and prettying up the *Create 
 functions functions like vkResult *Create( arg ,&arg, retptr) 
 to a fancy throw on misuse struct with constructors and that 
 kind of stuff.)

  the structure of the registry is as follows.


 struct KHR_registry
 {
 <snip>
 }
I'm glad to see more people looking to create a D binding from vk.xml! I was also working on this (http://forum.dlang.org/post/ygylvtuwwiwyqtcnlejh forum.dlang.org), unfortunately I won't be able to continue my work until early March, so I hope you'll do a good job in the mean time ;) Unfortunately the spec is written in a very C specific way so we need to have a bunch of special cases for parsing the C code that's interleaved with the XML tags. I haven't used std.xml yet, but I hope I may be able to help you.
 where "string content" is the string between the tags (does 
 this have a name?) and thing with  Tag are always in the tag 
 (although it is also inconsistent). I tried looking at the 
 docs for std.xml but they were not very helpful.
May be you can use http://dlang.org/phobos/std_xml#.Element.text in combination with http://dlang.org/phobos/std_xml#.Element.elements
 To further complicate things not all of the fields are present 
 in all cases also I need to keep track of the number of '*' 
 (see fields named indirections) after a tag has closed  for 
 pointer type declarations in function signatures and structs.
You mean things like: <param><type>void</type>** <name>ppData</name></param> (from the vkMapMemory command) ? Yeah this is extremely ugly. One way to do it is to go to the "param" XML element and check if the size of it's elements matches it's text size. In the case above everything between <param> and </param> is 39 characters and the two tags are 17 and 19 character respectively (total 36) which leaves 3 chars for the indirections. If we make a pass to remove all whitespace between tags (between ">" and "<") we should get exactly 2 characters which is the number of indirection we're looking for. Another way is to just strip the tags and leave only their internal text. In the above example the result should be: <param>void** ppData</param> which is valid D code and we can leave it that way for now.
 Also is there a way to do sane cross referencing without 
 having internal pointers everywhere.
I am no XML expert (so there may be an easier way), but I think it would be easier if we use associative arrays, instead of plain arrays. That way we can check for example: if ("VkBuffer" in types && "VkDevice" in types) types["VkBuffer"].parent = types["VkDevice"];
 I'm sure there should be a very elegant way to do this by 
 recursing down through the sub-structs
The simplest solution, IMO, is to insert all the types, commands, enums, etc. in associative arrays first and then do cross-referencing. E.g. Type[string] types; struct Type { // ... other members // Set to string first, and later change // to the proper Type, after the whole `types` table has been read. Algebraic(string, Type*) parent; // ... more members } Otherwise the only sane way to do forward reference resolution is to use D Fibers. I have used fibers in some of my projects, but the extra complications are not worth it for this task as the xml is only 5100 lines and the performance benefits probably would not be very large.
 Many thanks
 Nic
BTW if you run into some other issues with std.xml, you can also check http://arsdnet.net/web.d/dom.html and https://github.com/burner/std.xml2 Also don't forget to use unittests extensively!
Thanks for the tips. I used AA and just got it to compile! :) :| :( but fails to link. Undefined symbols for architecture x86_64: "_D28TypeInfo_xC4arsd3dom7Element6__initZ", referenced from: _D29TypeInfo_AxC4arsd3dom7Element6__initZ in app.o "_D4arsd3dom12__ModuleInfoZ", referenced from: _D3app12__ModuleInfoZ in app.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) --- errorlevel 1 any Ideas? structure is app.d app.o arsd/ dom.d dom.o ------------------- CODE ------------------- struct KHR_registry { string comment; // Not as xml comment but a <comment> ... </comment> struct vendorid { string name; string id; string comment; } vendorid[] vendorids; struct tag { string name; string author; string contact; } tag[] tags; struct validity_type // does not appear at this level but is common to commands and types { string[] str; } struct type { string name; // inconsistanly in tag and content string requires; string parent; // these are different things string parentName; // ditto string content; // <type ... > content </type> string category; bool returnedonly; validity_type validity; struct member { string typename; //name of the members type variable tag type string name; //name of the member variable string len; //companion member length variable for array pointers bool optional; //should generate as a default value bool noautovalidity; string enum_; } member[string] members; } type[string] types; struct enums { string name,comment, type, expand; // expand is the C enum's Prefix e.g. VK_QUERY_TYPE struct enum_ { string value, name, comment; } enum_[string] values; } enums[string] enumerations; struct command { string successcodes; string errorcodes; string returnType; string name; string content; struct param { string content; bool optional; string externsync; string len; size_t inderections; immutable invalidLen = "null-terminated"; // if len is this then it is not an array // although this is inconsistant. sometimes as a len // others in the validity } param[] params; validity_type validity; string queues, renderpass, cmdbufferlevel; } command[string] commands; struct feature { string api, name, number; /* <require comment="API version"> <type name="VK_API_VERSION"/> <enum name="VK_VERSION_MAJOR"/> <command name="VK_VERSION_MINOR"/> <type name="VK_VERSION_PATCH"/> </require>*/ struct require { string comment; string metaType; // type enum command see above variable tag type string name; } require requires; } struct extension { string name, number, supported; struct require { struct elem { string name, content; string metaType; string offset, extends; } } } extension[] extensions; } KHR_registry regFromArsdDocument(Document doc) { KHR_registry reg; size_t i; reg.comment = doc.getElementsByTagName("comment")[0].innerText(); // only one comment reg.vendorids = doc.getElementsByTagName("vendorids")[0] .getElementsByTagName("vendorid") .map!(elem => { KHR_registry.vendorid vid; vid.name = elem.getAttribute("name"); vid.id = elem.getAttribute("id"); vid.comment = elem.getAttribute("comment"); return vid; })() .array[].map!(a => a()).array; reg.tags = doc.getElementsByTagName("tags")[0] .getElementsByTagName("tag") .map!(elem => { KHR_registry.tag t; t.name = elem.getAttribute("name"); t.author = elem.getAttribute("author"); t.contact = elem.getAttribute("contact"); return t; }) .array.map!(a => a()).array[]; /*reg.types =*/ doc.getElementsByTagName("types")[0] //associatve array assignment: can we do this as a range? .getElementsByTagName("type") .map!(elem => { KHR_registry.type t; t.name = elem.getAttribute("name"); // for root types only if (t.name == null) { t.name = elem.getElementsByTagName("name")[0].innerText; t.parentName = elem.getElementsByTagName("type")[0].innerText; } t.requires = elem.getAttribute("requires"); t.parent = elem.getAttribute("parent"); t.content = elem.innerText; t.category = elem.getAttribute("catagory"); t.returnedonly = elem.getAttribute("returnedonly") != null; // if it is present it is true if (elem.getElementsByTagName("validity")) { t.validity.str = elem.getElementsByTagName("validity")[0] .getElementsByTagName("usage") .map!( elem2 => elem2.innerText).array[]; } if (t.category == "funcpointer") { t.parentName = null; // first <type> ... </type> is not actuall the parent type // we dont use the types in the signature because we use the inner text. } if (t.category == "struct") { // TODO: preserve the XML comments /*t.members = */ elem.getElementsByTagName("name") .map!(elem2 => { KHR_registry.type.member m; string name = elem2.getElementsByTagName("name")[0].innerText; m.name = name; m.typename = elem2.getElementsByTagName("type")[0].innerText; m.len = elem2.getAttribute("len"); m.enum_ = elem2.getAttribute("enum"); // for c-style array decls // dont store text its the //wrong way round for D fix it //later m.optional = elem2.getAttribute("optional") != null; // if it is present it is true m.noautovalidity = elem2.getAttribute("noautovalidity") != null; t.members[name] = m; }); } reg.types[t.name] = t; }); foreach(elem; doc.getElementsByTagName("enums")) // there are multiple <enums> elements { KHR_registry.enums e; e.name = elem.getAttribute("name"); e.comment = elem.getAttribute("comment"); e.expand = elem.getAttribute("expand"); foreach(elem2; elem.getElementsByTagName("enum")) { KHR_registry.enums.enum_ e2; e2.name = elem.getAttribute("name"); e2.comment = elem.getAttribute("comment"); e2.value = elem.getAttribute("value"); e.values[e2.name] = e2; } reg.enumerations[e.name] = e; } doc.getElementsByTagName("commands")[0] //associatve array assignment: can we do this as a range? .getElementsByTagName("command") .map!(elem => { KHR_registry.command c; c.successcodes = elem.getAttribute("successcodes"); c.errorcodes = elem.getAttribute("errorcodes"); c.queues = elem.getAttribute("queues"); c.renderpass = elem.getAttribute("renderpass"); c.cmdbufferlevel = elem.getAttribute("cmdbufferlevel"); c.content = elem.innerText; c.name = elem.getElementsByTagName("proto")[0].getElementsByTagName("name")[0].innerText; c.returnType = elem.getElementsByTagName("proto")[0].getElementsByTagName("type")[0].innerText; if (elem.getElementsByTagName("validity")) { c.validity.str = elem.getElementsByTagName("validity")[0] .getElementsByTagName("usage") .map!( elem2 => elem2.innerText).array[]; } foreach(elem2; elem.getElementsByTagName("param")) { KHR_registry.command.param p; p.optional = elem2.getAttribute("optional") != null; // present = true p.externsync = elem2.getAttribute("externsync"); p.len = elem2.getAttribute("len"); c.params ~= p; } }); return reg; }
Feb 21 2016
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 22 February 2016 at 07:00:42 UTC, Nicholas Wilson 
wrote:
 On Sunday, 21 February 2016 at 15:18:44 UTC, ZombineDev wrote:
 [...]
Thanks for the tips. I used AA and just got it to compile! :) :| :( but fails to link. Undefined symbols for architecture x86_64: "_D28TypeInfo_xC4arsd3dom7Element6__initZ", referenced from: _D29TypeInfo_AxC4arsd3dom7Element6__initZ in app.o "_D4arsd3dom12__ModuleInfoZ", referenced from: _D3app12__ModuleInfoZ in app.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) --- errorlevel 1 any Ideas? [...]
It was because i was using dmd directly as opposed to dub. Derp!
Feb 22 2016
prev sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 21 February 2016 at 15:18:44 UTC, ZombineDev wrote:
 On Sunday, 21 February 2016 at 12:52:33 UTC, Nicholas Wilson 
 wrote:
[...]
I'm glad to see more people looking to create a D binding from vk.xml! I was also working on this (http://forum.dlang.org/post/ygylvtuwwiwyqtcnlejh forum.dlang.org), unfortunately I won't be able to continue my work until early March, so I hope you'll do a good job in the mean time ;) [...]
AA's are nice in theory but the non-deterministic nature of their order of iteration is painful...
Feb 23 2016
parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 24 February 2016 at 00:50:40 UTC, Nicholas Wilson 
wrote:

 AA's are nice in theory but the non-deterministic nature of 
 their order of iteration is painful...
An ordered map as the default AA implementation would be worse. Most use cases for a hash map don't need ordering. Perhaps we'll have one in std.container at some point, but as it stands I'm unaware of any implementations out there. Neither the EMSI containers [1] nor dcollections [2] has one that I can see. [1] https://github.com/economicmodeling/containers/tree/master/src/containers [2] https://github.com/schveiguy/dcollections/tree/master/dcollections
Feb 23 2016
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/23/16 10:42 PM, Mike Parker wrote:
 On Wednesday, 24 February 2016 at 00:50:40 UTC, Nicholas Wilson wrote:

 AA's are nice in theory but the non-deterministic nature of their
 order of iteration is painful...
An ordered map as the default AA implementation would be worse. Most use cases for a hash map don't need ordering. Perhaps we'll have one in std.container at some point, but as it stands I'm unaware of any implementations out there. Neither the EMSI containers [1] nor dcollections [2] has one that I can see. [1] https://github.com/economicmodeling/containers/tree/master/src/containers [2] https://github.com/schveiguy/dcollections/tree/master/dcollections
I've contemplated adding one. I think it's a nice feature of php. But note that red black trees are ordered if you need some sort of ordering independent of insertion order. -Steve
Feb 24 2016