www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - json and ddoc

reply "Adam D. Ruppe" <destructionator gmail.com> writes:
I'm (finally) updating my dpldocs.info website again, and
before I just pulled names from the generated Phobos html.

This time, I want to use dmd's json output to avoid dependency
on the specific html layout of std.ddoc.


Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
one big .json file with all the info.

But, the comments aren't parsed at all.



So, my question is: do we have a ddoc parser that stands alone?


BTW: this is the complete listing from json so far:
http://dpldocs.info/search/awesome

I'm actually impressed with the speed, considering it is 
re-parsing
3 MB of generated json per load. std.variant and std.json might 
not
be speed demons, but they are plenty good enough for me.

The ultimate goal here is to revitalize my D search engine and to
make it as a nice index for all functions, with brief 
descriptions,
etc., hopefully for all D libs everywhere.

With the html, dealing with different people's macros to parse it
is a pain. But with the json, dmd always does it the same way, so
with some luck, I can just add a box "upload your project's json
file" and allow easy instant submissions for other libs.
Aug 01 2012
next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Aug 1, 2012 at 4:07 PM, Adam D. Ruppe <destructionator gmail.com> wrote:
 I'm (finally) updating my dpldocs.info website again, and

Yes!
 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.

Well, the comments are discarded as soon as lexical analysis is done, I guess. I suppose the DMD part that does the json output never ever sees any comment, just code tokens (see the discussion on lexing D)? I wonder how documentation generation is done with -D? Since with the -D flag, DMD correctly 'attach' doc comments on code part, (this is a modul-level doc, this is a field documentation, ...) I guess the problem is to somehow transport this information to the json generator. This generator does not give much information btw, but my impression is that people do not use it that much and, consequently, no feature request is emitted.
Aug 01 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud 
wrote:
 Well, the comments are discarded as soon as lexical analysis is 
 done, I guess.

They are actually there if you do both -X and -D together. (-X alone doesn't show it; it must be discarded then, but -D keeps it). The problem is though that it is the text from the comment without any organization or macro expansion. For example: { "name" : "map(fun...) if (fun.length >= 1)", "kind" : "template", "protection" : "public", "comment" : "\nImplements the homonym function (also known as $(D transform)) present\nin many languages of functional flavor. The call $(D map!(fun)(range))\nreturns a range of which elements are obtained by applying $(D fun(x))\nleft to right for all $(D x) in $(D range). The original ranges are\nnot changed. Evaluation is done lazily. The range returned by $(D map)\ncaches the last value such that evaluating $(D front) multiple times\ndoes not result in multiple calls to $(D fun).\n\nExample:\n----\nint[] arr1 = [ 1, 2, 3, 4 ];\nint[] arr2 = [ 5, 6 ];\nauto squares = map!(\"a * a\")(chain(arr1, arr2));\nassert(equal(squares, [ 1, 4, 9, 16, 25, 36 ]));\n----\n\nMultiple functions can be passed to $(D map). In that case, the\nelement type of $(D map) is a tuple containing one element for each\nfunction.\n\nExample:\n\n----\nauto arr1 = [ 1, 2, 3, 4 ];\nforeach (e; map!(\"a + a\", \"a * a\")(arr1))\n{\n writeln(e[0], \" \", e[1]);\n}\n----\n\nYou may alias $(D map) with some function(s) to a symbol and use\nit separately:\n\n----\nalias map!(to!string) stringize; \nassert(equal(stringize([ 1, 2, 3, 4 ]), [ \"1\", \"2\", \"3\", \"4\" ]));\n----\n", "line" : 367, I'd like to get the sections broken out and as many macros expanded as possible, so the summary text in search results can look pretty. One option is doing my own parsing, but Walter said the recursive expansion in the macros was actually very tricky to get right.. so I'd prefer to skip doing that again myself if I can.
 This generator does not give much information btw, but
 my impression is that people do not use it that much and,
 consequently, no feature request is emitted.

Yeah. It isn't bad though; more detailed than dpldocs.info version 1.0 which did nothing more than grep '<a name' in the generated html!
Aug 01 2012
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.08.2012 16:07, schrieb Adam D. Ruppe:
 I'm (finally) updating my dpldocs.info website again, and
 before I just pulled names from the generated Phobos html.

 This time, I want to use dmd's json output to avoid dependency
 on the specific html layout of std.ddoc.


 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.



 So, my question is: do we have a ddoc parser that stands alone?


 BTW: this is the complete listing from json so far:
 http://dpldocs.info/search/awesome

 I'm actually impressed with the speed, considering it is re-parsing
 3 MB of generated json per load. std.variant and std.json might not
 be speed demons, but they are plenty good enough for me.

 The ultimate goal here is to revitalize my D search engine and to
 make it as a nice index for all functions, with brief descriptions,
 etc., hopefully for all D libs everywhere.

 With the html, dealing with different people's macros to parse it
 is a pain. But with the json, dmd always does it the same way, so
 with some luck, I can just add a box "upload your project's json
 file" and allow easy instant submissions for other libs.

I have a simple DDOC parser in vibe.d (*). No guarantees that the output is always acting the same as DMD (e.g. there is no support for special sections yet), but it was good enough for me so far. Something like this should work: import vibe.vibe; void main(string[] args) { auto fin = openFile(args[1], FileMode.Read); auto ddoc = cast(string)fin.readAll(); auto html = appender!string(); filterDdocComment(html, ddoc); auto fout = openFile(args[2], FileMode.CreateTrunc); fout.write(formatDdocComment(html.data)); } It doesn't really have dependencies to vibe.d so ripping it out should also be easy. (*) http://vibed.org/api/vibe.textfilter.ddoc#filterDdocComment ...that module is not really documented atm
Aug 01 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:
 It doesn't really have dependencies to vibe.d so ripping it out 
 should also be easy.

Thanks! It looks like that will do the job.
Aug 01 2012
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 8/1/2012 8:20 AM, Adam D. Ruppe wrote:
 On Wednesday, 1 August 2012 at 15:09:45 UTC, Sönke Ludwig wrote:
 It doesn't really have dependencies to vibe.d so ripping it out should also be
easy.

Thanks! It looks like that will do the job.

I'd almost pay money for the ddoc support that's inside dmd to be pulled into a separate process that just uses the json output. I suspect it wouldn't be that hard to separate them, just never have had the time to try it.
Aug 01 2012
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Aug 1, 2012 at 4:46 PM, Adam D. Ruppe <destructionator gmail.com> wrote:
 On Wednesday, 1 August 2012 at 14:36:26 UTC, Philippe Sigaud wrote:
 Well, the comments are discarded as soon as lexical analysis is done, I
 guess.

They are actually there if you do both -X and -D together. (-X alone doesn't show it; it must be discarded then, but -D keeps it).

Ah, dammit. I was at work far from a D compiler. I never used -X *and* -D at the same time.
 The problem is though that it is the text from the comment
 without any organization or macro expansion.

OK.
 This generator does not give much information btw, but
 my impression is that people do not use it that much and,
 consequently, no feature request is emitted.

Yeah. It isn't bad though; more detailed than dpldocs.info version 1.0 which did nothing more than grep '<a name' in the generated html!

No no no, I wasn't talking about your app! dpldocs is cool! I was talking about the json generator in DMD, which, to my eyes, does not give enough information yet to be truly useful.
Aug 01 2012
prev sibling next sibling parent Brad Anderson <eco gnuk.net> writes:
--047d7b60399ce3e47404c637f2b0
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Aug 1, 2012 at 8:07 AM, Adam D. Ruppe <destructionator gmail.com>wrote:

 I'm (finally) updating my dpldocs.info website again, and
 before I just pulled names from the generated Phobos html.

 This time, I want to use dmd's json output to avoid dependency
 on the specific html layout of std.ddoc.


 Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got
 one big .json file with all the info.

 But, the comments aren't parsed at all.



 So, my question is: do we have a ddoc parser that stands alone?


 BTW: this is the complete listing from json so far:
 http://dpldocs.info/search/**awesome <http://dpldocs.info/search/awesome>

 I'm actually impressed with the speed, considering it is re-parsing
 3 MB of generated json per load. std.variant and std.json might not
 be speed demons, but they are plenty good enough for me.

 The ultimate goal here is to revitalize my D search engine and to
 make it as a nice index for all functions, with brief descriptions,
 etc., hopefully for all D libs everywhere.

 With the html, dealing with different people's macros to parse it
 is a pain. But with the json, dmd always does it the same way, so
 with some luck, I can just add a box "upload your project's json
 file" and allow easy instant submissions for other libs.

There is already a pull request to add (at least some) ddoc to json: http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv forum.dlang.org --047d7b60399ce3e47404c637f2b0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Wed, Aug 1, 2012 at 8:07 AM, Adam D. Ruppe <span dir=3D"ltr">&lt;<a href= =3D"mailto:destructionator gmail.com" target=3D"_blank">destructionator gma= il.com</a>&gt;</span> wrote:<br><div class=3D"gmail_quote"><blockquote clas= s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad= ding-left:1ex"> I&#39;m (finally) updating my <a href=3D"http://dpldocs.info" target=3D"_bl= ank">dpldocs.info</a> website again, and<br> before I just pulled names from the generated Phobos html.<br> <br> This time, I want to use dmd&#39;s json output to avoid dependency<br> on the specific html layout of std.ddoc.<br> <br> <br> Anyway I hit dmd -c -X -D *.d in the phobos dir. Boom, I got<br> one big .json file with all the info.<br> <br> But, the comments aren&#39;t parsed at all.<br> <br> <br> <br> So, my question is: do we have a ddoc parser that stands alone?<br> <br> <br> BTW: this is the complete listing from json so far:<br> <a href=3D"http://dpldocs.info/search/awesome" target=3D"_blank">http://dpl= docs.info/search/<u></u>awesome</a><br> <br> I&#39;m actually impressed with the speed, considering it is re-parsing<br> 3 MB of generated json per load. std.variant and std.json might not<br> be speed demons, but they are plenty good enough for me.<br> <br> The ultimate goal here is to revitalize my D search engine and to<br> make it as a nice index for all functions, with brief descriptions,<br> etc., hopefully for all D libs everywhere.<br> <br> With the html, dealing with different people&#39;s macros to parse it<br> is a pain. But with the json, dmd always does it the same way, so<br> with some luck, I can just add a box &quot;upload your project&#39;s json<b= r> file&quot; and allow easy instant submissions for other libs.<br> </blockquote></div><div><br></div>There is already a pull request to add (a= t least some) ddoc to json:<div><br><div><a href=3D"http://forum.dlang.org/= post/ycfyhcsbciguzaxwuppv forum.dlang.org">http://forum.dlang.org/post/ycfy= hcsbciguzaxwuppv forum.dlang.org</a> </div></div> --047d7b60399ce3e47404c637f2b0--
Aug 01 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 17:52:02 UTC, Brad Anderson wrote:
 There is already a pull request to add (at least some) ddoc to 
 json:

 http://forum.dlang.org/post/ycfyhcsbciguzaxwuppv forum.dlang.org

Well, we already have some... but that request looks like it does a lot of good stuff.
Aug 01 2012
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 August 2012 at 17:40:13 UTC, Philippe Sigaud 
wrote:
 No no no, I wasn't talking about your app! dpldocs is cool!

Thanks! I think the new version will be even better. Preview: http://dpldocs.info/search/search-2?searchTerm=post http://dpldocs.info/search/search-2?searchTerm=get I'm now extending the search to be able to look at the type names from the json and do more inter-linking. It also shows the brief description, if present, and will filter out private members for you. And, of course, it will be able to look at submitted libraries too. The little breadcrumbs I intend to make into nice indexes, giving some nice sorting that most ddoc sites don't currently do. Still a ways to go, but I think it will end up really nice.
 I was talking about the json generator in DMD, which, to my 
 eyes, does not give enough information yet to be truly useful.

Yea, but it's still better than nothing at least.
Aug 01 2012