www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DDoc and logically structured HTML

reply Stewart Gordon <smjg_1998 yahoo.com> writes:
The default macros built into DDoc leave something to be desired in terms of
HTML 
structure, for example:

- presentational <font>, <big> and <small> tags, rather than CSS equivalents
- presentational details coded directly into the macros called by DDoc, rather
than 
delegated to CSS classes
- <br><br> to separate paragraphs, rather than <p></p> to enclose them.

By redefining a lot of the macros in a .ddoc file, one can do better.  However,
there 
remain a number of limitations:

(a) No way to make a paragraph of the DDoc source a paragraph in the generated
HTML unless 
it's specified to be something else, such as a list or a table.  While one
could wrap the 
actual paragraphs in $(P ... ) and define DDOC_BLANKLINE to be empty, this
undermines the 
first two of DDoc's goals (looks good as embedded documentation, and is easy
and natural 
to write).

(b) No macro to be seen that corresponds to the whole block of named sections. 
So I can't 
(for example) wrap them all in a <dl>.  Meanwhile I've made do with a <dl>
containing a 
single <dt> and <dd> for each named section.

(c) No way to user-define sections - you can change the markup generated for
the standard 
sections, but for everything else there's just DDOC_SECTION and DDOC_SECTION_H,
precluding 
any distinctive highlighting of certain sections.

There are probably others I thought of earlier, but I can't think what they are
at the 
moment.  But as you see, DDoc falls short of what ought to be another of its
goals: to 
support the generation of logically structured output.

Has anybody else got further with getting logical HTML out of DDoc?  Have you
been using 
DDoc to generate some internal format and then some other program to turn it
into HTML, or 
what?


But some improvements to DDoc could certainly make things better.

Dealing with (a) admittedly isn't as simple as it may seem at first.  While you
could have 
a DDOC_PARAGRAPH macro for this, there are subtleties in determining what is
and isn't a 
paragraph.

(b) is straightforward - just create and use a DDOC_NAMED_SECTIONS macro.

As for (c), well, maybe we could have it so that

Custom_Section:
     blah blah blah

first looks for a macro called DDOC_Custom_Section and, if it's present, uses
it.  Or if 
no such macro is present, just use DDOC_SECTION.


I don't know how far we can go short of a doclet system like JavaDoc has.  Nor
even how 
exactly a doclet system would work under a language that compiles to native
code....

Stewart.
Mar 10 2012
next sibling parent reply Kevin Cox <kevincox.ca gmail.com> writes:
--00151758f8beb420ba04baee59af
Content-Type: text/plain; charset=UTF-8

On Mar 10, 2012 8:10 PM, "Stewart Gordon" <smjg_1998 yahoo.com> wrote>
 I don't know how far we can go short of a doclet system like JavaDoc has.

compiles to native code....
 Stewart.

I hate to say it but I think the ddoc system was way overthought. It would be better to use a markup language such as Creole that is meant to look nice and you can convert that into anything you want. You could leave the default html but provide an api for accessing a parsed tree. --00151758f8beb420ba04baee59af Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p><br> On Mar 10, 2012 8:10 PM, &quot;Stewart Gordon&quot; &lt;<a href=3D"mailto:s= mjg_1998 yahoo.com">smjg_1998 yahoo.com</a>&gt; wrote&gt; <br> &gt; I don&#39;t know how far we can go short of a doclet system like JavaD= oc has. =C2=A0Nor even how exactly a doclet system would work under a langu= age that compiles to native code....<br> &gt;<br> &gt; Stewart.</p> <p>I hate to say it but I think the ddoc system was way overthought.=C2=A0 = It would be better to use a markup language such as Creole that is meant to= look nice and you can convert that into anything you want.=C2=A0 You could= leave the default html but provide an api for accessing a parsed tree.<br> </p> --00151758f8beb420ba04baee59af--
Mar 10 2012
next sibling parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
On 11/03/2012 02:26, Kevin Cox wrote:
<snip>
 I hate to say it but I think the ddoc system was way overthought.  It would be
better to
 use a markup language such as Creole that is meant to look nice and you can
convert that
 into anything you want.  You could leave the default html but provide an api
for accessing
 a parsed tree.

I think this is more or less how Java doclets work. It works because Java's bytecode and dynamic class loading provide a means to call an arbitrary method of an arbitrary compiled class that need not be part of the same app. But how are we going to do this with D? It seems we would need to either: (a) compile the doclet on the fly and pass the parse tree into it (b) have in said API a parse tree constructor that translates it from a textual representation received through stdin. Stewart.
Mar 11 2012
parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 03/11/2012 12:46 PM, Stewart Gordon wrote:
 On 11/03/2012 02:26, Kevin Cox wrote:
 <snip>
 I hate to say it but I think the ddoc system was way overthought. It
 would be better to
 use a markup language such as Creole that is meant to look nice and
 you can convert that
 into anything you want. You could leave the default html but provide
 an api for accessing
 a parsed tree.

I think this is more or less how Java doclets work. It works because Java's bytecode and dynamic class loading provide a means to call an arbitrary method of an arbitrary compiled class that need not be part of the same app. But how are we going to do this with D? It seems we would need to either: (a) compile the doclet on the fly and pass the parse tree into it (b) have in said API a parse tree constructor that translates it from a textual representation received through stdin. Stewart.

I don't understand why you related dynamic class' loading with a documentation system/syntax for D. Some hours ago you closed a very old enhancement report I submited: http://d.puremagic.com/issues/show_bug.cgi?id=3161 (in fact I had totally forgotten I implemented that :-P, so thanks for reminding it to me... thought I didn't like that you closed it as invalid) Now, I implemented it purely based on dmd's code. I didn't add anything else to get that done. And D is not dynamic or doesn't have dynamic class loading. So... can you explain a bit more? And does somebody know why, being dmd able to compile files, which requires having exact type information for every method/template,whatever, generates such a poor documentation? Another question: ddoc is used (or was used) to generate d's main site. Wut?? So it's much more than just documenting D classes. I think this is wrong. That's when you end up having macros and all of that, because when generating a page you want to reuse things. But when documentation a library, hmmm... I never needed such thing. And even though I do care about the format of the documentation (how it looks), I definitely don't put formatting information inside a method's documentation (except for code snippets and lists). Some simple like markdown or rdoc (well, the idea, not rdoc itself for D :-P) should be more than enough. If there's interesent, I can hack dmd into producing such documentation, and send a pull request.
Mar 11 2012
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
On 11/03/2012 19:04, Ary Manzana wrote:
<snip>
 I don't understand why you related dynamic class' loading with a documentation
 system/syntax for D.

Because that's what Javadoc relies on as a means for a program distributed in binary form to call custom code. OK, so there are native ways to do this, like DLLs and SOs. I don't really know whether these are fit for the purpose. It might get a bit complicated supporting the D object model with DMD being written in C++.
 Some hours ago you closed a very old enhancement report I submited:

 http://d.puremagic.com/issues/show_bug.cgi?id=3161

 (in fact I had totally forgotten I implemented that :-P, so thanks for
reminding it to
 me... thought I didn't like that you closed it as invalid)

 Now, I implemented it purely based on dmd's code. I didn't add anything else
to get that
 done. And D is not dynamic or doesn't have dynamic class loading.

OK, so writing a new documentation generator based on the DMD front end code is another approach. Did you just write it in C++, or do the extra work so that you could write it in D? Either way, it's good that you've written a D documentation generator that's better than standard Ddoc, and no doubt handles D much better than Doxygen does. Have you released it anywhere?
 So... can you explain a bit more?

 And does somebody know why, being dmd able to compile files, which requires
having exact
 type information for every method/template,whatever, generates such a poor
documentation?

I guess because not that much work has gone into Ddoc so far.
 Another question: ddoc is used (or was used) to generate d's main site. Wut??
So it's much
 more than just documenting D classes. I think this is wrong. That's when you
end up having
 macros and all of that, because when generating a page you want to reuse
things. But when
 documentation a library, hmmm... I never needed such thing. And even though I
do care
 about the format of the documentation (how it looks), I definitely don't put
formatting
 information inside a method's documentation (except for code snippets and
lists).

By "formatting information" are you talking about semantic/logical markup or presentational/physical markup? http://webtips.dan.info/logical.html talks about the differences between them - written from an HTML point of view but I suppose equally applicable to Ddoc macros.
 Some simple like markdown or rdoc (well, the idea, not rdoc itself for D :-P)
should be
 more than enough.

 If there's interesent, I can hack dmd into producing such documentation, and
send a pull
 request.

To replace the current Ddoc comment format, or to give the user the choice between that and whatever else? Replacing the current format would be a breaking change. Having two formats built into DMD would seem silly, aside from the question of how you would specify which is being used in a given project. The idea of Ddoc is that it's the standard D way of doing documentation. If a compiler has built-in documentation generation, it should use either a format that is part of the language standard or some established format such as Doxygen, rather than some non-portable ad-hoc format. Which is what what you're thinking of doing would be, unless it's incorporated into the language standard. But having two documentation comment formats in the language standard is itself something that doesn't seem to make sense. It would be like the whole D language having two syntaxes. Stewart.
Mar 11 2012
parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 03/11/2012 05:33 PM, Stewart Gordon wrote:
 On 11/03/2012 19:04, Ary Manzana wrote:
 <snip>
 I don't understand why you related dynamic class' loading with a
 documentation
 system/syntax for D.

Because that's what Javadoc relies on as a means for a program distributed in binary form to call custom code.

What do you mean by "custom code"?
 OK, so there are native ways to do this, like DLLs and SOs. I don't
 really know whether these are fit for the purpose. It might get a bit
 complicated supporting the D object model with DMD being written in C++.

Sorry, I still don't understand you :-(
 Some hours ago you closed a very old enhancement report I submited:

 http://d.puremagic.com/issues/show_bug.cgi?id=3161

 (in fact I had totally forgotten I implemented that :-P, so thanks for
 reminding it to
 me... thought I didn't like that you closed it as invalid)

 Now, I implemented it purely based on dmd's code. I didn't add
 anything else to get that
 done. And D is not dynamic or doesn't have dynamic class loading.

OK, so writing a new documentation generator based on the DMD front end code is another approach. Did you just write it in C++, or do the extra work so that you could write it in D?

I wrote it in Java, because back then I had dmd ported to Java and I found it easy to do it that way (I'm not very efficient/fast programming in C++).
 Either way, it's good that you've written a D documentation generator
 that's better than standard Ddoc, and no doubt handles D much better
 than Doxygen does. Have you released it anywhere?

I posted it back then, it's in Descent's source code. There wasn't much interest, though. And instead of working on a decent documentation system you end up with documentation like this one: https://github.com/D-Programming-Language/phobos/blob/master/std/algorithm.d#L4 Nice, a javascript tag inside of the docs. And then an excerpt: $(BOOKTABLE , $(TR $(TH Category) $(TH Functions) ) $(TR $(TDNW Searching) $(TD $(MYREF balancedParens) $(MYREF ??? Ok, a table can't be done easily in markdown or rdoc either, but a list would have maybe been enough. Or just list the functions in alphabetical order, top down (because it's hard to do a binary search with the eyes from left to right, since it takes some time to spot the first letter of every word). Like finding join here: http://dlang.org/phobos/std_array.html instead of here: http://pastebin.com/ib4dJt48
 So... can you explain a bit more?

 And does somebody know why, being dmd able to compile files, which
 requires having exact
 type information for every method/template,whatever, generates such a
 poor documentation?

I guess because not that much work has gone into Ddoc so far.

Documentation is the primary interface to the users of a library, including the standard library. I believe it deserves a lot more attention than that.
 Another question: ddoc is used (or was used) to generate d's main
 site. Wut?? So it's much
 more than just documenting D classes. I think this is wrong. That's
 when you end up having
 macros and all of that, because when generating a page you want to
 reuse things. But when
 documentation a library, hmmm... I never needed such thing. And even
 though I do care
 about the format of the documentation (how it looks), I definitely
 don't put formatting
 information inside a method's documentation (except for code snippets
 and lists).

By "formatting information" are you talking about semantic/logical markup or presentational/physical markup? http://webtips.dan.info/logical.html talks about the differences between them - written from an HTML point of view but I suppose equally applicable to Ddoc macros.

Interesting reading. I talk about semantic/logical markup. But I'd like to use one which, when I read it, doesn't hurt my eyes (like std.algorithm docs).
 Some simple like markdown or rdoc (well, the idea, not rdoc itself for
 D :-P) should be
 more than enough.

 If there's interesent, I can hack dmd into producing such
 documentation, and send a pull
 request.

To replace the current Ddoc comment format, or to give the user the choice between that and whatever else?

Replace it. Macros to document code? I think that's a bit too much... Write a list in DDoc: $(UL $(LI Karatsuba multiplication) $(LI Squaring is optimized independently of multiplication) $(LI Divide-and-conquer division) $(LI Binary exponentiation) ) Write a list in rdoc or markdown: * Karatsuba multiplication * Squaring is optimized independently of multiplication * Divide-and-conquer division * Binary exponentiation Or links. With DDoc: $(WEB sgi.com/tech/stl/, Alexander Stepanov's Standard Template Library) With markdown [Alexander Stepanov's Standard Template Library](sgi.com/tech/stl/) (well, I admit there's not much difference in this last example. Still... it's shorter and cleaner :-P)
 Replacing the current format would be a breaking change.

Only for docs that heavily use macros. I hope there's not many! And you can always make a tool to convert ddoc to the new format (I believe with ddoc macros?)
 Having two formats built into DMD would seem silly, aside from the
 question of how you would specify which is being used in a given
 project. The idea of Ddoc is that it's the standard D way of doing
 documentation. If a compiler has built-in documentation generation, it
 should use either a format that is part of the language standard or some
 established format such as Doxygen, rather than some non-portable ad-hoc
 format. Which is what what you're thinking of doing would be, unless
 it's incorporated into the language standard. But having two
 documentation comment formats in the language standard is itself
 something that doesn't seem to make sense. It would be like the whole D
 language having two syntaxes.

Just leave one syntax.
Mar 11 2012
parent reply Stewart Gordon <smjg_1998 yahoo.com> writes:
On 11/03/2012 20:59, Ary Manzana wrote:
 On 03/11/2012 05:33 PM, Stewart Gordon wrote:

 Because that's what Javadoc relies on as a means for a program
 distributed in binary form to call custom code.

What do you mean by "custom code"?

Code written and compiled by a user of the tool, of course. Or maybe written by a third party unconnected with the tool's creator. <snip>
 Replace it. Macros to document code? I think that's a bit too much...

The advantage of a macro system is that it's easily extensible by the user. Does your proposed system provide this? <snip>
 And you can always make a tool to convert ddoc to the new format (I believe
with ddoc
 macros?)

You mean make a tool that leaves D code unchanged, but uses the ddoc macro system to convert documentation comments to the new format? Stewart.
Mar 11 2012
parent reply Ary Manzana <ary esperanto.org.ar> writes:
On 03/11/2012 07:01 PM, Stewart Gordon wrote:
 On 11/03/2012 20:59, Ary Manzana wrote:
 On 03/11/2012 05:33 PM, Stewart Gordon wrote:

 Because that's what Javadoc relies on as a means for a program
 distributed in binary form to call custom code.

What do you mean by "custom code"?

Code written and compiled by a user of the tool, of course. Or maybe written by a third party unconnected with the tool's creator. <snip>
 Replace it. Macros to document code? I think that's a bit too much...

The advantage of a macro system is that it's easily extensible by the user. Does your proposed system provide this?

No, but I never needed it. Nor I know anyone that needed it (in Java, C#, Ruby, Pyhton and PHP). Give me an example of a macro you'd use in your documentation, that's not: bold, italic, underlined, lists, tables, links, named sections, code blocks and images.
 <snip>
 And you can always make a tool to convert ddoc to the new format (I
 believe with ddoc
 macros?)

You mean make a tool that leaves D code unchanged, but uses the ddoc macro system to convert documentation comments to the new format?

Yes.
 Stewart.

Mar 11 2012
parent Stewart Gordon <smjg_1998 yahoo.com> writes:
On 11/03/2012 22:30, Ary Manzana wrote:
<snip>
 Give me an example of a macro you'd use in your documentation, that's not:
bold, italic,
 underlined, lists, tables, links, named sections, code blocks and images.

Emphasis possibly. Inline code snippets. Possibly sample input/output. But even if you do come up with a list of things that have a genuine use case that you think is exhaustive, someone'll sooner or later come up with another one that isn't on your list. Moreover, macros aren't just to provide access to the features of the target documentation format. They're also used to enable specific uses of these features to be written in the source more nicely. For example, I've just found myself setting one up for links to issues on Bugzilla. Stewart.
Mar 11 2012
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 11.03.2012 03:26, schrieb Kevin Cox:
 On Mar 10, 2012 8:10 PM, "Stewart Gordon" <smjg_1998 yahoo.com
 <mailto:smjg_1998 yahoo.com>> wrote>
  > I don't know how far we can go short of a doclet system like JavaDoc
 has.  Nor even how exactly a doclet system would work under a language
 that compiles to native code....
  >
  > Stewart.

 I hate to say it but I think the ddoc system was way overthought.  It
 would be better to use a markup language such as Creole that is meant to
 look nice and you can convert that into anything you want.  You could
 leave the default html but provide an api for accessing a parsed tree.

This is a bit what Go does with the go packages (ast, doc, etc) http://tip.golang.org/pkg/go/ -- Paulo
Mar 11 2012
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 03/11/2012 03:26 AM, Kevin Cox wrote:
 On Mar 10, 2012 8:10 PM, "Stewart Gordon" <smjg_1998 yahoo.com
 <mailto:smjg_1998 yahoo.com>> wrote>
  > I don't know how far we can go short of a doclet system like JavaDoc
 has.  Nor even how exactly a doclet system would work under a language
 that compiles to native code....
  >
  > Stewart.

 I hate to say it but I think the ddoc system was way overthought.  It
 would be better to use a markup language such as Creole that is meant to
 look nice and you can convert that into anything you want.  You could
 leave the default html but provide an api for accessing a parsed tree.

Afaik you can use ddoc together with any markup language you like, as long as it does not clash with the macro syntax.
Mar 11 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 11 March 2012 at 20:59:23 UTC, Ary Manzana wrote:
 Write a list in DDoc:

Better: $(LIST Karatsuba multiplication, Squaring is optimized independently of multiplication, Divide-and-conquer division, Binary exponentiation) Many of the default ddoc macros suck. They are hard to use and generate bad html. The ddoc system can do a LOT better.
Mar 11 2012
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 3/10/2012 5:08 PM, Stewart Gordon wrote:
 The default macros built into DDoc leave something to be desired in terms of
 HTML structure, for example:

 - presentational <font>, <big> and <small> tags, rather than CSS equivalents
 - presentational details coded directly into the macros called by DDoc, rather
 than delegated to CSS classes
 - <br><br> to separate paragraphs, rather than <p></p> to enclose them.

The idea behind this is simple - a naive use of it should generate content that is renderable. Requiring the user to provide a css file thwarts that goal.
Mar 11 2012
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 12 March 2012 at 02:03:03 UTC, Walter Bright wrote:
 The idea behind this is simple - a naive use of it should 
 generate content that is renderable. Requiring the user to 
 provide a css file thwarts that goal.

Simple solution: include some embedded css in the default DDOC macro as <style> tags. Then you have something that just works out of the box, and continues to work as you tweak it into something better.
Mar 11 2012
prev sibling next sibling parent reply Gour <gour atmarama.net> writes:
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Sun, 11 Mar 2012 01:08:20 +0000
Stewart Gordon <smjg_1998 yahoo.com> wrote:

 Has anybody else got further with getting logical HTML out of DDoc?
 Have you been using DDoc to generate some internal format and then
 some other program to turn it into HTML, or what?

Excuse me for jumping into the thread, but I hope it's relevant... I wonder whether you can recommend using Ddoc for writing general-purpose documentation like user's manual for e.g. application written in D which has to be split into logical parts (chapters, sections,...) as well as use hyperlinking, addding pictures, tables etc.? Or do you believe it's better to use more specialized tool like reST markup with Sphinx, AsciiDoc, LaTeX/LyX...? Let me add that we'd like to have nice HTML output which can be invoked within application itself as well as quaility output in the form of PDF file. Sincerely, Gour --=20 One who is not connected with the Supreme can have neither=20 transcendental intelligence nor a steady mind, without which=20 there is no possibility of peace. And how can there be any=20 happiness without peace? http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810
Mar 19 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/19/12 5:01 AM, Gour wrote:
 On Sun, 11 Mar 2012 01:08:20 +0000
 Stewart Gordon<smjg_1998 yahoo.com>  wrote:

 Has anybody else got further with getting logical HTML out of DDoc?
 Have you been using DDoc to generate some internal format and then
 some other program to turn it into HTML, or what?

Excuse me for jumping into the thread, but I hope it's relevant... I wonder whether you can recommend using Ddoc for writing general-purpose documentation like user's manual for e.g. application written in D which has to be split into logical parts (chapters, sections,...) as well as use hyperlinking, addding pictures, tables etc.? Or do you believe it's better to use more specialized tool like reST markup with Sphinx, AsciiDoc, LaTeX/LyX...? Let me add that we'd like to have nice HTML output which can be invoked within application itself as well as quaility output in the form of PDF file.

DDoc has two main assets: (a) it understands D structures, e.g. you can use it to document D APIs, and (b) it highlights D code nicely. Aside from that, it's a fairly simple macro system. If you don't have a need for (a)/(b), it is a passable document generation tool (pretty much as e.g. m4 would be). We use it for D's own website, and it's okay in helping with HTML generation. More specialized tools that have a focus on document formatting have naturally an advantage over DDoc for general-purpose documentation. Andrei
Mar 19 2012
prev sibling parent Gour <gour atmarama.net> writes:
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

On Mon, 19 Mar 2012 09:55:40 -0500
Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 DDoc has two main assets: (a) it understands D structures, e.g. you
 can use it to document D APIs, and (b) it highlights D code nicely.

Yeah, that's anticpated usage of it.
 More specialized tools that have a focus on document formatting have=20
 naturally an advantage over DDoc for general-purpose documentation.

After some research, it seems that the best is to use Ddoc for documenting our code and library, while leaving the rest to the specialized tool (we're heading towards AsciiDoc). Sincerely, Gour --=20 What is night for all beings is the time of awakening=20 for the self-controlled; and the time of awakening for=20 all beings is night for the introspective sage. http://atmarama.net | Hlapicina (Croatia) | GPG: 52B5C810
Mar 19 2012