www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Cross referencing in Ddoc

reply Jacob Carlborg <doob me.com> writes:
If nothing has happened recently the current situation of cross 
referencing in Ddoc sucks. What's currently being used in the Phobos 
documentation is the XREF, CXREF and ECXREF ddoc macros. These macros 
take two arguments, append "std", "core" or "etc" and form a link of the 
arguments. The problem with this is that it doesn't work so good for 
referring to symbols in a deeper package hierarchy.

Example, adding a reference to std.path.buildPath works fine but symbols 
that are any deeper than that will not work very well, like 
std.digest.digest.toHexString.

$(XREF path, buildPath)

The above will generate a link to std_path.html#buildPath with the text 
"std.path.buildPath". If we want to create a link to 
std.digest.digest.toHexString we get in to some problems, XREF doesn't 
accept a variable amount of arguments, the following will not work:

$(XREF digest, digest, toHexString)

What will work is this:

$(XREF digest_digest, toHexString)

The above will properly create a link to 
std_digest_digest.html#toHexString but that relies on the fact that DMD 
replaces dots with underscores in module names when generating 
documentation. But the text of the link will be 
"std.digest_digest.toHexString" which doesn't look very pretty.

The XREF macro also hard codes the root package, making it only work for 
"std".

I think we need a better solution for creating cross referencing links 
in ddoc. Here are a couple of ideas:

A. Automatic cross reference

Automatically create links for all matching symbols in the current 
scope. That means to create a link to a symbol in the current scope it's 
enough to mention it's name in the documentation. To refer to a symbol 
in another module the fully qualified name is used. Examples:

"Read and validates (using $(XREF utf, validate)) a text file."

Would become:

"Read and validates std.utf.validate a text file."

And

"The default $(XREF _algorithm, move) performs a potentially"

Would become:

"The default move performs a potentially"

Advantages:

* Everything happens automatically, the developer doesn't have to do 
anything special to add cross referencing

Disadvantages:

* It might create links where it's not wanted
* Requires support from the compiler

B. REF macro

A special macro that the compiler knows about. Pass the symbol name 
(local or fully qualified) to the macro and the compiler will generate 
the link. Examples:

"Read and validates (using $(XREF utf, validate)) a text file."

Would become:

"Read and validates $(REF std.utf.validate) a text file."

And

"The default $(XREF _algorithm, move) performs a potentially"

Would become:

"The default $(REF move) performs a potentially"

Advantages:

* No links are created where it's not wanted

Disadvantages:

* The developer need to manually add macros, still better than today though
* Requires support from the compiler

C. A combination of A and B

By default ddoc behaves as option B, but in See_Also sections ddoc 
behaves as option A.

Advantages:

* No links are created where it's not wanted

Disadvantages:

* The developer need to manually add macros but hopefully not in as many 
places as with option B
* Requires support from the compiler

What do you think?

-- 
/Jacob Carlborg
Dec 29 2013
next sibling parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 29.12.2013 18:38, schrieb Jacob Carlborg:
 A. Automatic cross reference

This is done for the DDOX based docs that were supposed to end up on the home page at some point: http://forum.dlang.org/thread/l9poef$210u$1 digitalmars.com http://vibed.org/temp/dlang.org/library/index.html
 * It might create links where it's not wanted

The DDOC specs say that preceding an identifier with an underscore avoids it getting highlighted. DDOX currently uses the same indicator to avoid it getting a link. See the "Emphasis" section in http://dlang.org/ddoc.html
Dec 29 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-29 19:35, S÷nke Ludwig wrote:

 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:

 http://forum.dlang.org/thread/l9poef$210u$1 digitalmars.com
 http://vibed.org/temp/dlang.org/library/index.html

Right, forgot about that. When _are_ DDOX becoming the official documentation format. I really don't want to manually add cross referencing.
 The DDOC specs say that preceding an identifier with an underscore
 avoids it getting highlighted. DDOX currently uses the same indicator to
 avoid it getting a link.

 See the "Emphasis" section in http://dlang.org/ddoc.html

I see. -- /Jacob Carlborg
Dec 29 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-29 19:35, S÷nke Ludwig wrote:

 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:

BTW, although DDOX seems to be the best solution currently. It still feels like a workaround for something that should be fixed in the compiler. -- /Jacob Carlborg
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 11:24, schrieb Jacob Carlborg:
 On 2013-12-29 19:35, S÷nke Ludwig wrote:
 
 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:

BTW, although DDOX seems to be the best solution currently. It still feels like a workaround for something that should be fixed in the compiler.

Although I think that it is more important to have a well defined documentation format that can be processed by external tools, it would indeed be much nicer if for example DMD could automatically emit $(XREF ...) or similar for recognized symbol names. Another thing that would sometimes be nice is to group functions by category (even though most of the time using different modules is arguably the better choice). Using custom HTML macros is sub optimal for many reasons and some kind of little DDOC syntax similar to sections could do wonders there. All in all, it's good to have a built-in solution available (and especially important to facilitate a standard documentation format), but many things that a separate tool can do (global indexes, search, custom page structure, per declaration pages, etc.) would IMHO be overkill to have in the compiler. It's still a compiler after all.
Dec 30 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 12:39, S÷nke Ludwig wrote:

 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.

Sure, it always leads back to the same problem: have the compiler available as a library. I guess the JSON output is some kind of middle ground. -- /Jacob Carlborg
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 13:47, schrieb Jacob Carlborg:
 On 2013-12-30 12:39, S÷nke Ludwig wrote:
 
 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.

Sure, it always leads back to the same problem: have the compiler available as a library. I guess the JSON output is some kind of middle ground.

Absolutely! I hope at least the lexer will find it's way into Phobos soon. A lot of potential is effectively sleeping there.
Dec 30 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 4:55 AM, S÷nke Ludwig wrote:
 Am 30.12.2013 13:47, schrieb Jacob Carlborg:
 On 2013-12-30 12:39, S÷nke Ludwig wrote:

 Although I think that it is more important to have a well defined
 documentation format that can be processed by external tools, it would
 indeed be much nicer if for example DMD could automatically emit $(XREF
 ...) or similar for recognized symbol names. Another thing that would
 sometimes be nice is to group functions by category (even though most of
 the time using different modules is arguably the better choice). Using
 custom HTML macros is sub optimal for many reasons and some kind of
 little DDOC syntax similar to sections could do wonders there.

 All in all, it's good to have a built-in solution available (and
 especially important to facilitate a standard documentation format), but
 many things that a separate tool can do (global indexes, search, custom
 page structure, per declaration pages, etc.) would IMHO be overkill to
 have in the compiler. It's still a compiler after all.

Sure, it always leads back to the same problem: have the compiler available as a library. I guess the JSON output is some kind of middle ground.

Absolutely! I hope at least the lexer will find it's way into Phobos soon. A lot of potential is effectively sleeping there.

Yah, https://github.com/Hackerpilot/lexer-work/blob/master/lexer.d is great! More pre-review discussion? Brian Schott: would be awesome to encode a couple more lexer examples, such as HTML (recognition of all tags and character entities - there are comprehensive lists on the net) and C++? Andrei
Dec 30 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 3:39 AM, S÷nke Ludwig wrote:
 Am 30.12.2013 11:24, schrieb Jacob Carlborg:
 On 2013-12-29 19:35, S÷nke Ludwig wrote:

 This is done for the DDOX based docs that were supposed to end up on the
 home page at some point:

BTW, although DDOX seems to be the best solution currently. It still feels like a workaround for something that should be fixed in the compiler.

Although I think that it is more important to have a well defined documentation format that can be processed by external tools, it would indeed be much nicer if for example DMD could automatically emit $(XREF ...) or similar for recognized symbol names.

I'm not sure automatic cross-references are a good thing. Too many litter the document with links, and catch casual uses of words that happen to be indexable (consider e.g. "in" would be cross-referenced automatically). Andrei
Dec 30 2013
parent reply =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 16:57, schrieb Andrei Alexandrescu:
 On 12/30/13 3:39 AM, S÷nke Ludwig wrote:
 Am 30.12.2013 11:24, schrieb Jacob Carlborg:
 On 2013-12-29 19:35, S÷nke Ludwig wrote:

 This is done for the DDOX based docs that were supposed to end up on
 the
 home page at some point:

BTW, although DDOX seems to be the best solution currently. It still feels like a workaround for something that should be fixed in the compiler.

Although I think that it is more important to have a well defined documentation format that can be processed by external tools, it would indeed be much nicer if for example DMD could automatically emit $(XREF ...) or similar for recognized symbol names.

I'm not sure automatic cross-references are a good thing. Too many litter the document with links, and catch casual uses of words that happen to be indexable (consider e.g. "in" would be cross-referenced automatically). Andrei

In my experience it is far more annoying to have the source files littered with $(MACROS) instead of putting a few _underscores to avoid bogus links (not to mention that most of the time this results in no links at all due to laziness). In some modules it may happen more frequently, but in general this seems to be rare enough for not really having to worry about it - at least in the examples that I've seen up to now. Also remember that you often have to escape such words anyway, because they would otherwise still be highlighted as D code even without the cross references.
Dec 30 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 8:10 AM, S÷nke Ludwig wrote:
 In my experience it is far more annoying to have the source files
 littered with $(MACROS) instead of putting a few _underscores to avoid
 bogus links (not to mention that most of the time this results in no
 links at all due to laziness).

Oh the default is to link and then disable manually? That's still manual, just the default is different :o). I can work with that. Andrei
Dec 30 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.
Dec 30 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 22:00, schrieb Walter Bright:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

I agree the reverse would be better, but that's what is already in DDOC now. Also, quoting http://dlang.org/ddoc.html:
 Identifiers in documentation comments that are function parameters or

in the output. So the same problem is already reality - you already have to go through the documentation to see if there are any mis-highlighted words.
Dec 30 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 1:25 PM, S├Ânke Ludwig wrote:
 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.

1. having a bad idea is not justification for doing it again :-) 2. the requirement that the name be "in scope" does a pretty good job of reducing the problem. But cross references, are by definition CROSS references, i.e. they cross scopes. The whole point of them is they cross scopes, which means this becomes a much bigger issue.
Dec 30 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 23:13, schrieb Walter Bright:
 On 12/30/2013 1:25 PM, S├Ânke Ludwig wrote:
 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.

1. having a bad idea is not justification for doing it again :-)

Of course not, but my point was that the mistake has already been made, so we may as well make the best out of it... not really doing it again in that sense. But if the decision is that in fact this was a mistake, it should get removed and replaced by a proper solution sooner rather than later (it's not that bad to lose some highlighting and Phobos uses $(D ...) anyway, so just removing it wouldn't hurt much). And in that regard I'd like to point to "D's goals for embedded documentation" <http://dlang.org/ddoc.htm>. I very much agree with them, but the current reality looks like the opposite - lots of macro-pseudo-HTML markup, very difficult to read unprocessed in many places. So something like using "#" or "()" to mark symbols + automatic cross referencing for marked symbols would be a *really* nice to have.
 
 2. the requirement that the name be "in scope" does a pretty good job of
 reducing the problem. But cross references, are by definition CROSS
 references, i.e. they cross scopes. The whole point of them is they
 cross scopes, which means this becomes a much bigger issue.
 

This is how I see it: single identifiers are only searched in the current scope and in parent scopes of the same module. Cross references to foreign scopes need to use qualified names (either ClassName.member or pack.mod.Symbol). This avoids any inter-module issues and also a lot of intra-module issues. Symbols in code blocks on the other hand can be cross referenced completely automatically according to the usual D lookup rules (including imports).
Dec 30 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 3:13 PM, S├Ânke Ludwig wrote:
 Am 30.12.2013 23:13, schrieb Walter Bright:
 On 12/30/2013 1:25 PM, S├Ânke Ludwig wrote:
 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.

1. having a bad idea is not justification for doing it again :-)

Of course not, but my point was that the mistake has already been made, so we may as well make the best out of it... not really doing it again in that sense. But if the decision is that in fact this was a mistake, it should get removed and replaced by a proper solution sooner rather than later (it's not that bad to lose some highlighting and Phobos uses $(D ...) anyway, so just removing it wouldn't hurt much).

Ddoc is very heavily used. I don't believe we can make such silent, breaking changes.
 And in that regard I'd like to
 point to "D's goals for embedded documentation"
 <http://dlang.org/ddoc.htm>. I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something
 like using "#" or "()" to mark symbols + automatic cross referencing for
 marked symbols would be a *really* nice to have.

I don't see any particular advantage to using (name) rather than $(REF name), and some serious disadvantages with false positives like func(param). Oops! A markup language should really strive to minimize special syntax. (I'm often tripped up by false positives in the wacky markup languages in Skype, Github, Reddit, etc., each of which feels compelled to reinvent markup.)
 2. the requirement that the name be "in scope" does a pretty good job of
 reducing the problem. But cross references, are by definition CROSS
 references, i.e. they cross scopes. The whole point of them is they
 cross scopes, which means this becomes a much bigger issue.

This is how I see it: single identifiers are only searched in the current scope and in parent scopes of the same module. Cross references to foreign scopes need to use qualified names (either ClassName.member or pack.mod.Symbol). This avoids any inter-module issues and also a lot of intra-module issues. Symbols in code blocks on the other hand can be cross referenced completely automatically according to the usual D lookup rules (including imports).

My bias is towards being extremely conservative in adding new features to Ddoc, instead we should strive to get the max out of the existing system, and even rely on external post processors.
Dec 30 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 00:37, schrieb Walter Bright:
 On 12/30/2013 3:13 PM, S├Ânke Ludwig wrote:
 Am 30.12.2013 23:13, schrieb Walter Bright:
 On 12/30/2013 1:25 PM, S├Ânke Ludwig wrote:
 Identifiers in documentation comments that are function parameters or
 are names that are in scope at the associated declaration are
 emphasized
 in the output.

 So the same problem is already reality - you already have to go through
 the documentation to see if there are any mis-highlighted words.

1. having a bad idea is not justification for doing it again :-)

Of course not, but my point was that the mistake has already been made, so we may as well make the best out of it... not really doing it again in that sense. But if the decision is that in fact this was a mistake, it should get removed and replaced by a proper solution sooner rather than later (it's not that bad to lose some highlighting and Phobos uses $(D ...) anyway, so just removing it wouldn't hurt much).

Ddoc is very heavily used. I don't believe we can make such silent, breaking changes.

It's just word highlighting! Breaking changes in the language are one thing, but this kind of fix is a whole different category. Keeping it will only make the problem worse over time. If there is a reasonable deprecation path, that would be better, of course, but the current situation is the worst of all possibilities.
 
 And in that regard I'd like to
 point to "D's goals for embedded documentation"
 <http://dlang.org/ddoc.htm>. I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something
 like using "#" or "()" to mark symbols + automatic cross referencing for
 marked symbols would be a *really* nice to have.

I don't see any particular advantage to using (name) rather than $(REF name), and some serious disadvantages with false positives like func(param). Oops! A markup language should really strive to minimize special syntax. (I'm often tripped up by false positives in the wacky markup languages in Skype, Github, Reddit, etc., each of which feels compelled to reinvent markup.)

It's not "(name)", but "name()" or "name(param)", which is recognized as a function call. "#name" also forces recognition as an identifier, as well as "::name" in C++'s case. I agree that going overboard with special syntax is a bad idea, but symbols in a code documentation language is such a fundamental thing that it should surely be among the first things, if not *the* first thing, that deserves a concise syntax. Also, the official first four goals of DDOC are:
 1. It looks good as embedded documentation, not just after it is
    extracted and processed.
 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
    other clumsy forms one would never see in a finished document.
 3. It does not repeat information that the compiler already knows from
    parsing the code.
 4. It doesn't rely on embedded HTML, as such will impede extraction
    and formatting for other purposes.

Using $(XREF), $(TABLE $(TR $(TD ...))), $(LINK a b) $(D ...) and so on is nothing but markup. If we go that route, we could as well remove all syntax, except for the macros: $(SUMMARY ...) $(SECTION See_Also ... ) $(PARAMS $(PARAM x ...) $(PARAM y ...) ) $(SECTION Example $(CODEBLOCK ... ) ) But you added the syntax for a reason back then (readability ...and writability). And the same goes for the auto-detection of identifiers. I think the only problem is that the default would have been better off the other way around.
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 01:05, S├Ânke Ludwig wrote:

 Also, the official first four goals of DDOC are:

 1. It looks good as embedded documentation, not just after it is
     extracted and processed.


It certainly does not. Just have a look at the top of std.algorithm or std.uni. Hundreds of lines of Ddoc madness.
 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
     other clumsy forms one would never see in a finished document.


Fails again. As soon as you need something more advanced than the standard sections you need to use macros. The macros are just as tags, just with a different syntax and name.
 3. It does not repeat information that the compiler already knows from
     parsing the code.


You sure need to do if you want good documentation. Take a look at std.algorithm, it has over 300 lines of Ddoc code to create a symbol table that Ddoc should be able to automatically generate.
 4. It doesn't rely on embedded HTML, as such will impede extraction
     and formatting for other purposes.


As far as I know this isn't very useful. For the other formats we use, like PDF, it uses the HTML output as a base. To me it looks like it failed on all four of these goals.
 Using $(XREF), $(TABLE $(TR $(TD ...))), $(LINK a b) $(D ...) and so on
 is nothing but markup. If we go that route, we could as well remove all
 syntax, except for the macros:

 $(SUMMARY ...)
 $(SECTION See_Also
     ...
 )
 $(PARAMS
    $(PARAM x ...)
    $(PARAM y ...)
 )
 $(SECTION Example
    $(CODEBLOCK
      ...
    )
 )

 But you added the syntax for a reason back then (readability ...and
 writability). And the same goes for the auto-detection of identifiers. I
 think the only problem is that the default would have been better off
 the other way around.

I completely agree. -- /Jacob Carlborg
Dec 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 4:26 AM, Jacob Carlborg wrote:
 On 2013-12-31 01:05, S├Ânke Ludwig wrote:

 Also, the official first four goals of DDOC are:

 1. It looks good as embedded documentation, not just after it is
     extracted and processed.


It certainly does not. Just have a look at the top of std.algorithm or std.uni. Hundreds of lines of Ddoc madness.

That's an exaggeration. The top of std.algorithm produces complex output (two tables). The rest of the documentation is nice and legible.
 2. It's easy and natural to write, i.e. minimal reliance on <tags> and
     other clumsy forms one would never see in a finished document.


Fails again. As soon as you need something more advanced than the standard sections you need to use macros. The macros are just as tags, just with a different syntax and name.

As one who's written both, I can testify this isn't true. Ddoc macros need to be mentioned once and then the closing paren completes them. Additionally, they can expand to arbitrarily complex output. In contrast, tags need to be mentioned twice (an often-mentioned issue with writing HTML by hand), are syntactically heavier, and therefore litter the text a lot more.
 3. It does not repeat information that the compiler already knows from
     parsing the code.


You sure need to do if you want good documentation. Take a look at std.algorithm, it has over 300 lines of Ddoc code to create a symbol table that Ddoc should be able to automatically generate.

No. There are two tables there. One is a categorization of algorithms, which is human-made. There are several solutions to materializing that table but they all would require the intervention of a human being. I figured std.algorithm adds new algorithms rarely enough to allow myself a solution that requires manual intervention. The second table is a "cheat sheet" with human-produced short descriptions for each function. Those short descriptions wouldn't fit well with the longer descriptions.
 4. It doesn't rely on embedded HTML, as such will impede extraction
     and formatting for other purposes.


As far as I know this isn't very useful. For the other formats we use, like PDF, it uses the HTML output as a base.

Again no. You are misinformed. The PDF manual is generated via LaTeX from the same ddoc sources as the HTML pages. Andrei
Dec 31 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 11:51 AM, H. S. Teoh wrote:
 On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:
 On 12/31/13 4:26 AM, Jacob Carlborg wrote:

 4. It doesn't rely on embedded HTML, as such will impede extraction
 and formatting for other purposes.


As far as I know this isn't very useful. For the other formats we use, like PDF, it uses the HTML output as a base.

Again no. You are misinformed. The PDF manual is generated via LaTeX from the same ddoc sources as the HTML pages.

This is true. I will note, however, that limitations in Ddoc make the LaTeX output less than ideal. Many fine points of LaTeX formatting are ignored or simply not possible (e.g., proper use of em- and en-dashes, proper use of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some cases, postprocessing is needed to take full advantage of the LaTeX format (esp. for display of math formulas, which is one of LaTeX's biggest selling points). Even for HTML, the lack of semantic support for paragraphs means XHTML compliance is out of the question, and any proper tag nesting where paragraphs are involved is a fragile hack that is likely non-HTML compliant. But nobody notices this because most browsers are too permissive in what they accept.

These should be fixed in ddoc. Andrei
Dec 31 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 18:08, Andrei Alexandrescu wrote:

 That's an exaggeration. The top of std.algorithm produces complex output
 (two tables). The rest of the documentation is nice and legible.

std.algorithm contains 335 lines of ddoc comments. I would consider that "hundreds", or does "hundreds" mean something else. What I'm saying is that those two tables should be automatically generated, at least the first one.
 As one who's written both, I can testify this isn't true. Ddoc macros
 need to be mentioned once and then the closing paren completes them.
 Additionally, they can expand to arbitrarily complex output. In
 contrast, tags need to be mentioned twice (an often-mentioned issue with
 writing HTML by hand), are syntactically heavier, and therefore litter
 the text a lot more.

I'm considering it to be a sort of tag, just with a different syntax. Who said tags need to have an end tag. A less verbose a minimal reliance on tags would be Markdown or similar. Creating a list in Markdown is simple as: * this * is * a * list * in Markdown
 No. There are two tables there. One is a categorization of algorithms,
 which is human-made. There are several solutions to materializing that
 table but they all would require the intervention of a human being. I
 figured std.algorithm adds new algorithms rarely enough to allow myself
 a solution that requires manual intervention.

If there's a specific macro that would indicate a category then it would be easy to generate a table like that. /// $(CATEGORY searching) void foo ()
 The second table is a "cheat sheet" with human-produced short
 descriptions for each function. Those short descriptions wouldn't fit
 well with the longer descriptions.

That can also be generated automatically. Ddoc can just insert the summary [1] for each symbol. [1] The Ddoc documentation says: "The first section is the Summary, and does not have a section name. It is first paragraph, up to a blank line or a section name. While the summary can be any length, try to keep it to one line." http://dlang.org/ddoc.html
 Again no. You are misinformed. The PDF manual is generated via LaTeX
 from the same ddoc sources as the HTML pages.

Ok, fair enough. -- /Jacob Carlborg
Jan 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/1/14 4:56 AM, Jacob Carlborg wrote:
 On 2013-12-31 18:08, Andrei Alexandrescu wrote:

 That's an exaggeration. The top of std.algorithm produces complex output
 (two tables). The rest of the documentation is nice and legible.

std.algorithm contains 335 lines of ddoc comments. I would consider that "hundreds", or does "hundreds" mean something else.

I replied to "Hundreds of lines of Ddoc madness" with "that's an exaggeration". Do you really think I was refuting the factual 335 lines?
 What I'm saying is
 that those two tables should be automatically generated, at least the
 first one.

Yah, that would be nice.
 As one who's written both, I can testify this isn't true. Ddoc macros
 need to be mentioned once and then the closing paren completes them.
 Additionally, they can expand to arbitrarily complex output. In
 contrast, tags need to be mentioned twice (an often-mentioned issue with
 writing HTML by hand), are syntactically heavier, and therefore litter
 the text a lot more.

I'm considering it to be a sort of tag, just with a different syntax. Who said tags need to have an end tag. A less verbose a minimal reliance on tags would be Markdown or similar. Creating a list in Markdown is simple as: * this * is * a * list * in Markdown

I prefer ddoc to markdown for its flexibility. Ddoc is an almost pure macro expansion system with minimal syntax. Markdown defines a bunch of syntax but has (last time I looked) is weak on macro capability. Case in point: markdown lists will be styled one way. With DDoc macros one can define any style of lists needed. (We've defined BOOKTABLE in addition to a couple of other TABLEs, which works beautifully in HTML and LaTeX; not sure how can be done in Markdown.
 If there's a specific macro that would indicate a category then it would
 be easy to generate a table like that.

 /// $(CATEGORY searching)
 void foo ()

The issue is with writing text out of order, something DDoc is not very apt at. Would be nice to improve on that (we do some of it with javascript). Andrei
Jan 01 2014
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, January 02, 2014 13:38:22 H. S. Teoh wrote:
 But postprocessing also removes some of the advantage of ddoc: if you
 translate everything down to HTML tags, the postprocessor will have to
 reparse the HTML and resynthesize information on where sections
 begin/end, what constitutes a declaration, etc., which may not be
 reliable. So now you have to use an intermediate format (like docbook)
 that still captures some of the original structure, so that the
 postprocessor can reliably recover the original document structure from
 it. But then, you might as well just write in the intermediate format to
 begin with, instead of using ddoc.

That depends on how you go about post-processing. If you design your ddoc macros with the idea that there's going to be post-processing, then they can convert to other ddoc macros which are designed for the post-processor to be able to handle, and then the post-processor can convert those into another set of macros that you run ddoc on again to get the final result. That's essentially what I've done when using ddoc for stuff like a table of contents or index. Now, if the post-processing is really the last step rather than just post one of multiple runs of ddoc, then it's stuck dealing with whatever the target format is, but if you do multiple runs of ddoc with a separate program operating on the intermediate results of the macro expansion and then generating a new set of ddoc macros for another run of ddoc, then your post- processor doesn't have to understand the target format. - Jonathan M Davis
Jan 02 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-03 01:56, Jonathan M Davis wrote:

 That depends on how you go about post-processing. If you design your ddoc
 macros with the idea that there's going to be post-processing, then they can
 convert to other ddoc macros which are designed for the post-processor to be
 able to handle, and then the post-processor can convert those into another set
 of macros that you run ddoc on again to get the final result. That's
 essentially what I've done when using ddoc for stuff like a table of contents
 or index.

 Now, if the post-processing is really the last step rather than just post one
 of multiple runs of ddoc, then it's stuck dealing with whatever the target
 format is, but if you do multiple runs of ddoc with a separate program
 operating on the intermediate results of the macro expansion and then
 generating a new set of ddoc macros for another run of ddoc, then your post-
 processor doesn't have to understand the target format.

Seems unnecessary complicated. -- /Jacob Carlborg
Jan 03 2014
prev sibling next sibling parent =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.01.2014 22:38, schrieb H. S. Teoh:
 For example,
 using `backticks` to write code snippets is definitely more readable
 than writing $(D backticks), among many other things. But that does
 introduce more syntax, which makes parsing more involved and also
 requires learning more syntax. So, it's a trade-off.

I don't want to argue for as much syntax as Markdown has - even if I'd actually find that great, Markdown has become a kind of de-facto standard in the programming area - but, overall, at least the aspect of learning new syntax is not really worse than learning new macros. There may be cases where a form of syntax is more difficult to remember, but there are also many cases where the macros are actually harder to learn and remember. Having said that, Markdown, if not implemented using a series of regexes, is a crazy format to parse. There is an existing parser in vibe.d, though.
 Case in point: markdown lists will be styled one way. With DDoc
 macros one can define any style of lists needed. (We've defined
 BOOKTABLE in addition to a couple of other TABLEs, which works
 beautifully in HTML and LaTeX; not sure how can be done in Markdown.

Styling is an orthogonal issue. Markdown was designed to translate to HTML, so you'd just write CSS stylesheets to customize the output. Ddoc, OTOH, was designed to be a generic macro system, so obviously it's better at doing macro-based stuff.

Also, an important point is that markdown defines *semantic* primitives. They can be translated to whatever is required by the output format. In DDOC's case that could simply be some standard macros ($(UL), $(LI)). BTW to Markdown HTML basically is what macros are to DDOC (just that it also has defined semantics). Inline HTML explicitly supported to handle advanced cases and, if restricted to semantic tags, provides a pretty good basis for translating into other output formats.
Jan 02 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-02 22:38, H. S. Teoh wrote:

 The limitation of macro systems is that, fundamentally speaking, no
 semantics are assigned to macros. A macro is just an arbitrary name that
 gets recursively substituted with some pre-specified pattern.  The macro
 engine doesn't know what any of the macros "mean", and therefore has no
 concept of things like cross-referencing, categorization, etc., which
 are all higher-level, logical concepts, that are required in order to
 move data around.

We could start adding semantic meaning to some macros. They would become built-in macros ddoc knows about. -- /Jacob Carlborg
Jan 03 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 00:37, Walter Bright wrote:

 I don't see any particular advantage to using (name) rather than $(REF
 name), and some serious disadvantages with false positives like
 func(param). Oops! A markup language should really strive to minimize
 special syntax. (I'm often tripped up by false positives in the wacky
 markup languages in Skype, Github, Reddit, etc., each of which feels
 compelled to reinvent markup.)

You never addressed any of my other suggestions. Would suggestion B, that is having a special macro that compiler knows about, be an acceptable solution? Because the current behavior is not good enough.
 My bias is towards being extremely conservative in adding new features
 to Ddoc, instead we should strive to get the max out of the existing
 system, and even rely on external post processors.

If Ddoc requires post processing to be useful it's a complete failure. Why don't we instead strive to make it the best documentation generator for D? -- /Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 4:14 AM, Jacob Carlborg wrote:
 If Ddoc requires post processing to be useful it's a complete failure.

Not at all. The same ddoc sources are used to generate an ebook and a Windows help file - which are based on html but require different html to be generated, a difference handled by ddoc. The ebook and chm both require post processing with tools supplied by Amazon and Microsoft, respectively. It works, it is not a failure because the source files do not need to be modified to get this working. A failure would be having to modify the source files each time a different output needs to be generated.
 Why don't we instead strive to make it the best documentation generator for D?

We all of course want that - but there is certainly room for disagreement about what "best" is. Reinventing Amazon's "kindlegen.exe" seems pointless to me.
Dec 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 22:08, Walter Bright wrote:

 Not at all. The same ddoc sources are used to generate an ebook and a
 Windows help file - which are based on html but require different html
 to be generated, a difference handled by ddoc. The ebook and chm both
 require post processing with tools supplied by Amazon and Microsoft,
 respectively.

 It works, it is not a failure because the source files do not need to be
 modified to get this working. A failure would be having to modify the
 source files each time a different output needs to be generated.

Ok, I would not consider it a failure if it needs to be converted to a different format. But adding cross referencing, generating an index or similar would still be a failure, if that requires post processing.
 We all of course want that - but there is certainly room for
 disagreement about what "best" is.

Yes, but who would not want cross referencing to work properly?
 Reinventing Amazon's "kindlegen.exe" seems pointless to me.

Yes, I agree. -- /Jacob Carlborg
Jan 01 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 10:31, Marco Leise wrote:

 I am VERY much in favor of a more MarkDown style syntax and #
 for identifier references! It is also very verbose to create
 bullet-point lists and you always have to double-check you
 don't forget to replace all ) with $(RPAREN). If you misspell
 anything it is lost in the output.

I completely agree. I suggested the $(REF) macro as an alternative to automatically add cross referencing because I thought that would have a greater chance of being accepted. -- /Jacob Carlborg
Dec 31 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2013 at 06:12:02PM -0800, Andrei Alexandrescu wrote:
 On 12/31/13 11:51 AM, H. S. Teoh wrote:
On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:
On 12/31/13 4:26 AM, Jacob Carlborg wrote:

4. It doesn't rely on embedded HTML, as such will impede
extraction and formatting for other purposes.


As far as I know this isn't very useful. For the other formats we use, like PDF, it uses the HTML output as a base.

Again no. You are misinformed. The PDF manual is generated via LaTeX from the same ddoc sources as the HTML pages.

This is true. I will note, however, that limitations in Ddoc make the LaTeX output less than ideal. Many fine points of LaTeX formatting are ignored or simply not possible (e.g., proper use of em- and en-dashes, proper use of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some cases, postprocessing is needed to take full advantage of the LaTeX format (esp. for display of math formulas, which is one of LaTeX's biggest selling points). Even for HTML, the lack of semantic support for paragraphs means XHTML compliance is out of the question, and any proper tag nesting where paragraphs are involved is a fragile hack that is likely non-HTML compliant. But nobody notices this because most browsers are too permissive in what they accept.

These should be fixed in ddoc.

For HTML: https://d.puremagic.com/issues/show_bug.cgi?id=9731 For LaTeX, though, it would require some pretty radical changes to ddoc, that I'm not sure will be accepted. T -- Blunt statements really don't have a point.
Jan 01 2014
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 01, 2014 at 06:41:50PM -0800, Andrei Alexandrescu wrote:
 On 1/1/14 4:56 AM, Jacob Carlborg wrote:

I'm considering it to be a sort of tag, just with a different syntax.
Who said tags need to have an end tag. A less verbose a minimal
reliance on tags would be Markdown or similar. Creating a list in
Markdown is simple as:

* this
* is
* a
* list
* in Markdown

I prefer ddoc to markdown for its flexibility. Ddoc is an almost pure macro expansion system with minimal syntax. Markdown defines a bunch of syntax but has (last time I looked) is weak on macro capability.

Well, ddoc was designed to be a macro system, so from that POV, it's pretty good at what it was intended to do. Whether it's the best system for *generating documentation*, though, is a matter of opinion. Markdown has the advantage of having very close visual appearance in the input as compared to the output -- a stated goal of ddoc, but obviously the unified macro syntax was a higher priority in ddoc's case. For example, using `backticks` to write code snippets is definitely more readable than writing $(D backticks), among many other things. But that does introduce more syntax, which makes parsing more involved and also requires learning more syntax. So, it's a trade-off.
 Case in point: markdown lists will be styled one way. With DDoc
 macros one can define any style of lists needed. (We've defined
 BOOKTABLE in addition to a couple of other TABLEs, which works
 beautifully in HTML and LaTeX; not sure how can be done in Markdown.

Styling is an orthogonal issue. Markdown was designed to translate to HTML, so you'd just write CSS stylesheets to customize the output. Ddoc, OTOH, was designed to be a generic macro system, so obviously it's better at doing macro-based stuff.
If there's a specific macro that would indicate a category then it
would be easy to generate a table like that.

/// $(CATEGORY searching)
void foo ()

The issue is with writing text out of order, something DDoc is not very apt at. Would be nice to improve on that (we do some of it with javascript).

The limitation of macro systems is that, fundamentally speaking, no semantics are assigned to macros. A macro is just an arbitrary name that gets recursively substituted with some pre-specified pattern. The macro engine doesn't know what any of the macros "mean", and therefore has no concept of things like cross-referencing, categorization, etc., which are all higher-level, logical concepts, that are required in order to move data around. There are, of course, ways of simulating this with macros, such as with a global top-level macro that takes an arbitrary number of arguments and can permute them at will, but this is ultimately just a hack. Macro systems simply don't provide enough expressive power to build abstractions like "this block of text is a `declaration` with `identifier` as key, so add `identifier` to the index structure in the output". Postprocessing is the only way to reliably add this kind of semantics to it. But postprocessing also removes some of the advantage of ddoc: if you translate everything down to HTML tags, the postprocessor will have to reparse the HTML and resynthesize information on where sections begin/end, what constitutes a declaration, etc., which may not be reliable. So now you have to use an intermediate format (like docbook) that still captures some of the original structure, so that the postprocessor can reliably recover the original document structure from it. But then, you might as well just write in the intermediate format to begin with, instead of using ddoc. So while I do agree that ddoc is pretty good at what it does -- expanding macros -- I'm still on the fence as to whether that's the best approach to documentation generation. T -- Sometimes the best solution to morale problems is just to fire all of the unhappy people. -- despair.com
Jan 02 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/30/13 1:00 PM, Walter Bright wrote:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

Your arguments are convincing. Andrei
Dec 30 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
 On 12/30/13 1:00 PM, Walter Bright wrote:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

Your arguments are convincing. Andrei

The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account. - Visual noise and effort when using $(XREF) style macros - $(XREF) isn't sufficient! A multitude of macros is actually needed - There is no standard set of macros, it's project specific -> The whole documentation needs to be revised when moving code between projects! - Macros need to be carefully crafted so that they don't encode knowledge about the final documentation/file structure - Macros alone can't cope with nested directory structures - The cognitive load for multiple such macros would very likely cause them to go unused in many cases My favorite solution would be to introduce a simple #identifier.chain or identifier.chain syntax to let the compiler or documentation generator insert the proper links or macro calls. Using a sepecial $(REF identifier.chain) pseudo macro, like Jacob suggested, would work as well, but I can't see how the additional syntax/visual/typing overhead is justified for such a common feature - at least when trying to find an optimal solution, compared to the current macro based system it would of course still be a big improvement. But still, considering that there is precedence for the _negative approach (i.e. the problems of it are already there), I find using that acceptable, too, even if the concept is not as pretty/clean. * *Some* solution is needed. The different structure of the DDOX documentation for example makes the $(XREF) macros worthless, so they are currently completely disabled and automatic link generation is used instead.
Dec 31 2013
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 11:05, schrieb Vladimir Panteleev:
 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format

Although I can't promise that I didn't tamper with these settings, I didn't find any non-default value in about:config in that area, but there is a default setting "mailnews.send_plaintext_flowed = false". This is now manually set to true, let's see.
Dec 31 2013
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.01.2014 02:21, schrieb Vladimir Panteleev:
 On Tuesday, 31 December 2013 at 10:37:23 UTC, S├Ânke Ludwig wrote:
 Am 31.12.2013 11:05, schrieb Vladimir Panteleev:
 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format

Although I can't promise that I didn't tamper with these settings, I didn't find any non-default value in about:config in that area, but there is a default setting "mailnews.send_plaintext_flowed = false". This is now manually set to true, let's see.

Did you install the Enigmail extension? It seems to have a bug which turns off format=flowed generation even if you never use it.

Yep, that seems to be it. Good to know, I've never noticed that up to now.
Jan 01 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 11:05, Vladimir Panteleev wrote:

 I agree that a good compromise would be to lighten the syntactic overhead.

 How about using the $(D) macro instead of $(REF)? This macro
 syntax-highlights its parameter as D code, and in Phobos it is already
 used for identifiers, so I think a good solution would be to find and
 create cross-reference links only for $(D contents).

 Also, I'd like to mention a previous idea I posted here: this problem,
 and several other problems we have with DDoc, would be gone if we'd put
 what we already have with what we already have, and allow writing D code
 that gets interpreted to translate documentation comments into
 documentation output, using our existing CTFE capabilities. This way,
 the entire DDoc syntax can be reimplemented using "library" D code,
 interpreted by the compiler during documentation generation, and anyone
 could build upon it or completely replace it with another syntax.

Isn't this something S├Ânke has done with DDOX and with the help of the JSON output? I've been thinking that it probably wouldn't be that hard to create a __trait that return the ddoc comments of a symbol. Don't know how much that would be different from the JSON output though. -- /Jacob Carlborg
Dec 31 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 1:28 AM, S├Ânke Ludwig wrote:
 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
 On 12/30/13 1:00 PM, Walter Bright wrote:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

Your arguments are convincing. Andrei

The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account. - Visual noise and effort when using $(XREF) style macros - $(XREF) isn't sufficient! A multitude of macros is actually needed - There is no standard set of macros, it's project specific -> The whole documentation needs to be revised when moving code between projects! - Macros need to be carefully crafted so that they don't encode knowledge about the final documentation/file structure - Macros alone can't cope with nested directory structures - The cognitive load for multiple such macros would very likely cause them to go unused in many cases

In a nutshell, "good cross-referencing is hard". That's exactly the experience of anyone who's created an index. In publishing there are automatic index generators and they're universally known to be vastly inferior to professional-produced ones. I've created my own index by hand for TDPL and it's been difficult, but probably the result is better than an automated index. Andrei
Dec 31 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 31.12.2013 17:48, schrieb Andrei Alexandrescu:
 On 12/31/13 1:28 AM, S├Ânke Ludwig wrote:
 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
 On 12/30/13 1:00 PM, Walter Bright wrote:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

Your arguments are convincing. Andrei

The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account. - Visual noise and effort when using $(XREF) style macros - $(XREF) isn't sufficient! A multitude of macros is actually needed - There is no standard set of macros, it's project specific -> The whole documentation needs to be revised when moving code between projects! - Macros need to be carefully crafted so that they don't encode knowledge about the final documentation/file structure - Macros alone can't cope with nested directory structures - The cognitive load for multiple such macros would very likely cause them to go unused in many cases

In a nutshell, "good cross-referencing is hard".

I don't think so at all (on the technical level). What is the issue with using the #identifier.chain pattern for example? It seems simple enough and I can't imagine a more natural thing than to reference symbols in documentation the same way as in code.
 That's exactly the
 experience of anyone who's created an index. In publishing there are
 automatic index generators and they're universally known to be vastly
 inferior to professional-produced ones. I've created my own index by
 hand for TDPL and it's been difficult, but probably the result is better
 than an automated index.

But index generation is a higher level issue. The basis for it is to have some kind of universal anchor that can be used as a reference. #identifier.chain or $(REF identifier.chain) is nothing more than that and just solves the issue of supporting arbitrary nested scopes and different file system or document hierarchies.
Dec 31 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/31/13 9:39 AM, S├Ânke Ludwig wrote:
 Am 31.12.2013 17:48, schrieb Andrei Alexandrescu:
 In a nutshell, "good cross-referencing is hard".

I don't think so at all (on the technical level). What is the issue with using the #identifier.chain pattern for example? It seems simple enough and I can't imagine a more natural thing than to reference symbols in documentation the same way as in code.

I see #identifier.chain similar to e.g. $(X identifier.chain) so no issue with that. It just adds more syntax without necessity. I'd instead work on improving macros to generate proper xrefs out of $(X ...) rather than adding new syntax. But either way is fine.
 That's exactly the
 experience of anyone who's created an index. In publishing there are
 automatic index generators and they're universally known to be vastly
 inferior to professional-produced ones. I've created my own index by
 hand for TDPL and it's been difficult, but probably the result is better
 than an automated index.

But index generation is a higher level issue. The basis for it is to have some kind of universal anchor that can be used as a reference. #identifier.chain or $(REF identifier.chain) is nothing more than that and just solves the issue of supporting arbitrary nested scopes and different file system or document hierarchies.

Agreed. In all likelihood I misunderstood something because I thought you were referring to cross-referencing text without any user intervention. Andrei
Dec 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-31 19:16, Andrei Alexandrescu wrote:

 Agreed. In all likelihood I misunderstood something because I thought
 you were referring to cross-referencing text without any user intervention.

I made two suggestions, one which is completely automatic and one which requires some syntax. It seems most posts here only address the first suggestion. -- /Jacob Carlborg
Dec 31 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 03, 2014 12:12:44 Jacob Carlborg wrote:
 On 2014-01-03 01:56, Jonathan M Davis wrote:
 That depends on how you go about post-processing. If you design your ddoc
 macros with the idea that there's going to be post-processing, then they
 can convert to other ddoc macros which are designed for the
 post-processor to be able to handle, and then the post-processor can
 convert those into another set of macros that you run ddoc on again to
 get the final result. That's essentially what I've done when using ddoc
 for stuff like a table of contents or index.
 
 Now, if the post-processing is really the last step rather than just post
 one of multiple runs of ddoc, then it's stuck dealing with whatever the
 target format is, but if you do multiple runs of ddoc with a separate
 program operating on the intermediate results of the macro expansion and
 then generating a new set of ddoc macros for another run of ddoc, then
 your post- processor doesn't have to understand the target format.

Seems unnecessary complicated.

It depends on what you're trying to do. For generating something like a table of contents, it's pretty straightforward. For instance, $(CHAPTER Introduction, $(P Contents of chapter) ) could be processed by a short program to have that chapter's name put in the table of contents which it generates (which is then also ddoc). e.g. $(TABLE_OF_CONTENTS $(CHAPTER_ENTRY Introduction, 1) $(CHAPTER_ENTRY Some Other Chapter, 2) $(CHAPTER_ENTRY Conclusion, 3) ) and if any other operations need to be done on CHAPTER to make it suitable for your target formats, then it can be tweaked as necessary. You then take the resulting .dd file and generate the documentation with the ddoc file that corresponds to the target format (e.g. html.ddoc or latex.ddoc or whatever). I've found that it works quite well. But that's for generating a book, not documentation for code. You could do something similar to generate cross-references in code documentation, I'm sure, but I wasn't necessarily suggesting that we do. I was just pointing out that it's quite possible (and not necessarily difficult) to manipulate a .dd file (or a .d file with ddoc in it) to generate another .dd file which you then actually use to generate the html or latex or whatever you need to generate and that you don't necessarily need to manipulate the target format to get what you want. - Jonathan M Davis
Jan 03 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
XSLT is pretty useful for format conversion. It is also built 
into most browsers, so you can just emit XML attach a XSLT 
stylesheet and send it to a browser for transformation and 
rendering.

Would be an advantage for anyone writing a book or article on D 
(transform XML over to docbook, and from there to any format).
Jan 03 2014
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 31, 2013 at 09:08:07AM -0800, Andrei Alexandrescu wrote:
 On 12/31/13 4:26 AM, Jacob Carlborg wrote:

4. It doesn't rely on embedded HTML, as such will impede extraction
and formatting for other purposes.


As far as I know this isn't very useful. For the other formats we use, like PDF, it uses the HTML output as a base.

Again no. You are misinformed. The PDF manual is generated via LaTeX from the same ddoc sources as the HTML pages.

This is true. I will note, however, that limitations in Ddoc make the LaTeX output less than ideal. Many fine points of LaTeX formatting are ignored or simply not possible (e.g., proper use of em- and en-dashes, proper use of '.\ ', non-breaking spaces, hyphenation hints, etc.). In some cases, postprocessing is needed to take full advantage of the LaTeX format (esp. for display of math formulas, which is one of LaTeX's biggest selling points). Even for HTML, the lack of semantic support for paragraphs means XHTML compliance is out of the question, and any proper tag nesting where paragraphs are involved is a fragile hack that is likely non-HTML compliant. But nobody notices this because most browsers are too permissive in what they accept. So even though I think Ddoc isn't as bad as some people make it out to be, I'm still on the fence about it. T -- By understanding a machine-oriented language, the programmer will tend to use a much more efficient method; it is much closer to reality. -- D. Knuth
Dec 31 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/29/13 10:35 AM, S÷nke Ludwig wrote:
 Am 29.12.2013 18:38, schrieb Jacob Carlborg:
 A. Automatic cross reference

This is done for the DDOX based docs that were supposed to end up on the home page at some point:

It's past time we do this. So the code is in there, we need to build it appropriately. Where do I start reading? Andrei
Dec 30 2013
parent =?ISO-8859-1?Q?S=F6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 30.12.2013 17:31, schrieb Andrei Alexandrescu:
 On 12/29/13 10:35 AM, S÷nke Ludwig wrote:
 Am 29.12.2013 18:38, schrieb Jacob Carlborg:
 A. Automatic cross reference

This is done for the DDOX based docs that were supposed to end up on the home page at some point:

It's past time we do this. So the code is in there, we need to build it appropriately. Where do I start reading? Andrei

I've just added a small alternative link below the normal Phobos documentation and added it to the "all" target of posix.mak. It will automatically call dub to build "dpl-docs" in the tools repository. https://github.com/D-Programming-Language/dlang.org/pull/462 If there are no more issues with it, it should be the first step that actually gets something online.
Dec 30 2013
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 December 2013 at 09:29:20 UTC, S├Ânke Ludwig wrote:
 My favorite solution would be to introduce a simple 
 #identifier.chain or
  identifier.chain syntax to let the compiler or documentation 
 generator
 insert the proper links or macro calls. Using a sepecial $(REF
 identifier.chain) pseudo macro, like Jacob suggested, would 
 work as
 well, but I can't see how the additional syntax/visual/typing 
 overhead
 is justified for such a common feature - at least when trying 
 to find an
 optimal solution, compared to the current macro based system it 
 would of
 course still be a big improvement.

I agree that a good compromise would be to lighten the syntactic overhead. How about using the $(D) macro instead of $(REF)? This macro syntax-highlights its parameter as D code, and in Phobos it is already used for identifiers, so I think a good solution would be to find and create cross-reference links only for $(D contents). Also, I'd like to mention a previous idea I posted here: this problem, and several other problems we have with DDoc, would be gone if we'd put what we already have with what we already have, and allow writing D code that gets interpreted to translate documentation comments into documentation output, using our existing CTFE capabilities. This way, the entire DDoc syntax can be reimplemented using "library" D code, interpreted by the compiler during documentation generation, and anyone could build upon it or completely replace it with another syntax. P.S. How come your user agent (Thunderbird) is not emitting format=flowed messages? According to [1], it supports format=flowed, and can only be disabled via editing option strings. [1]: http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format
Dec 31 2013
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 December 2013 at 10:37:23 UTC, S├Ânke Ludwig wrote:
 Am 31.12.2013 11:05, schrieb Vladimir Panteleev:
 P.S. How come your user agent (Thunderbird) is not emitting
 format=flowed messages? According to [1], it supports 
 format=flowed, and
 can only be disabled via editing option strings.

 [1]:
 http://kb.mozillazine.org/Plain_text_e-mail_(Thunderbird)#Flowed_format

Although I can't promise that I didn't tamper with these settings, I didn't find any non-default value in about:config in that area, but there is a default setting "mailnews.send_plaintext_flowed = false". This is now manually set to true, let's see.

Did you install the Enigmail extension? It seems to have a bug which turns off format=flowed generation even if you never use it.
Dec 31 2013
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Dec 29, 2013 at 06:38:55PM +0100, Jacob Carlborg wrote:
 If nothing has happened recently the current situation of cross
 referencing in Ddoc sucks. What's currently being used in the Phobos
 documentation is the XREF, CXREF and ECXREF ddoc macros. These
 macros take two arguments, append "std", "core" or "etc" and form a
 link of the arguments. The problem with this is that it doesn't work
 so good for referring to symbols in a deeper package hierarchy.

Not only so, even without cross-referencing, the way Ddoc currently does referencing *within* a module is faulty, because it does not take symbols declared in nested scopes into account. For example: module mymodule; /// docs here struct S { /// docs here void func() { ... } } /// docs here void func() { ... } Ddoc will use "func" as the identifier for *both* mymodule.func and mymodule.S.func, so any hyperlinks to "func" will likely point to the wrong overload of func (depending on declaration order). This problem is actually being exhibited on dlang.org at this very moment, in std.algorithm. Look for 'remove' in the navigation table at the top of the page, and click on it; you'll see it jumps to the wrong place because there's a member function called 'remove' in a struct that took precedence over the module-level 'remove' function. (Not to mention it breaks HTML compliance because it gives multiple elements the same ID.) T -- People tell me that I'm paranoid, but they're just out to get me.
Dec 29 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 05:05, H. S. Teoh wrote:

 Not only so, even without cross-referencing, the way Ddoc currently does
 referencing *within* a module is faulty, because it does not take
 symbols declared in nested scopes into account. For example:

 	module mymodule;

 	/// docs here
 	struct S {
 		/// docs here
 		void func() { ... }
 	}

 	/// docs here
 	void func() { ... }

 Ddoc will use "func" as the identifier for *both* mymodule.func and
 mymodule.S.func, so any hyperlinks to "func" will likely point to the
 wrong overload of func (depending on declaration order).

 This problem is actually being exhibited on dlang.org at this very
 moment, in std.algorithm. Look for 'remove' in the navigation table at
 the top of the page, and click on it; you'll see it jumps to the wrong
 place because there's a member function called 'remove' in a struct that
 took precedence over the module-level 'remove' function. (Not to mention
 it breaks HTML compliance because it gives multiple elements the same
 ID.)

Yeah, that's another problem. -- /Jacob Carlborg
Dec 30 2013
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
30-Dec-2013 14:19, Jacob Carlborg đ┐đŞĐłđÁĐé:
 On 2013-12-30 05:05, H. S. Teoh wrote:

 Not only so, even without cross-referencing, the way Ddoc currently does
 referencing *within* a module is faulty, because it does not take
 symbols declared in nested scopes into account. For example:

     module mymodule;

     /// docs here
     struct S {
         /// docs here
         void func() { ... }
     }

     /// docs here
     void func() { ... }

 Ddoc will use "func" as the identifier for *both* mymodule.func and
 mymodule.S.func, so any hyperlinks to "func" will likely point to the
 wrong overload of func (depending on declaration order).

 This problem is actually being exhibited on dlang.org at this very
 moment, in std.algorithm. Look for 'remove' in the navigation table at
 the top of the page, and click on it; you'll see it jumps to the wrong
 place because there's a member function called 'remove' in a struct that
 took precedence over the module-level 'remove' function. (Not to mention
 it breaks HTML compliance because it gives multiple elements the same
 ID.)

Yeah, that's another problem.

And couple of years old at that... -- Dmitry Olshansky
Dec 30 2013
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 29 December 2013 at 17:38:55 UTC, Jacob Carlborg wrote:
 If nothing has happened recently the current situation of cross 
 referencing in Ddoc sucks. What's currently being used in the 
 Phobos documentation is the XREF, CXREF and ECXREF ddoc macros. 
 These macros take two arguments, append "std", "core" or "etc" 
 and form a link of the arguments. The problem with this is that 
 it doesn't work so good for referring to symbols in a deeper 
 package hierarchy.

bootDoc[1] fixes this (thanks to Denis) with a number of standard macros that are reasonably easy to use[2]. It also uses JavaScript to "fix" (insofar as JS is a fix for anything) the issue of page anchors not supporting nested symbols properly. [1] https://github.com/JakobOvrum/bootDoc [2] https://github.com/JakobOvrum/bootDoc/wiki/Macros
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 10:43, Jakob Ovrum wrote:

 bootDoc[1] fixes this (thanks to Denis) with a number of standard macros
 that are reasonably easy to use[2].

Based on the documentation it looks like it has the same problems as the current approach, that is: * Multiple macros are needed for something that at most one single macro should be needed * Doesn't work for arbitrary deep hierarchies Also, it feels like a workaround for something that should be fixed in the compiler. -- /Jacob Carlborg
Dec 30 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-12-30 11:26, Jakob Ovrum wrote:

 Yep, its only benefit is that it's a standardized set of macros and
 works for any reasonably deep hierarchy.

Not as far as I can see. What I can see it only supports one level of packages. That means basically a flat hierarchy like Phobos. All of my projects have a deeper hierarchy than that, usually one extra level of packages. I would consider a nested package to be within the limit of a reasonably deep hierarchy. -- /Jacob Carlborg
Dec 30 2013
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 30 December 2013 at 10:23:25 UTC, Jacob Carlborg wrote:
 Based on the documentation it looks like it has the same 
 problems as the current approach, that is:

Yep, its only benefit is that it's a standardized set of macros and works for any reasonably deep hierarchy. bootDoc is about doing the best that is possible within the limitations of DDoc. I think it is a better example of DDoc's best than Phobos, as it's much cleaner and actually has documentation. That said, DDoc's best is hardly impressive.
Dec 30 2013
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/29/2013 9:38 AM, Jacob Carlborg wrote:
 A. Automatic cross reference

 Automatically create links for all matching symbols in the current scope. That
 means to create a link to a symbol in the current scope it's enough to mention
 it's name in the documentation.

This will create many unwanted links, in three scenarios: 1. In the description for S, there should not be hyperlinks to S. These would be annoying - when you click on them, nothing will happen. 2. Using the word S as a word, not in reference to symbol S, would generate a hyperlink which would not make sense. 3. If there are multiple symbols S, hyperlinks to the wrong one would be created. This is worse than useless. In my adding of such hyperlinks, 1..3 happen regularly, this is not a "may" happen. It's a "will" happen, a lot.
Dec 30 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 20:08, Walter Bright wrote:

 This will create many unwanted links, in three scenarios:

I don't think so, see below.
 1. In the description for S, there should not be hyperlinks to S. These
 would be annoying - when you click on them, nothing will happen.

The compiler knows which symbol it currently generating documentation for, just exclude that.
 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.

I've have seen that happened a couple of times with documentation generators that support it, it has never bother me. As S├Ânke has said, you can solve that by appending an underscore and the compiler would not create a link of the symbol.
 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.

Most of the times there are not multiple symbols with the same name, the language make sure of that. Although there will be a problem with function overloading.
 In my adding of such hyperlinks, 1..3 happen regularly, this is not a
 "may" happen. It's a "will" happen, a lot.

I don't agree, I don't think it's a problem. It's far better than having to add macros everywhere instead. -- /Jacob Carlborg
Dec 30 2013
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am I slow... ;)
Dec 30 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 12:14 PM, Jacob Carlborg wrote:
 On 2013-12-30 20:08, Walter Bright wrote:

 This will create many unwanted links, in three scenarios:

I don't think so, see below.
 1. In the description for S, there should not be hyperlinks to S. These
 would be annoying - when you click on them, nothing will happen.

The compiler knows which symbol it currently generating documentation for, just exclude that.

That's true.
 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.

I've have seen that happened a couple of times with documentation generators that support it, it has never bother me. As S├Ânke has said, you can solve that by appending an underscore and the compiler would not create a link of the symbol.

While possible, I'm not too enamored with this. It makes for a maintenance problem. Add a new symbol to a module, and then you have to manually search for any other instances of the word. Naturally, the latter won't get done, and you're left with nonsense links in the doc.
 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.

Most of the times there are not multiple symbols with the same name, the language make sure of that. Although there will be a problem with function overloading.

I've run into this a few times, and it cannot be dismissed easily. Worse, there is the issue of how one overrides the auto-link generation to be the right link.
 It's far better than having to add
 macros everywhere instead.

I'll have to disagree on that. I just finished doing it with std.datetime, and it didn't take more than a few minutes with global query-search-replace.
Dec 30 2013
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 21:58, schrieb Walter Bright:
 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.

I've have seen that happened a couple of times with documentation generators that support it, it has never bother me. As S├Ânke has said, you can solve that by appending an underscore and the compiler would not create a link of the symbol.

While possible, I'm not too enamored with this. It makes for a maintenance problem. Add a new symbol to a module, and then you have to manually search for any other instances of the word. Naturally, the latter won't get done, and you're left with nonsense links in the doc.

True, but I'd argue that this is really the exception. New symbols also influence just the documentation of the current scope (i.e. a new class member won't affect the documentation of other classes in the same module).
 
 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.

Most of the times there are not multiple symbols with the same name, the language make sure of that. Although there will be a problem with function overloading.

I've run into this a few times, and it cannot be dismissed easily. Worse, there is the issue of how one overrides the auto-link generation to be the right link.

Qualified names. The qualifying prefix could be hidden from the user or displayed as is. But after all it also provides useful information to the reader, as it's now possible to tell which symbol is meant without following the link first.
Dec 30 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 21:58, Walter Bright wrote:

 I've run into this a few times, and it cannot be dismissed easily.
 Worse, there is the issue of how one overrides the auto-link generation
 to be the right link.

I guess that would require a macro.
 I'll have to disagree on that. I just finished doing it with
 std.datetime, and it didn't take more than a few minutes with global
 query-search-replace.

How did you limit that search to only the documentation? -- /Jacob Carlborg
Dec 30 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 1:31 PM, Jacob Carlborg wrote:
 On 2013-12-30 21:58, Walter Bright wrote:
 I'll have to disagree on that. I just finished doing it with
 std.datetime, and it didn't take more than a few minutes with global
 query-search-replace.

How did you limit that search to only the documentation?

The "query" part, meaning it highlighted each one and asked for yes/no. http://www.gnu.org/software/emacs/manual/html_node/emacs/Query-Replace.html
Dec 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-30 23:10, Walter Bright wrote:

 The "query" part, meaning it highlighted each one and asked for yes/no.

 http://www.gnu.org/software/emacs/manual/html_node/emacs/Query-Replace.html

That doesn't sound very efficient. Regardless of that, the current macros still don't work. How do you do cross referencing for something that has a deeper hierarchy and what most of Phobos has? Just take a look at the documentation of std.digest.crc[1]. None of the cross referencing to std.digest.digest.toHexString work. It's due to a misconception of how the $(XREF) works. [1] http://dlang.org/phobos/std_digest_crc.html -- /Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 4:37 AM, Jacob Carlborg wrote:
 That doesn't sound very efficient.

You don't need an efficient solution for ddoc files of readable size. It only takes a few moments. I've spent far, far more time typing about it here than it took to do. Some solutions are simply not worth the bother.
 Regardless of that, the current macros still
 don't work. How do you do cross referencing for something that has a deeper
 hierarchy and what most of Phobos has?

 Just take a look at the documentation of std.digest.crc[1]. None of the cross
 referencing to std.digest.digest.toHexString work. It's due to a
 misconception of how the $(XREF) works.

Using the macros incorrectly is not necessarily an indictment of macros.
Dec 31 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 21:52, Walter Bright wrote:

 You don't need an efficient solution for ddoc files of readable size. It
 only takes a few moments. I've spent far, far more time typing about it
 here than it took to do. Some solutions are simply not worth the bother.

I've probably spent at least as much time on the documentation as on the code for std.serialization/Orange. The documentation could still be improved a lot, like more cross referencing. That's why I started this thread.
 Using the macros incorrectly is not necessarily an indictment of macros.

So which macro should I use to make cross referencing for arbitrary deep hierarchies? -- /Jacob Carlborg
Dec 31 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/31/2013 1:17 PM, Jacob Carlborg wrote:
 So which macro should I use to make cross referencing for arbitrary deep
 hierarchies?

I used this one in std.datetime: LREF2=<a href="#$1">$(D $2)</a>
Dec 31 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-31 23:27, Walter Bright wrote:

 I used this one in std.datetime:

          LREF2=<a href="#$1">$(D $2)</a>

That would require specific knowledge about how Ddoc generate the HTML and file structure. Like replaces dots with an underscore in the module name for the file name. std.datetime becomes std_datetime.htm. Is that wise? -- /Jacob Carlborg
Jan 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/1/2014 4:43 AM, Jacob Carlborg wrote:
 On 2013-12-31 23:27, Walter Bright wrote:

 I used this one in std.datetime:

          LREF2=<a href="#$1">$(D $2)</a>

That would require specific knowledge about how Ddoc generate the HTML and file structure.

Ddoc doesn't generate HTML (the macros do), and the # means it's a local reference, no file structure required.
Jan 01 2014
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 01.01.2014 21:38, schrieb Walter Bright:
 the # means it's a local reference, no file structure required.

A declaration being inside the *same* file is *also* part of the file/directory structure. I could also be in different files. The point is that if this kind of knowledge is encoded in the documentation, or it is tied to that particular structure. This is the case for the Phobos documentation and is the reason why the DDOX based docs have to completely ignore the manual cross references.
Jan 02 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-01 21:38, Walter Bright wrote:

 Ddoc doesn't generate HTML (the macros do), and the # means it's a local
 reference, no file structure required.

Right, I failed to see that. But then that macro won't work for symbols in other modules. The file structure I was referring to was that for the Phobos documentation the HTML files are created with the same name as the fully qualified module name but with dots replaced with underscores. If I want to create a link to a symbol in an other module I need to know about that. -- /Jacob Carlborg
Jan 02 2014
prev sibling next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 30.12.2013 20:08, schrieb Walter Bright:
 On 12/29/2013 9:38 AM, Jacob Carlborg wrote:
 A. Automatic cross reference

 Automatically create links for all matching symbols in the current
 scope. That
 means to create a link to a symbol in the current scope it's enough to
 mention
 it's name in the documentation.

This will create many unwanted links, in three scenarios: 1. In the description for S, there should not be hyperlinks to S. These would be annoying - when you click on them, nothing will happen.

This one can easily be implemented by the documentation generator.
 
 2. Using the word S as a word, not in reference to symbol S, would
 generate a hyperlink which would not make sense.

The _S scheme can be used to avoid the hyperlink.
 
 3. If there are multiple symbols S, hyperlinks to the wrong one would be
 created. This is worse than useless.

If the same lookup rules are used as for the D code itself, the results should turn out pretty much as expected. DDOX currently doesn't follow imports correctly, yet, but even with just a simple hierarchical lookup in the current scope and a fallback to a global lookup (across all symbols) this seems to work pretty well.
 
 In my adding of such hyperlinks, 1..3 happen regularly, this is not a
 "may" happen. It's a "will" happen, a lot.
 

It will happen, but you have to weigh disabling these places manually against having to add an explicit link every time. From what I've seen so far, the latter means a *lot* more work. Personally, I'd prefer an explicit, but unobtrusive, marker like Doxygen uses (e.g. "func()", "::func" or "#func" are recognized as cross references). But since the _ rule is already there, it seems to be a natural choice.
Dec 30 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/30/2013 12:33 PM, S├Ânke Ludwig wrote:
 [...]

I've pretty much replied to these points in my response to Jacob and Andrei.
Dec 30 2013
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 1 January 2014 at 20:38:43 UTC, Walter Bright wrote:
 Ddoc doesn't generate HTML (the macros do)

Ddoc doesn't generate HTML at all, not even with macros. /// Returns true iff a > b bool greaterThan(int a, int b); Ddoc does /not/ do the right thing there. The best you can do is write $(GT), which is stupid and insufficient or &gt; which is wrong.
Jan 01 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 00:13:41 +0100
schrieb S=C3=B6nke Ludwig <sludwig outerproduct.org>:

 [...] I very much agree with them, but the
 current reality looks like the opposite - lots of macro-pseudo-HTML
 markup, very difficult to read unprocessed in many places. So something
 like using "#" or "()" to mark symbols + automatic cross referencing for
 marked symbols would be a *really* nice to have.

I am VERY much in favor of a more MarkDown style syntax and # for identifier references! It is also very verbose to create bullet-point lists and you always have to double-check you don't forget to replace all ) with $(RPAREN). If you misspell anything it is lost in the output. --=20 Marco
Dec 31 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 10:28:46 +0100
schrieb S=C3=B6nke Ludwig <sludwig outerproduct.org>:

 Am 31.12.2013 05:39, schrieb Andrei Alexandrescu:
 On 12/30/13 1:00 PM, Walter Bright wrote:
 On 12/30/2013 8:23 AM, Andrei Alexandrescu wrote:
 Oh the default is to link and then disable manually? That's still
 manual, just
 the default is different :o). I can work with that.

As I mentioned elsewhere, I dislike the requirement to use the inverse feature. Consider maintenance - I add a symbol, and then have to grep the rest of the module for any inadvertent collisions? Features should be positive ones, not negative.

Your arguments are convincing. =20 Andrei =20

The argument in isolation is sound, no doubt. But alone it doesn't lead to a solution*. There are a lot of arguments regarding the current situation and if the decision is not to use the existing _negative rule, then all of those should be carefully taken into account. =20 - Visual noise and effort when using $(XREF) style macros - $(XREF) isn't sufficient! A multitude of macros is actually needed - There is no standard set of macros, it's project specific -> The whole documentation needs to be revised when moving code between projects! - Macros need to be carefully crafted so that they don't encode knowledge about the final documentation/file structure - Macros alone can't cope with nested directory structures - The cognitive load for multiple such macros would very likely cause them to go unused in many cases =20 My favorite solution would be to introduce a simple #identifier.chain or identifier.chain syntax to let the compiler or documentation generator insert the proper links or macro calls. Using a sepecial $(REF identifier.chain) pseudo macro, like Jacob suggested, would work as well, but I can't see how the additional syntax/visual/typing overhead is justified for such a common feature - at least when trying to find an optimal solution, compared to the current macro based system it would of course still be a big improvement. =20 But still, considering that there is precedence for the _negative approach (i.e. the problems of it are already there), I find using that acceptable, too, even if the concept is not as pretty/clean. =20 =20 * *Some* solution is needed. The different structure of the DDOX documentation for example makes the $(XREF) macros worthless, so they are currently completely disabled and automatic link generation is used instead.

I'm fully with you on that one and I would like to add that DDoc's macro system reverses the responsibilities. Instead of writing easy to read documentation that is agnostic to the target output, you have to take into account any potential output format's peculiarities. This starts with the list of escaped characters in your custom template and ends with structuring elements for lists and tables. Without abstraction of structuring elements in DDoc lists and tables look like HTML and are likely not portable to other formats that use a different or implicit structuring scheme. E.g. if you wanted to support MarkDown with its explicit numbers for ordered lists, you'd have to replace all $(LI) with $(LI1), $(LI2), ... I guess it is a little too late to ask for a complete reimplementation of DDoc? --=20 Marco
Dec 31 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 31 Dec 2013 13:27:17 +0100
schrieb Jacob Carlborg <doob me.com>:

 On 2013-12-31 10:31, Marco Leise wrote:
 
 I am VERY much in favor of a more MarkDown style syntax and #
 for identifier references! It is also very verbose to create
 bullet-point lists and you always have to double-check you
 don't forget to replace all ) with $(RPAREN). If you misspell
 anything it is lost in the output.

I completely agree. I suggested the $(REF) macro as an alternative to automatically add cross referencing because I thought that would have a greater chance of being accepted.

Yes that's more practical, unfortunately. :p -- Marco
Dec 31 2013