www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Broken links continue to exist on major pages on dlang.org

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
The first D-related hit on searching for

std.algorithm

on Google is https://dlang.org/library/std/algorithm.html. That is the 
255th most accessed file on dlang.org (including the .ico, css, .js etc 
many of which are accessed from most other pages).

Within that page, the top table has links to various constituents such 
as std.algorithm.searching, std.algorithm.comparison etc. None of those 
links work.

I recall I reported this a while ago. It either has never been fixed, or 
it became a problem again. Folks, I can't babysit all this stuff. I 
can't babysit all this stuff.

This is important and urgent. Who can get on this and stay on this until 
it's done?

For the future, who can work on a simple tool (or hook us into one of 
the many existing ones) that detects bad links?


Thanks,

Andrei
Jun 02 2016
next sibling parent reply Seb <seb wilzba.ch> writes:
On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:
 The first D-related hit on searching for

 std.algorithm

 on Google is https://dlang.org/library/std/algorithm.html. That 
 is the 255th most accessed file on dlang.org (including the 
 .ico, css, .js etc many of which are accessed from most other 
 pages).

 Within that page, the top table has links to various 
 constituents such as std.algorithm.searching, 
 std.algorithm.comparison etc. None of those links work.

 I recall I reported this a while ago. It either has never been 
 fixed, or it became a problem again. Folks, I can't babysit all 
 this stuff. I can't babysit all this stuff.

 This is important and urgent. Who can get on this and stay on 
 this until it's done?

 For the future, who can work on a simple tool (or hook us into 
 one of the many existing ones) that detects bad links?


 Thanks,

 Andrei
There seems to some confusion between "phobos" (ddoc) and "library" (ddox). Both are publicly available. Afaict the plan is to move over to ddox at some point, because ddoc is way too limited, but (if i am not mistaken) this is blocked by existing bugs in ddoc. Imho we should only host on documentation on dlang.org and mark the other as experimental by a separate subdomain.
 Within that page, the top table has links to various 
 constituents such as std.algorithm.searching, 
 std.algorithm.comparison etc. None of those links work.
The PR already has been merged, but you have to wait for the next release: https://github.com/dlang/phobos/pull/4303
Jun 02 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2016 02:36 PM, Seb wrote:
 Imho we should only host on documentation on dlang.org and mark the
 other as experimental by a separate subdomain.
We're fine as we are, many documentations offer distinct formats. -- Andrei
Jun 02 2016
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:
 The first D-related hit on searching for

 std.algorithm

 on Google is https://dlang.org/library/std/algorithm.html. That 
 is the 255th most accessed file on dlang.org (including the 
 .ico, css, .js etc many of which are accessed from most other 
 pages).
/library/ pages are DDOX pages. That is the "NEW Library Reference Preview". It's not ready for prime time. It should be hidden from Google until it's ready. Here's a quick PR to do that: https://github.com/dlang/dlang.org/pull/1320 I guess no one really feels responsible for this stuff.
 I recall I reported this a while ago. It either has never been 
 fixed, or it became a problem again. Folks, I can't babysit all 
 this stuff. I can't babysit all this stuff.
Meh.
Jun 02 2016
next sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.06.2016 um 20:44 schrieb ag0aep6g:
 /library/ pages are DDOX pages. That is the "NEW Library Reference
 Preview". It's not ready for prime time. It should be hidden from Google
 until it's ready. Here's a quick PR to do that:
 https://github.com/dlang/dlang.org/pull/1320

 I guess no one really feels responsible for this stuff.
Which are the remaining issues that you know of? I've searched bugzilla earlier today and couldn't find anything apart from broken "Improve this page" links for package.d modules.
Jun 02 2016
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 06/02/2016 09:09 PM, Sönke Ludwig wrote:
 Which are the remaining issues that you know of? I've searched bugzilla
 earlier today and couldn't find anything apart from broken "Improve this
 page" links for package.d modules.
The big one I know of is (M)REF_ALTTEXT. We need something better than the ugly parentheses thing we've got. Other than that I don't have any blockers on my mind. We've made good progress. I haven't really looked for issues, though. Maybe there is no point in hiding the pages from Google now. But they should have been hidden before. Then again, we would have had less pressure to fix things then.
Jun 02 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.06.2016 um 21:29 schrieb ag0aep6g:
 On 06/02/2016 09:09 PM, Sönke Ludwig wrote:
 Which are the remaining issues that you know of? I've searched bugzilla
 earlier today and couldn't find anything apart from broken "Improve this
 page" links for package.d modules.
The big one I know of is (M)REF_ALTTEXT. We need something better than the ugly parentheses thing we've got. Other than that I don't have any blockers on my mind. We've made good progress. I haven't really looked for issues, though.
Okay, that should be solved in a minute!
 Maybe there is no point in hiding the pages from Google now. But they
 should have been hidden before. Then again, we would have had less
 pressure to fix things then.
Agreed, unlisting from search engines + making the link to the docs more prominent to keep the pressure would probably have been the best choice.
Jun 02 2016
parent ag0aep6g <anonymous example.com> writes:
On 06/02/2016 09:57 PM, Sönke Ludwig wrote:
 Okay, that should be solved in a minute!
Nice :D
Jun 02 2016
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 06/02/2016 09:09 PM, Sönke Ludwig wrote:
 Which are the remaining issues that you know of?
Next up: MYREF. Example page with missing links: http://dlang.org/library/std/string.html In std.ddoc, MYREF is defined as: MYREF = <a href="#.$1">$(TT $1)</a>$(NBSP) In std-ddox-override.ddoc it's: MYREF = $1 That's no good, of course. But I also don't see a way to define it properly. It would have to be something like this: MYREF = <a href="./$(MODULE)/$1.html">$(TT $1)</a>$(NBSP) where $(MODULE) is the current module name without packages. But: 1) Something like $(MODULE) doesn't seem to exist. (Loosely related: DDOX doesn't emit $(TITLE) properly.) 2) camelCase strikes again. $1 would have to be transformed from fooBar to foo_bar to match DDOX's naming scheme. No can do with macros alone. I think those underscore names may have been a mistakee.
Jun 03 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 03.06.2016 um 11:49 schrieb ag0aep6g:
 On 06/02/2016 09:09 PM, Sönke Ludwig wrote:
 Which are the remaining issues that you know of?
Next up: MYREF. Example page with missing links: http://dlang.org/library/std/string.html In std.ddoc, MYREF is defined as: MYREF = <a href="#.$1">$(TT $1)</a>$(NBSP) In std-ddox-override.ddoc it's: MYREF = $1 That's no good, of course. But I also don't see a way to define it properly. It would have to be something like this: MYREF = <a href="./$(MODULE)/$1.html">$(TT $1)</a>$(NBSP) where $(MODULE) is the current module name without packages. But: 1) Something like $(MODULE) doesn't seem to exist. (Loosely related: DDOX doesn't emit $(TITLE) properly.) 2) camelCase strikes again. $1 would have to be transformed from fooBar to foo_bar to match DDOX's naming scheme. No can do with macros alone. I think those underscore names may have been a mistakee.
Since only 12 modules are affected, probably the easiest way is to use the same approach as std.algorithm and define MYREF in each module separately as: MYREF = $(REF_ALTTEXT $(TT $1), $1, std, xxx)$(NBSP) And then remove the definition from std.ddoc
Jun 03 2016
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 03.06.2016 um 13:47 schrieb Sönke Ludwig:
 Am 03.06.2016 um 11:49 schrieb ag0aep6g:
 On 06/02/2016 09:09 PM, Sönke Ludwig wrote:
 Which are the remaining issues that you know of?
Next up: MYREF. Example page with missing links: http://dlang.org/library/std/string.html In std.ddoc, MYREF is defined as: MYREF = <a href="#.$1">$(TT $1)</a>$(NBSP) In std-ddox-override.ddoc it's: MYREF = $1 That's no good, of course. But I also don't see a way to define it properly. It would have to be something like this: MYREF = <a href="./$(MODULE)/$1.html">$(TT $1)</a>$(NBSP) where $(MODULE) is the current module name without packages. But: 1) Something like $(MODULE) doesn't seem to exist. (Loosely related: DDOX doesn't emit $(TITLE) properly.) 2) camelCase strikes again. $1 would have to be transformed from fooBar to foo_bar to match DDOX's naming scheme. No can do with macros alone. I think those underscore names may have been a mistakee.
Since only 12 modules are affected, probably the easiest way is to use the same approach as std.algorithm and define MYREF in each module separately as: MYREF = $(REF_ALTTEXT $(TT $1), $1, std, xxx)$(NBSP) And then remove the definition from std.ddoc
Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should actually do the job just fine. Those links should have actually worked before until I removed full-text automatic cross-references from DDOX.
Jun 03 2016
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 06/03/2016 01:59 PM, Sönke Ludwig wrote:
 Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
 actually do the job just fine. Those links should have actually worked
 before until I removed full-text automatic cross-references from DDOX.
So DDOX is supposed to link any symbol just by its name, without being fully qualified? I'm not sure if that's good. We've had a couple problems with Ddoc being annoyingly eager in highlighting parameter names and such. This goes even further, doesn't it?
Jun 03 2016
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 3 June 2016 at 12:08:33 UTC, ag0aep6g wrote:
 On 06/03/2016 01:59 PM, Sönke Ludwig wrote:
 Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc 
 should
 actually do the job just fine. Those links should have 
 actually worked
 before until I removed full-text automatic cross-references 
 from DDOX.
So DDOX is supposed to link any symbol just by its name, without being fully qualified?
Only in code blocks (i.e. $(D ...) or `...`).
Jun 03 2016
parent ag0aep6g <anonymous example.com> writes:
On 06/03/2016 02:15 PM, Vladimir Panteleev wrote:
 Only in code blocks (i.e. $(D ...) or `...`).
Makes sense :)
Jun 03 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/3/16 7:59 AM, Sönke Ludwig wrote:
 Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
 actually do the job just fine. Those links should have actually worked
 before until I removed full-text automatic cross-references from DDOX.
This seems good, please try it out and post a PR if it works. -- Andrei
Jun 03 2016
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 03.06.2016 um 14:16 schrieb Andrei Alexandrescu:
 On 6/3/16 7:59 AM, Sönke Ludwig wrote:
 Wait a minute... MYREF = $(D $1) in std-ddox-override.ddoc should
 actually do the job just fine. Those links should have actually worked
 before until I removed full-text automatic cross-references from DDOX.
This seems good, please try it out and post a PR if it works. -- Andrei
I'm already on it: https://github.com/dlang/dlang.org/pull/1324
Jun 03 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2016 02:44 PM, ag0aep6g wrote:
 I guess no one really feels responsible for this stuff.
Vladimir is responsible (I cc'd him with the post and he was quick to reply). He can of course delegate but he's the entry point. All - if you find any issues please report to him. As has become apparent in this thread, Sönke is a great resource as well. Let's keep the band rolling. Thanks! -- Andrei
Jun 02 2016
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu 
wrote:
 Within that page, the top table has links to various 
 constituents such as std.algorithm.searching, 
 std.algorithm.comparison etc. None of those links work.
I fixed it months ago on my fork http://dpldocs.info/std.algorithm (I also removed the "this is a submodule of std.algorithm" since that's a useless sentence anyway, given my autogenerated breadcrumb navigation that makes the submodule relationship clear outside the text) The continued existence of such bugs is a reason I laugh at the idea that ddox is a serious contender btw.
Jun 02 2016
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/2/16 2:44 PM, Adam D. Ruppe wrote:
 (I also removed the "this is a submodule of std.algorithm" since that's
 a useless sentence anyway, given my autogenerated breadcrumb navigation
 that makes the submodule relationship clear outside the text)
http://dpldocs.info/experimental-docs/std.algorithm.html The breadcrumb for std is missing on that page. -Steve
Jun 02 2016
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.06.2016 um 20:44 schrieb Adam D. Ruppe:
 On Thursday, 2 June 2016 at 18:24:30 UTC, Andrei Alexandrescu wrote:
 Within that page, the top table has links to various constituents such
 as std.algorithm.searching, std.algorithm.comparison etc. None of
 those links work.
I fixed it months ago on my fork http://dpldocs.info/std.algorithm (I also removed the "this is a submodule of std.algorithm" since that's a useless sentence anyway, given my autogenerated breadcrumb navigation that makes the submodule relationship clear outside the text) The continued existence of such bugs is a reason I laugh at the idea that ddox is a serious contender btw.
I know you are a hacker by heart and of course you can patch and work around everything. But I'd hate to implement some kind of special case here instead of simply fixing the Phobos sources to use macros that carry proper semantical meaning. Those changes now simply have to get into the latest stable version to appear on the doc pages. Apart from that, this is really the only major known issue left (links to GitHub for package.d modules are still broken and that's about it).
Jun 02 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 2 June 2016 at 19:06:42 UTC, Sönke Ludwig wrote:
 But I'd hate to implement some kind of special case here 
 instead of simply fixing the Phobos sources to use macros that 
 carry proper semantical meaning.
That's actually exactly what I did: https://github.com/dlang/dlang.org/pull/1177 The foundation even got merged into upstream dlang.org, six months ago! Sadly, the follow-up, just about two weeks after I laid the foundation* got stalled out <https://github.com/dlang/phobos/pull/3926> until recently, now a PR based on that one <https://github.com/dlang/phobos/pull/4303> got merged ten days ago. * Cross-linking in ddoc has been remarkably non-trivial until recently, and even so, MREF, REF, and REF_ALTTEXT, while much better than it was before, still aren't ideal. Fixing links is, in theory, one of the easiest tasks people can do, but in practice the plethora of bizarre macros makes it far harder than it should be. Even the REF macro family, which aims to unify them, has hit hurdles: https://github.com/dlang/dlang.org/pull/1257
Jun 02 2016
next sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 02.06.2016 um 21:27 schrieb Adam D. Ruppe:
 On Thursday, 2 June 2016 at 19:06:42 UTC, Sönke Ludwig wrote:
 But I'd hate to implement some kind of special case here instead of
 simply fixing the Phobos sources to use macros that carry proper
 semantical meaning.
That's actually exactly what I did: https://github.com/dlang/dlang.org/pull/1177 The foundation even got merged into upstream dlang.org, six months ago! Sadly, the follow-up, just about two weeks after I laid the foundation* got stalled out <https://github.com/dlang/phobos/pull/3926> until recently, now a PR based on that one <https://github.com/dlang/phobos/pull/4303> got merged ten days ago. * Cross-linking in ddoc has been remarkably non-trivial until recently, and even so, MREF, REF, and REF_ALTTEXT, while much better than it was before, still aren't ideal. Fixing links is, in theory, one of the easiest tasks people can do, but in practice the plethora of bizarre macros makes it far harder than it should be. Even the REF macro family, which aims to unify them, has hit hurdles: https://github.com/dlang/dlang.org/pull/1257
I know and I think I misunderstood your last message. So you actually display the documentation for your own Phobos fork, right? That would make a lot more sense then. This hasn't been an option for the DDOX based documentation for obvious reasons, though, so no reason to laugh at it..
Jun 02 2016
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 2 June 2016 at 19:53:29 UTC, Sönke Ludwig wrote:
 So you actually display the documentation for your own
 Phobos fork, right?
Yeah.
 This hasn't been an option for the DDOX based documentation for 
 obvious reasons, though, so no reason to laugh at it..
Well, I think ddox is a competent implementation (of an iffy design, I don't love ddoc itself), and the fact that it is ahead on search engine rankings shows the page breakdown is more SEO-friendly than stock ddoc, among other advantages. But what I find laughable is the surrounding ecosystem. It got added to the official website without a real support plan: a clear commitment was never made to fix Phobos issues, and nobody knew if we were actually going to transition from ddoc, or live side-by-side, or what. Certainly, nobody knew when, so there was no sense of urgency to fix the Phobos source regardless. It just kinda sat in a dark corner of the website for years with little attention from the Phobos crew. I doubt many of them use it. I feel the same thing is happening with dub by the way: we're told that it is the D package manager and there's code.dlang.org... but where's the follow-up action to back it up? It isn't included with the dmd zip and has very little marketing on the website. There's still a drive to get stuff into Phobos proper, or at least std.experimental, instead of dogfooding the package manager. Sure, there's a link to it on the website, but there seems to be little personal use of it - and thus little drive to improve it - from among the Phobos core themselves.
Jun 02 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2016 04:27 PM, Adam D. Ruppe wrote:
 It got added to the official website without a real support plan: a
 clear commitment was never made to fix Phobos issues, and nobody knew if
 we were actually going to transition from ddoc, or live side-by-side, or
 what. Certainly, nobody knew when, so there was no sense of urgency to
 fix the Phobos source regardless.

 It just kinda sat in a dark corner of the website for years with little
 attention from the Phobos crew. I doubt many of them use it.
Interestingly it came as encouraging and empowering some fledgling work that had compelling things going for it (including but not limited to enthusiastic receipt in this forum), which ironically is exactly what you just asked for. Yes, I do wish /library/ were better and got more TLC. But I'm not sorry for approving it. Andrei
Jun 02 2016
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 2 June 2016 at 20:34:24 UTC, Andrei Alexandrescu 
wrote:
 Interestingly it came as encouraging and empowering some 
 fledgling work that had compelling things going for it 
 (including but not limited to enthusiastic receipt in this 
 forum), which ironically is exactly what you just asked for.
Yes, indeed, it was a good first (and second) step. But further steps are necessary too in order to finish a project. Here's what would have been ideal to me: 1) Someone writes a cool thing. 2) We encourage further exploration and see interest. 3) After deciding there's serious potential, we decide on the end goal, a timeframe, and set the conditions of success. For example: ddox becomes the official documentation generator at the end of the year if there are no major bugs remaining open. 4) We put it on the website and work toward the goal, with all the teams - Phobos, dlang.org, RejectedSoftware, etc., understanding their role. 5) When the goal deadline arrives, if it passes the major bug test, it goes live and we are committed to it going forward. Why this order? First, someone writing the cool thing means we actually have something to sink our teeth into and a de facto champion in the original author. Second, we need to incubate this work and not discourage the author. ddox got a decent go up to here. But then we need to decide what's next - a clear goal, including a due date, gets us all aligned and removes a lot of the uncertainty on the author's side; it is some reassurance that they aren't wasting their time, and encourages outside teams to get onboard. That leads directly into step four, and then step five actually proves that the others were not in vain.
Jun 02 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2016 03:27 PM, Adam D. Ruppe wrote:
 Fixing links is, in theory, one of the easiest tasks people can do, but
 in practice the plethora of bizarre macros makes it far harder than it
 should be.
We'd be greatly helped by some automation here (i.e. pinpoint the wrong links). -- Andrei
Jun 02 2016
parent cym13 <cpicard openmailbox.org> writes:
On Thursday, 2 June 2016 at 20:09:57 UTC, Andrei Alexandrescu 
wrote:
 On 06/02/2016 03:27 PM, Adam D. Ruppe wrote:
 Fixing links is, in theory, one of the easiest tasks people 
 can do, but
 in practice the plethora of bizarre macros makes it far harder 
 than it
 should be.
We'd be greatly helped by some automation here (i.e. pinpoint the wrong links). -- Andrei
I did a (quite long actually) “wget --spider -o ~/wget.log -e robots=off -w 1 -r -p http://dlang.org/” to get a list of unused links. There are 690 of them, you can find the list there [1]. If you need the full log of the scan ask for it by mail, it's about 350000 lines long so I don't feel like pasting it. [1]: http://paste.devys.org/mydalona/raw
Jun 07 2016