www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dlang.org endpoint for permanent links to DIPs

reply Dennis <dkorpel gmail.com> writes:
Given an issue number, getting the URL for that issue is not 
hard. Just put it into this string:
https://issues.dlang.org/show_bug.cgi?id=#####

DIPs, however, have it harder. Early DIPs are at:

https://wiki.dlang.org/DIP##

DIPs above 1000 under review can be found like this:

https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

But after review, it will be moved to sub folders like 
'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
can easily become dead, for example:

https://github.com/dlang/dmd/pull/10817

You can use a permanent link to a commit on GitHub, but those are 
very long and refer to the DIP at that moment, not the most 
recent version.

Can we create an endpoint on dlang.org that redirects to the 
latest version of a DIP? Something like:

https://dips.dlang.org/####
https://www.dlang.org/dips/####

I would try to implement this myself but I'm not an experienced 
web developer, and as far as I can tell only the static content 
of dlang.org is open source, so help from people in charge of the 
dlang.org domain would be appreciated.
Jun 03
next sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:
 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
 can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the most 
 recent version.

 Can we create an endpoint on dlang.org that redirects to the 
 latest version of a DIP? Something like:

 https://dips.dlang.org/####
 https://www.dlang.org/dips/####

 I would try to implement this myself but I'm not an experienced 
 web developer, and as far as I can tell only the static content 
 of dlang.org is open source, so help from people in charge of 
 the dlang.org domain would be appreciated.
Yeah, I fully agree that DIP links breaking when their status changes is a really big problem. I also wrote to Mike about this a while ago. I guess the easiest/quickest option is to add redirects to dlang.org. This can be done by adding "Redirect 301 /dips/1028 https://github.com/.../" https://github.com/dlang/dlang.org/blob/master/.htaccess However, I think this will require quite a bit of manual maintenance and is easy to forget, so the best option is to render the Markdown into HTML automatically at the DIPs repository, which can then continuously be deployed to e.g. dips.dlang.org If someone is interested in doing this little project, it can be done entirely with D, e.g. https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown For deployment with Netlify + D, a netlify.sh and netlify.toml are required. Something like this: cat > netlify.toml << EOF [build] base = "" publish = "out" command = "bash netlify.sh" EOF cat > netlify.sh << 'EOF' DMD_VERSION="2.092.0" BUILD_DIR="out" CURL_FLAGS=(-fsSL --retry 10 --retry-delay 30 --retry-max-time 600 --connect-timeout 5 --speed-time 30 --speed-limit 1024) DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" cd "${DIR}" . "$(curl "${CURL_FLAGS[ ]}" https://dlang.org/install.sh | bash -s install "dmd-${DMD_VERSION}" --activate)" dub EOF
Jun 03
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Wednesday, 3 June 2020 at 13:25:05 UTC, Seb wrote:
 I guess the easiest/quickest option is to add redirects to 
 dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"

 https://github.com/dlang/dlang.org/blob/master/.htaccess
I didn't know about that file, that is very useful. I think manually adding redirects is a great start.
 However, I think this will require quite a bit of manual 
 maintenance and is easy to forget, so the best option is to 
 render the Markdown into HTML automatically at the DIPs 
 repository, which can then continuously be deployed to e.g. 
 dips.dlang.org
It's better than the current situation where every instance of a link to a DIP is a maintenance problem. Would it be hard to automatically update the redirects file based on the DIP respository?
 If someone is interested in doing this little project, it can 
 be done entirely with D, e.g.

 https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown

 For deployment with Netlify + D, a netlify.sh and netlify.toml 
 are required. Something like this:
That sounds a bit more complicated than needed. DDoc now supports markdown, so if they should be hosted on the site, couldn't they simply become .dd files in the dlang.org repository?
Jun 03
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 17:42:26 UTC, Dennis wrote:
 On Wednesday, 3 June 2020 at 13:25:05 UTC, Seb wrote:
 I guess the easiest/quickest option is to add redirects to 
 dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"

 https://github.com/dlang/dlang.org/blob/master/.htaccess
I didn't know about that file, that is very useful. I think manually adding redirects is a great start.
 However, I think this will require quite a bit of manual 
 maintenance and is easy to forget, so the best option is to 
 render the Markdown into HTML automatically at the DIPs 
 repository, which can then continuously be deployed to e.g. 
 dips.dlang.org
It's better than the current situation where every instance of a link to a DIP is a maintenance problem. Would it be hard to automatically update the redirects file based on the DIP respository?
It certainly won't be easy as the tech stack of dlang.org is rather old. With the current setup it would probably need to be a bot that monitors commits to dlang/dips, parses the current DIP list, updates the .htaccess based on this and opens this as a PR if the diff changed. Not super hard, but certainly not an hour's work. Furthermore, a lot of people won't be aware that the links from GitHub can break and might not even know about dips.dlang.org
 If someone is interested in doing this little project, it can 
 be done entirely with D, e.g.

 https://vibed.org/api/vibe.textfilter.markdown/filterMarkdown

 For deployment with Netlify + D, a netlify.sh and netlify.toml 
 are required. Something like this:
That sounds a bit more complicated than needed.
Not sure, it's not that hard to use: https://run.dlang.io/is/zWgHH2 All that's missing is to detect the title, loop over all dips and generate a ToC. Vibe.d's Markdown package is also used to render the package READMEs on code.dlang.org and while it used to not support some features of GitHub-flavored Markdown (GFM) compatibility, this compatibility is not a huge issue these days anymore at least with a recent Vibe.d (see e.g. the recent https://github.com/vibe-d/vibe.d/pull/2446). Though I do admit that using a Markdown package that provides 100% well-tested feature parity with GFM might be an alternative approach, but it wouldn't be dog-feeding Vibe.d + D.
 DDoc now supports markdown, so if they should be hosted on the 
 site, couldn't they simply become .dd files in the dlang.org 
 repository?
Well, the Markdown supported by Ddoc has been limited to "only the useful parts" by Walter. That's why the documentation calls it "Markdown inspired", e.g. https://dlang.org/changelog/2.087.0.html#ddoc_markdown I guess it might work with most DIPs, but if one doesn't even try to have feature parity with normal Markdown folks are going to be very confused (and I'm not even talking about GitHub-flavored Markdown). BTW dlang.org already uses Vibe.d its ddox documentation, so there wouldn't be a big "hurdle" of using it to render a few more pages.
Jun 03
next sibling parent David Gileadi <gileadisNOSPM gmail.com> writes:
On 6/3/20 11:22 AM, Seb wrote:
 Well, the Markdown supported by Ddoc has been limited to "only the 
 useful parts" by Walter. That's why the documentation calls it "Markdown 
 inspired", e.g. https://dlang.org/changelog/2.087.0.html#ddoc_markdown
 I guess it might work with most DIPs, but if one doesn't even try to 
 have feature parity with normal Markdown folks are going to be very 
 confused (and I'm not even talking about GitHub-flavored Markdown).
To be specific, compared to regular Markdown GitHub-flavored Markdown adds support for tables, task list items, strikethrough, bare URLs as autolinks and disallows some raw HTML. Of the above, Ddoc "markdown" supports tables and bare autolinks but not task list items or strikethrough. Compared to regular Markdown, Ddoc doesn't support underscores for emphasis (just asterisks), underline style headings, nor indented code blocks (just fenced code blocks).
Jun 03
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Wednesday, 3 June 2020 at 18:22:34 UTC, Seb wrote:
 With the current setup it would probably need to be a bot that 
 monitors commits to dlang/dips, parses the current DIP list, 
 updates the .htaccess based on this and opens this as a PR if 
 the diff changed. Not super hard, but certainly not an hour's 
 work.
Watching the dlang/dips repository and parsing the DIP list is something that needs to happen regardless of whether you want to change .htaccess or generate .html pages based on it. Because I don't know the underlying machinery, I don't get why the former is hard and the latter is easy.
 Furthermore, a lot of people won't be aware that the links from 
 GitHub can break and might not even know about dips.dlang.org
People who do know can safely use the links in e.g. the DMD source code. People who don't know will not use it, regardless of whether it redirects to GitHub or renders the DIP on the site.
 Though I do admit that using a Markdown package that provides 
 100% well-tested feature parity with GFM might be an 
 alternative approach, but it wouldn't be dog-feeding Vibe.d + D.
Is the Vibe.d server that runs the dlang website open source?
 I guess it might work with most DIPs, but if one doesn't even 
 try to have feature parity with normal Markdown folks are going 
 to be very confused (and I'm not even talking about 
 GitHub-flavored Markdown).
Fair enough.
Jun 03
parent Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 18:57:18 UTC, Dennis wrote:
 On Wednesday, 3 June 2020 at 18:22:34 UTC, Seb wrote:
 With the current setup it would probably need to be a bot that 
 monitors commits to dlang/dips, parses the current DIP list, 
 updates the .htaccess based on this and opens this as a PR if 
 the diff changed. Not super hard, but certainly not an hour's 
 work.
Watching the dlang/dips repository and parsing the DIP list is something that needs to happen regardless of whether you want to change .htaccess or generate .html pages based on it. Because I don't know the underlying machinery, I don't get why the former is hard and the latter is easy.
HTML generation can happen directly at the dlang/dips repository and can easily be pushed to any file storage (like e.g. netlify). When you want to adjust the .htaccess you can't just update the file in-place, but you need to automatically create PRs which (1) need to be reviewed by humans every time (there's no direct merge into master) and (2) you need to make sure GitHub doesn't block your bot account and (3) that there won't ever be a possibility for a merge conflict. Furthermore, it's a lot harder to debug when/why the bot doesn't fire compared to just running a HTML build & deploy script which almost every CI can do.
 Furthermore, a lot of people won't be aware that the links 
 from GitHub can break and might not even know about 
 dips.dlang.org
People who do know can safely use the links in e.g. the DMD source code. People who don't know will not use it, regardless of whether it redirects to GitHub or renders the DIP on the site.
The rendered site could be promoted as the official way by basically moving the ToC to it and let all links point to it. Then it would be the only way almost all people look at DIPs (like Python does with its PEPs).
 Is the Vibe.d server that runs the dlang website open source?
Technically speaking Vibe.d isn't used to run dlang.org. Ddox (Vibe.d) like ddoc generate HTML files which are automatically uploaded to an Apache2 server.
Jun 03
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2020-06-03 15:25, Seb wrote:

 Yeah, I fully agree that DIP links breaking when their status changes is 
 a really big problem. I also wrote to Mike about this a while ago.
 
 I guess the easiest/quickest option is to add redirects to dlang.org.
 This can be done by adding "Redirect 301 /dips/1028 
 https://github.com/.../"
 
 https://github.com/dlang/dlang.org/blob/master/.htaccess
 
 However, I think this will require quite a bit of manual maintenance and 
 is easy to forget, so the best option is to render the Markdown into 
 HTML automatically at the DIPs repository, which can then continuously 
 be deployed to e.g. dips.dlang.org
An alternative would be, for a URL like this: https://dips.dlang.org/1234 Extract everything after the slash, "1234" in this case, and check a list of URLs to redirect to: https://github.com/dlang/DIPs/blob/master/DIPs/archive/DIP1234.md https://wiki.dlang.org/DIP1234 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1234.md https://github.com/dlang/DIPs/blob/master/DIPs/DIP1234.md ... and so on Redirect to the first one that exists. I don't know if that's possible to do directly in Apache, or if an additonal server side process is required to handle this. I remember we did something similar at work for server side caching of web pages. But that checked a local file instead of another URL. That was purely done inside Apache. -- /Jacob Carlborg
Jun 04
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 5 June 2020 at 06:53:13 UTC, Jacob Carlborg wrote:
 On 2020-06-03 15:25, Seb wrote:

 [...]
An alternative would be, for a URL like this: https://dips.dlang.org/1234 Extract everything after the slash, "1234" in this case, and check a list of URLs to redirect to: https://github.com/dlang/DIPs/blob/master/DIPs/archive/DIP1234.md https://wiki.dlang.org/DIP1234 https://github.com/dlang/DIPs/blob/master/DIPs/accepted/DIP1234.md https://github.com/dlang/DIPs/blob/master/DIPs/DIP1234.md ... and so on Redirect to the first one that exists. I don't know if that's possible to do directly in Apache, or if an additonal server side process is required to handle this. I remember we did something similar at work for server side caching of web pages. But that checked a local file instead of another URL. That was purely done inside Apache.
It's already done: https://github.com/dlang/dlang.org/pull/2811 For example: https://dlang.org/dips/1 https://dlang.org/dips/75 https://dlang.org/dips/1018 ;) P.S. Thanks, Dennis!
Jun 05
parent Jacob Carlborg <doob me.com> writes:
On 2020-06-05 12:55, Petar Kirov [ZombineDev] wrote:

 It's already done: https://github.com/dlang/dlang.org/pull/2811
Hmm, I see. I tried to come up with a solution that didn't require manual work to keep the list up to date. I see that all the old ones are redirected to the wiki, although there are a couple of ones on GitHub in the "archived" directory. -- /Jacob Carlborg
Jun 05
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:
 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to DIPs 
 can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the most 
 recent version.
Permanent links to the DIPs are all here: https://github.com/dlang/DIPs/blob/master/DIPs/README.md
Jun 03
parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 3 June 2020 at 14:44:18 UTC, Mike Parker wrote:
 On Wednesday, 3 June 2020 at 12:29:09 UTC, Dennis wrote:
 Given an issue number, getting the URL for that issue is not 
 hard. Just put it into this string:
 https://issues.dlang.org/show_bug.cgi?id=#####

 DIPs, however, have it harder. Early DIPs are at:

 https://wiki.dlang.org/DIP##

 DIPs above 1000 under review can be found like this:

 https://github.com/dlang/DIPs/blob/master/DIPs/DIP####.md

 But after review, it will be moved to sub folders like 
 'accepted', 'rejected' and 'other'. Consequently, links to 
 DIPs can easily become dead, for example:

 https://github.com/dlang/dmd/pull/10817

 You can use a permanent link to a commit on GitHub, but those 
 are very long and refer to the DIP at that moment, not the 
 most recent version.
Permanent links to the DIPs are all here: https://github.com/dlang/DIPs/blob/master/DIPs/README.md
Dennis is talking about the fact that if you link to a DIP under review the link while cease to work once its status changes as the document has been moved to a different folder.
Jun 03
parent Les De Ridder <les lesderid.net> writes:
On Wednesday, 3 June 2020 at 14:48:22 UTC, Seb wrote:
 On Wednesday, 3 June 2020 at 14:44:18 UTC, Mike Parker wrote:
[...]
 Permanent links to the DIPs are all here:

 https://github.com/dlang/DIPs/blob/master/DIPs/README.md
Dennis is talking about the fact that if you link to a DIP under review the link while cease to work once its status changes as the document has been moved to a different folder.
A simple but imperfect solution would be to have a directory in the DIPs repo with symlinks to all DIPs, that get updated when DIPs change status.
Jun 03