www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - code.dlang.org downtime

reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
As you may have already noticed, the main registry server, 
code.dlang.org got unreachable yesterday. This was caused by an old VPS 
of mine getting terminated. The registry had already moved to a 
different server years ago, but, without me realizing it, the DNS entry 
still pointed to the old one, with a "temporary" HTTP proxy forwarding 
to the new server being set up.

By now the DNS entry has been corrected, an up-to-date TLS certificate 
is in place, and the registry is running stable. There are still reports 
of people not being able to access code.dlang.org, which is apparently 
caused by intermediate DNS servers still reporting the old IP address 
and should start working during the next few hours. A temporary 
workaround is to specify --registry=http://31.15.67.41/ on the dub 
command line.

Unfortunately both fallback servers have been down for a while now, so 
that this resulted in a total blackout. I plan to move the main registry 
to a powerful dedicated server in January, which will fix all memory 
resource related issues that sometimes show up, and could then keep the 
current VPS as a relatively reliable fallback server. Both together 
should guarantee virtually 100% uptime, although more fallback servers 
are of course highly desirable.

In addition to that, I plan to separate the repository polling process 
form the web and REST frontend, as the former appears to be the main 
cause for failures (a GC memory leak of some kind and a possibly codegen 
related crash when being compiled with DMD being the two known issues, 
which both need further investigation).
Dec 16 2019
next sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:

 As you may have already noticed, the main registry server, 
 code.dlang.org got unreachable yesterday. This was caused by an 
 old VPS of mine getting terminated. The registry had already 
 moved to a different server years ago, but, without me 
 realizing it, the DNS entry still pointed to the old one, with 
 a "temporary" HTTP proxy forwarding to the new server being set 
 up.

 By now the DNS entry has been corrected, an up-to-date TLS 
 certificate is in place, and the registry is running stable. 
 There are still reports of people not being able to access 
 code.dlang.org, which is apparently caused by intermediate DNS 
 servers still reporting the old IP address and should start 
 working during the next few hours. A temporary workaround is to 
 specify --registry=http://31.15.67.41/ on the dub command line.

 Unfortunately both fallback servers have been down for a while 
 now, so that this resulted in a total blackout. I plan to move 
 the main registry to a powerful dedicated server in January, 
 which will fix all memory resource related issues that 
 sometimes show up, and could then keep the current VPS as a 
 relatively reliable fallback server. Both together should 
 guarantee virtually 100% uptime, although more fallback servers 
 are of course highly desirable.

 In addition to that, I plan to separate the repository polling 
 process form the web and REST frontend, as the former appears 
 to be the main cause for failures (a GC memory leak of some 
 kind and a possibly codegen related crash when being compiled 
 with DMD being the two known issues, which both need further 
 investigation).


yay thanks for fixing this so soon.

In my experience having a background task fetching the whole time 
with vibe.d has nearly always been a bad idea in terms of memory 
for me. These days I started using cronjobs which run every so 
often instead and let the OS do all the memory freeing which 
works a lot better. This also scales a lot better because all 
workers just read/write to the database server and can be 
increased or decreased at any point.

Have you maybe also considered making the package zip downloads a 
separate server? It could be load balanced using nginx as well.
Dec 16 2019
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 16.12.2019 um 12:23 schrieb WebFreak001:

 On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:

 As you may have already noticed, the main registry server, 
 code.dlang.org got unreachable yesterday. This was caused by an old 
 VPS of mine getting terminated. The registry had already moved to a 
 different server years ago, but, without me realizing it, the DNS 
 entry still pointed to the old one, with a "temporary" HTTP proxy 
 forwarding to the new server being set up.

 By now the DNS entry has been corrected, an up-to-date TLS certificate 
 is in place, and the registry is running stable. There are still 
 reports of people not being able to access code.dlang.org, which is 
 apparently caused by intermediate DNS servers still reporting the old 
 IP address and should start working during the next few hours. A 
 temporary workaround is to specify --registry=http://31.15.67.41/ on 
 the dub command line.

 Unfortunately both fallback servers have been down for a while now, so 
 that this resulted in a total blackout. I plan to move the main 
 registry to a powerful dedicated server in January, which will fix all 
 memory resource related issues that sometimes show up, and could then 
 keep the current VPS as a relatively reliable fallback server. Both 
 together should guarantee virtually 100% uptime, although more 
 fallback servers are of course highly desirable.

 In addition to that, I plan to separate the repository polling process 
 form the web and REST frontend, as the former appears to be the main 
 cause for failures (a GC memory leak of some kind and a possibly 
 codegen related crash when being compiled with DMD being the two known 
 issues, which both need further investigation).


 
 yay thanks for fixing this so soon.
 
 In my experience having a background task fetching the whole time with 
 vibe.d has nearly always been a bad idea in terms of memory for me. 
 These days I started using cronjobs which run every so often instead and 
 let the OS do all the memory freeing which works a lot better. This also 
 scales a lot better because all workers just read/write to the database 
 server and can be increased or decreased at any point.
 
 Have you maybe also considered making the package zip downloads a 
 separate server? It could be load balanced using nginx as well.


The zips are currently just redirects to GitHub/Bitbucket, but 
ultimately we should really cache them, if just to keep old versions of 
packages available in case they disappear from the original repository 
for whatever reason.

BTW, it looks like most of the CI failures that are usually attributed 
to the registry are in fact caused by GitHub, probably in combination 
with the borderline short timeout that is currently configured for 
dub/curl. I'd imagine that the timeout must be a frequent problem in 
particular for countries such as China, where the network latency adds a 
few hundred milliseconds on top of the server response time.
Dec 16 2019
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:

 As you may have already noticed, the main registry server, 
 code.dlang.org got unreachable yesterday. This was caused by an 
 old VPS of mine getting terminated. The registry had already 
 moved to a different server years ago, but, without me 
 realizing it, the DNS entry still pointed to the old one, with 
 a "temporary" HTTP proxy forwarding to the new server being set 
 up.

 By now the DNS entry has been corrected, an up-to-date TLS 
 certificate is in place, and the registry is running stable. 
 There are still reports of people not being able to access 
 code.dlang.org, which is apparently caused by intermediate DNS 
 servers still reporting the old IP address and should start 
 working during the next few hours. A temporary workaround is to 
 specify --registry=http://31.15.67.41/ on the dub command line.

 Unfortunately both fallback servers have been down for a while 
 now, so that this resulted in a total blackout. I plan to move 
 the main registry to a powerful dedicated server in January, 
 which will fix all memory resource related issues that 
 sometimes show up, and could then keep the current VPS as a 
 relatively reliable fallback server. Both together should 
 guarantee virtually 100% uptime, although more fallback servers 
 are of course highly desirable.

 In addition to that, I plan to separate the repository polling 
 process form the web and REST frontend, as the former appears 
 to be the main cause for failures (a GC memory leak of some 
 kind and a possibly codegen related crash when being compiled 
 with DMD being the two known issues, which both need further 
 investigation).


Hi Sonke,

Thanks a lot for taking care of the issues so quickly and taking 
actions.
I was a bit sad yesterday not being able to upgrade, but now the 
new registry server is a lot faster here! It makes browsing the 
registry a pleasure.

---------------

For all future readers, here is something you can do in case of 
DUB failure.

What to do if the DUB registry is down:
- Checkout all dependencies manually.

    $ git clone https://hosting.com/mydep.git mydep
    $ cd mydep
    $ git checkout <desired-tag>

- Add those dependencies as local packages:

    $ dub add-local .
    $ cd ..

- Build your project normally. The local packages should override 
the online packages.

    $ dub build


---------------
Dec 16 2019
parent reply Temtaime <temtaime gmail.com> writes:
My ISP still serves old IP.
Thanks for such a blackout.
D is still not for production use, just a toy that may break 
accidentally by a will of its creators.
Dec 16 2019
parent reply bachmeier <no spam.net> writes:
On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:

 My ISP still serves old IP.
 Thanks for such a blackout.
 D is still not for production use, just a toy that may break 
 accidentally by a will of its creators.


I wish I could disagree, but from the view of large enterprise 
adoption, this is indeed a very bad situation.
Dec 16 2019
parent reply Pham <home home.com> writes:
On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:

 On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:

 My ISP still serves old IP.
 Thanks for such a blackout.
 D is still not for production use, just a toy that may break 
 accidentally by a will of its creators.


 I wish I could disagree, but from the view of large enterprise 
 adoption, this is indeed a very bad situation.


Interprise must always use local copy -> never blindly trust the 
internet for reliable!
Dec 16 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 17, 2019 at 12:55:35AM +0000, Pham via Digitalmars-d-announce wrote:

 On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:

 On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:

 My ISP still serves old IP.
 Thanks for such a blackout.
 D is still not for production use, just a toy that may break
 accidentally by a will of its creators.


 
 I wish I could disagree, but from the view of large enterprise
 adoption, this is indeed a very bad situation.


 
 Interprise must always use local copy -> never blindly trust the
 internet for reliable!


+100.  I've never liked the idea that the possibility to build my
projects depends on some random internet server somewhere out there in
the ether that I have no control over.


T

-- 
Never ascribe to malice that which is adequately explained by incompetence. --
Napoleon Bonaparte
Dec 16 2019
prev sibling parent reply bachmeier <no spam.net> writes:
On Tuesday, 17 December 2019 at 00:55:35 UTC, Pham wrote:

 On Monday, 16 December 2019 at 22:43:58 UTC, bachmeier wrote:

 On Monday, 16 December 2019 at 20:21:16 UTC, Temtaime wrote:

 My ISP still serves old IP.
 Thanks for such a blackout.
 D is still not for production use, just a toy that may break 
 accidentally by a will of its creators.


 I wish I could disagree, but from the view of large enterprise 
 adoption, this is indeed a very bad situation.


 Interprise must always use local copy -> never blindly trust 
 the internet for reliable!


Oh, I don't doubt that. My point was that it makes the D language 
project look like a small-scale open source project relying on 
volunteers (in this case Sonke) being generous with time and 
resources. What manager is going to trust a project like that 
with a major project? It seems unlikely that this would not be 
considered a major issue.
Dec 16 2019
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 17 December 2019 at 01:34:16 UTC, bachmeier wrote:

 Oh, I don't doubt that. My point was that it makes the D 
 language project look like a small-scale open source project 
 relying on volunteers (in this case Sonke) being generous with 
 time and resources. What manager is going to trust a project 
 like that with a major project? It seems unlikely that this 
 would not be considered a major issue.


I totally agree. I am very grateful for Sönke for everything he 
has and continues to do, but I believe we should move away from 
self hosting the dub registry. Not only will that increase the 
uptime, but it will also increase the bus factor and take some 
work off Sönke's hands into the d organisation.

In another post I outlined a very crude draft proposal 
https://forum.dlang.org/post/arbvxihnxnboqcifhlbm forum.dlang.org 
Since no one replied I assume everyone is onboard. (just kidding)

The architecture isn't difficult, but the migration path is. In 
February I should have some time and I am willing to setup a 
proof of concept.
Dec 17 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Dec 17, 2019 at 01:34:16AM +0000, bachmeier via Digitalmars-d-announce
wrote:
[...]

 Oh, I don't doubt that. My point was that it makes the D language
 project look like a small-scale open source project relying on
 volunteers (in this case Sonke) being generous with time and
 resources. What manager is going to trust a project like that with a
 major project? It seems unlikely that this would not be considered a
 major issue.


This looks like something the D Foundation ought to fund.  Buy one of
the many VPS's out there to be used as the "official", Foundation-backed
dub server, preferably with a hosting provider that has multiple
redundant network connections and automatic backup service.  Depending
on availability of funds, add a hot failover server as backup in case
things go wrong with the primary server.  This ought to alleviate any
concerns about downtime.

The current volunteer-provided servers can still act as mirrors for
extra availability.

If a volunteer-driven opensource project like Debian can have worldwide
multiple redundant FTP mirrors that almost guarantees 100% uptime, why
can't we?


T

-- 
INTEL = Only half of "intelligence".
Dec 17 2019
next sibling parent sarn <sarn theartofmachinery.com> writes:
On Tuesday, 17 December 2019 at 17:34:07 UTC, H. S. Teoh wrote:

 On Tue, Dec 17, 2019 at 01:34:16AM +0000, bachmeier via 
 Digitalmars-d-announce wrote: [...]

 Oh, I don't doubt that. My point was that it makes the D 
 language project look like a small-scale open source project 
 relying on volunteers (in this case Sonke) being generous with 
 time and resources. What manager is going to trust a project 
 like that with a major project? It seems unlikely that this 
 would not be considered a major issue.


 This looks like something the D Foundation ought to fund.
 ...
 If a volunteer-driven opensource project like Debian can have 
 worldwide multiple redundant FTP mirrors that almost guarantees 
 100% uptime, why can't we?


I agree.  I sent an email to the Foundation in October offering 
to help fund some proper infrastructure.  Maybe the email got 
lost.  The offer still stands.
Dec 17 2019
prev sibling parent reply James Blachly <james.blachly gmail.com> writes:
On 12/17/19 12:34 PM, H. S. Teoh wrote:

 This looks like something the D Foundation ought to fund.  Buy one of
 the many VPS's out there to be used as the "official", Foundation-backed
 dub server, preferably with a hosting provider that has multiple
 redundant network connections and automatic backup service.  Depending
 on availability of funds, add a hot failover server as backup in case
 things go wrong with the primary server.  This ought to alleviate any
 concerns about downtime.
 ...


I am sorry to post on an older (but < 30 days!) thread that I've just seen.

While it would be very reasonable for the Foundation to be running this 
infra, Sebastiaan's proposal is quite right -- eliminate the need for a 
server entirely.

I would add one thing to his proposal:
The metadata are already stored in github/bitbucket/gitlab/etc. -- in 
the form of the dubfile. All we need is a K:V lookup mapping package 
name to canonical git repository. This K:V store could be served perhaps 
more cheaply , and more reliably, than the current infra which must 
serve all metadata (presumably). If dub could be retooled to only query 
for the git repo, it could then clone and work entirely with metadata 
stored in the repo, minimizing dependence on a hosted server, whether 6 
nines reliable or not.

If, though, the foundation takes over the current infra and hosting, 
please put out a call for donations -- I will donate.
Jan 09 2020
parent reply suncarpet <suncarpet firemail.cc> writes:
On Friday, 10 January 2020 at 02:36:51 UTC, James Blachly wrote:

 I am sorry to post on an older (but < 30 days!) thread that 
 I've just seen.

 While it would be very reasonable for the Foundation to be 
 running this infra, Sebastiaan's proposal is quite right -- 
 eliminate the need for a server entirely.

 I would add one thing to his proposal:
 The metadata are already stored in github/bitbucket/gitlab/etc. 
 -- in the form of the dubfile. All we need is a K:V lookup 
 mapping package name to canonical git repository. This K:V 
 store could be served perhaps more cheaply , and more reliably, 
 than the current infra which must serve all metadata 
 (presumably). If dub could be retooled to only query for the 
 git repo, it could then clone and work entirely with metadata 
 stored in the repo, minimizing dependence on a hosted server, 
 whether 6 nines reliable or not.

 If, though, the foundation takes over the current infra and 
 hosting, please put out a call for donations -- I will donate.


I can't say I like the idea of requiring users to use the 
aforementioned or similar services and necessitating the use of 
git over other VCS solutions.
Jan 09 2020
parent reply James Blachly <james.blachly gmail.com> writes:
On 1/9/20 10:03 PM, suncarpet wrote:

 I can't say I like the idea of requiring users to use the aforementioned 
 or similar services and necessitating the use of git over other VCS 
 solutions.


How is it that dub works today?
Jan 09 2020
parent reply suncarpet <suncarpet firemail.cc> writes:
On Friday, 10 January 2020 at 03:14:29 UTC, James Blachly wrote:

 On 1/9/20 10:03 PM, suncarpet wrote:

 I can't say I like the idea of requiring users to use the 
 aforementioned or similar services and necessitating the use 
 of git over other VCS solutions.


 How is it that dub works today?


It works such that I don't post anything to code.dlang.org, lol. 
(Not that I would because all my stuff is very niche.) If things 
were to change, I'd like for restrictions on where code is hosted 
and the VCS to be loosened. I may be looking at this very 
naively, however.
Jan 09 2020
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 10/01/2020 4:37 PM, suncarpet wrote:

 If things were to change, I'd like for restrictions on where code is 
 hosted and the VCS to be loosened. I may be looking at this very 
 naively, however.


You are.

It is an implementation detail, not policy.

To do it 100% generically dub repository will have to understand each 
VCS system, check it out and then archive it before shipping it off on 
request. Instead of just downloading an archive from the provider.

You may not know this, but in the past the repository has had problems 
crashing from doing these updates and then performing downloads (its 
been resolved, but it really doesn't have the infrastructure to do it 
properly I think).
Jan 09 2020
parent suncarpet <suncarpet firemail.cc> writes:
On Friday, 10 January 2020 at 04:06:28 UTC, rikki cattermole 
wrote:

 You are.

 It is an implementation detail, not policy.

 To do it 100% generically dub repository will have to 
 understand each VCS system, check it out and then archive it 
 before shipping it off on request. Instead of just downloading 
 an archive from the provider.

 You may not know this, but in the past the repository has had 
 problems crashing from doing these updates and then performing 
 downloads (its been resolved, but it really doesn't have the 
 infrastructure to do it properly I think).


And this is why I dig ditches for a living. Thank you for the 
(what should've been obvious) explanation.
Jan 09 2020
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 16 December 2019 at 11:04:38 UTC, Sönke Ludwig wrote:

 As you may have already noticed, the main registry server, 
 code.dlang.org got unreachable yesterday. This was caused by an 
 old VPS of mine getting terminated. The registry had already 
 moved to a different server years ago, but, without me 
 realizing it, the DNS entry still pointed to the old one, with 
 a "temporary" HTTP proxy forwarding to the new server being set 
 up.

 By now the DNS entry has been corrected, an up-to-date TLS 
 certificate is in place, and the registry is running stable. 
 There are still reports of people not being able to access 
 code.dlang.org, which is apparently caused by intermediate DNS 
 servers still reporting the old IP address and should start 
 working during the next few hours. A temporary workaround is to 
 specify --registry=http://31.15.67.41/ on the dub command line.

 Unfortunately both fallback servers have been down for a while 
 now, so that this resulted in a total blackout. I plan to move 
 the main registry to a powerful dedicated server in January, 
 which will fix all memory resource related issues that 
 sometimes show up, and could then keep the current VPS as a 
 relatively reliable fallback server. Both together should 
 guarantee virtually 100% uptime, although more fallback servers 
 are of course highly desirable.

 In addition to that, I plan to separate the repository polling 
 process form the web and REST frontend, as the former appears 
 to be the main cause for failures (a GC memory leak of some 
 kind and a possibly codegen related crash when being compiled 
 with DMD being the two known issues, which both need further 
 investigation).


This is still down for me, regardless of using the IP or address. 
I don't think it's just me either: 
https://stats.uptimerobot.com/6mQX4Crw2L/783838659
Dec 18 2019
next sibling parent drug <drug2004 bk.ru> writes:
On 12/18/19 12:29 PM, John Colvin wrote:

 
 This is still down for me, regardless of using the IP or address. I 
 don't think it's just me either: 
 https://stats.uptimerobot.com/6mQX4Crw2L/783838659


The same with me. I can't visit code.dlang.org from home pc, but able to 
do it from work one. (I did it in different times though)
Dec 18 2019
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 18 December 2019 at 09:29:50 UTC, John Colvin wrote:

 This is still down for me, regardless of using the IP or 
 address. I don't think it's just me either: 
 https://stats.uptimerobot.com/6mQX4Crw2L/783838659


Anytime you see the metadata working you can add 
`--registry=https://dub.bytecraft.nl` to dub.

I am really tempted to cache the metadata as well. Although that 
brings up the question of how to purge it when new versions are 
released. (I could setup a job to import the dump every now and 
then, hmm).

This stuff just begs to be fixed.
Dec 18 2019
parent John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 18 December 2019 at 10:18:03 UTC, Sebastiaan Koppe 
wrote:

 On Wednesday, 18 December 2019 at 09:29:50 UTC, John Colvin 
 wrote:

 This is still down for me, regardless of using the IP or 
 address. I don't think it's just me either: 
 https://stats.uptimerobot.com/6mQX4Crw2L/783838659


 Anytime you see the metadata working you can add 
 `--registry=https://dub.bytecraft.nl` to dub.

 I am really tempted to cache the metadata as well. Although 
 that brings up the question of how to purge it when new 
 versions are released. (I could setup a job to import the dump 
 every now and then, hmm).

 This stuff just begs to be fixed.


can't get metadata, so no good right now.
Dec 18 2019
prev sibling parent reply Andrea Fontana <nospam example.com> writes:
On Wednesday, 18 December 2019 at 09:29:50 UTC, John Colvin wrote:

 This is still down for me, regardless of using the IP or 
 address. I don't think it's just me either: 
 https://stats.uptimerobot.com/6mQX4Crw2L/783838659


https://downforeveryoneorjustme.com/code.dlang.org

Connection timeout here.
Dec 18 2019
parent Andrea Fontana <nospam example.com> writes:
On Wednesday, 18 December 2019 at 11:03:04 UTC, Andrea Fontana 
wrote:

 On Wednesday, 18 December 2019 at 09:29:50 UTC, John Colvin 
 wrote:

 This is still down for me, regardless of using the IP or 
 address. I don't think it's just me either: 
 https://stats.uptimerobot.com/6mQX4Crw2L/783838659


 https://downforeveryoneorjustme.com/code.dlang.org

 Connection timeout here.


Just said. Up&Running now.
Dec 18 2019