www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - code.dlang.org major outage

reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
The virtual server that is running code.dlang.org has frozen about an 
hour ago and fails to boot further than to the bootloader. Initial 
attempts to recover from within Grub have failed and it's unclear what 
the root cause is. I will instead set up a replacement server with the 
latest backup from 20 hours ago as soon as it becomes available.

It's really about time to install a permanent backup server to keep the 
registry available in such cases. I will try my best to allocate time to 
make this happen, but I'm severely constrained currently and any helping 
hand would make a big difference. What needs to be implemented is a mode 
for the DUB registry [1] that works without the userman database and 
instead just polls code.dlang.org for changes (may also require some new 
REST API end points to be able to acquire all needed data).

[1]: https://github.com/dlang/dub-registry/
Feb 23
next sibling parent Moritz Maxeiner <moritz ucworks.org> writes:
On Thursday, 23 February 2017 at 22:55:05 UTC, Sönke Ludwig wrote:
 The virtual server that is running code.dlang.org has frozen 
 about an hour ago and fails to boot further than to the 
 bootloader. Initial attempts to recover from within Grub have 
 failed and it's unclear what the root cause is. I will instead 
 set up a replacement server with the latest backup from 20 
 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to 
 keep the registry available in such cases. I will try my best 
 to allocate time to make this happen, but I'm severely 
 constrained currently and any helping hand would make a big 
 difference. What needs to be implemented is a mode for the DUB 
 registry [1] that works without the userman database and 
 instead just polls code.dlang.org for changes (may also require 
 some new REST API end points to be able to acquire all needed 
 data).

 [1]: https://github.com/dlang/dub-registry/
Thank you for the information. Not sure if this helps, but if you ever want to failover code.dlang.org onto multiple servers (so that if this happens on one of them others can takeover transparently) I can offer an IPv6 LXD instance.
Feb 23
prev sibling next sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 23.02.2017 um 23:55 schrieb Sönke Ludwig:
 The virtual server that is running code.dlang.org has frozen about an
 hour ago and fails to boot further than to the bootloader. Initial
 attempts to recover from within Grub have failed and it's unclear what
 the root cause is. I will instead set up a replacement server with the
 latest backup from 20 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to keep the
 registry available in such cases. I will try my best to allocate time to
 make this happen, but I'm severely constrained currently and any helping
 hand would make a big difference. What needs to be implemented is a mode
 for the DUB registry [1] that works without the userman database and
 instead just polls code.dlang.org for changes (may also require some new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
The system is estimated to be back up at 9:00 UTC (1:00 PST) (AM).
Feb 23
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 24.02.2017 um 01:46 schrieb Sönke Ludwig:
 Am 23.02.2017 um 23:55 schrieb Sönke Ludwig:
 The virtual server that is running code.dlang.org has frozen about an
 hour ago and fails to boot further than to the bootloader. Initial
 attempts to recover from within Grub have failed and it's unclear what
 the root cause is. I will instead set up a replacement server with the
 latest backup from 20 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to keep the
 registry available in such cases. I will try my best to allocate time to
 make this happen, but I'm severely constrained currently and any helping
 hand would make a big difference. What needs to be implemented is a mode
 for the DUB registry [1] that works without the userman database and
 instead just polls code.dlang.org for changes (may also require some new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
The system is estimated to be back up at 9:00 UTC (1:00 PST) (AM).
Running again now. Minor outages are still expected while the rest of the system is set up.
Feb 24
parent reply Eugene Wissner <belka caraus.de> writes:
On Friday, 24 February 2017 at 08:50:12 UTC, Sönke Ludwig wrote:
 Am 24.02.2017 um 01:46 schrieb Sönke Ludwig:
 Am 23.02.2017 um 23:55 schrieb Sönke Ludwig:
 The virtual server that is running code.dlang.org has frozen 
 about an
 hour ago and fails to boot further than to the bootloader. 
 Initial
 attempts to recover from within Grub have failed and it's 
 unclear what
 the root cause is. I will instead set up a replacement server 
 with the
 latest backup from 20 hours ago as soon as it becomes 
 available.

 It's really about time to install a permanent backup server 
 to keep the
 registry available in such cases. I will try my best to 
 allocate time to
 make this happen, but I'm severely constrained currently and 
 any helping
 hand would make a big difference. What needs to be 
 implemented is a mode
 for the DUB registry [1] that works without the userman 
 database and
 instead just polls code.dlang.org for changes (may also 
 require some new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
The system is estimated to be back up at 9:00 UTC (1:00 PST) (AM).
Running again now. Minor outages are still expected while the rest of the system is set up.
travis fails: $ source "$(CURL_USER_AGENT="$CURL_USER_AGENT" bash install.sh dmd-2.072.1 --activate)" curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 Failed to download 'http://code.dlang.org/download/LATEST' /home/travis/build.sh: line 62: : No such file or directory http://code.dlang.org/download/ is empty.
Feb 24
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 24.02.2017 um 10:01 schrieb Eugene Wissner:
 On Friday, 24 February 2017 at 08:50:12 UTC, Sönke Ludwig wrote:
 Am 24.02.2017 um 01:46 schrieb Sönke Ludwig:
 Am 23.02.2017 um 23:55 schrieb Sönke Ludwig:
 The virtual server that is running code.dlang.org has frozen about an
 hour ago and fails to boot further than to the bootloader. Initial
 attempts to recover from within Grub have failed and it's unclear what
 the root cause is. I will instead set up a replacement server with the
 latest backup from 20 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to keep the
 registry available in such cases. I will try my best to allocate
 time to
 make this happen, but I'm severely constrained currently and any
 helping
 hand would make a big difference. What needs to be implemented is a
 mode
 for the DUB registry [1] that works without the userman database and
 instead just polls code.dlang.org for changes (may also require some
 new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
The system is estimated to be back up at 9:00 UTC (1:00 PST) (AM).
Running again now. Minor outages are still expected while the rest of the system is set up.
travis fails: $ source "$(CURL_USER_AGENT="$CURL_USER_AGENT" bash install.sh dmd-2.072.1 --activate)" curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 curl: (22) The requested URL returned error: 404 Failed to download 'http://code.dlang.org/download/LATEST' /home/travis/build.sh: line 62: : No such file or directory http://code.dlang.org/download/ is empty.
Should be fixed now. The downloads were uploaded a bit later.
Feb 24
prev sibling next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 24/02/2017 11:55 AM, Sönke Ludwig wrote:
 The virtual server that is running code.dlang.org has frozen about an
 hour ago and fails to boot further than to the bootloader. Initial
 attempts to recover from within Grub have failed and it's unclear what
 the root cause is. I will instead set up a replacement server with the
 latest backup from 20 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to keep the
 registry available in such cases. I will try my best to allocate time to
 make this happen, but I'm severely constrained currently and any helping
 hand would make a big difference. What needs to be implemented is a mode
 for the DUB registry [1] that works without the userman database and
 instead just polls code.dlang.org for changes (may also require some new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
Another thing I would like us to have is a local cache of all packages known, so that when outages do happen it can be managed ok-ish for how are up to date.
Feb 23
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 23.02.2017 um 23:55 schrieb Sönke Ludwig:
 The virtual server that is running code.dlang.org has frozen about an
 hour ago and fails to boot further than to the bootloader. Initial
 attempts to recover from within Grub have failed and it's unclear what
 the root cause is. I will instead set up a replacement server with the
 latest backup from 20 hours ago as soon as it becomes available.

 It's really about time to install a permanent backup server to keep the
 registry available in such cases. I will try my best to allocate time to
 make this happen, but I'm severely constrained currently and any helping
 hand would make a big difference. What needs to be implemented is a mode
 for the DUB registry [1] that works without the userman database and
 instead just polls code.dlang.org for changes (may also require some new
 REST API end points to be able to acquire all needed data).

 [1]: https://github.com/dlang/dub-registry/
I used the waiting time during the backup upload to prepare a PR for a mirror mode: https://github.com/dlang/dub-registry/pull/187 Would be good if someone could have a look. Once this is merged, anyone can run this on a server using `./dub-registry --mirror https://code.dlang.org/` and then forward incoming requests using a reverse proxy to http://127.0.0.1:8005. In the current form, this will enable manual fallback in the form of users specifying --registry=https://some.code.dlang.mirror on the DUB command line. In the future we can then keep a list of official mirrors as DNS entries (maybe of the form mirror1.dub.pm) and let DUB use those automatically.
Feb 24
next sibling parent reply Moritz Maxeiner <moritz ucworks.org> writes:
On Friday, 24 February 2017 at 11:22:34 UTC, Sönke Ludwig wrote:
 [...]

 Would be good if someone could have a look. Once this is 
 merged, anyone can run this on a server using `./dub-registry 
 --mirror https://code.dlang.org/` and then forward incoming 
 requests using a reverse proxy to http://127.0.0.1:8005.
To be clear: Would the argument need to be specified for every launch, or is this a one-time call to set it up for future mirroring? Is the mirrored data stored in a permanent location?
 In the current form, this will enable manual fallback in the 
 form of users specifying 
 --registry=https://some.code.dlang.mirror on the DUB command 
 line. In the future we can then keep a list of official mirrors 
 as DNS entries (maybe of the form mirror1.dub.pm) and let DUB 
 use those automatically.
Cool, I'll see about setting one up when a new release of dub-registry with that is out.
Feb 24
parent =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig outerproduct.org> writes:
Am 24.02.2017 um 14:50 schrieb Moritz Maxeiner:
 On Friday, 24 February 2017 at 11:22:34 UTC, Sönke Ludwig wrote:
 [...]

 Would be good if someone could have a look. Once this is merged,
 anyone can run this on a server using `./dub-registry --mirror
 https://code.dlang.org/` and then forward incoming requests using a
 reverse proxy to http://127.0.0.1:8005.
To be clear: Would the argument need to be specified for every launch, or is this a one-time call to set it up for future mirroring? Is the mirrored data stored in a permanent location?
On every launch, there is already a configuration file, so adding support for storing it there could be an alternative. The package database is stored in a MongoDB instance on localhost currently (creates a "vpmreg" collection).
 In the current form, this will enable manual fallback in the form of
 users specifying --registry=https://some.code.dlang.mirror on the DUB
 command line. In the future we can then keep a list of official
 mirrors as DNS entries (maybe of the form mirror1.dub.pm) and let DUB
 use those automatically.
Cool, I'll see about setting one up when a new release of dub-registry with that is out.
Feb 24
prev sibling parent Seb <seb wilzba.ch> writes:
On Friday, 24 February 2017 at 11:22:34 UTC, Sönke Ludwig wrote:
 I used the waiting time during the backup upload to prepare a 
 PR for a mirror mode: 
 https://github.com/dlang/dub-registry/pull/187

 Would be good if someone could have a look. Once this is 
 merged, anyone can run this on a server using `./dub-registry 
 --mirror https://code.dlang.org/` and then forward incoming 
 requests using a reverse proxy to http://127.0.0.1:8005.

 In the current form, this will enable manual fallback in the 
 form of users specifying 
 --registry=https://some.code.dlang.mirror on the DUB command 
 line. In the future we can then keep a list of official mirrors 
 as DNS entries (maybe of the form mirror1.dub.pm) and let DUB 
 use those automatically.
Awesome work! I setup a testing instance at alpha.dub.pm, so in case someone wants to give it a try with DUB directly, you can use sth. like
 dub fetch --registry=http://alpha.dub.pm/ -v mir-algorithm
Feb 24