www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dub 502 bad gateway

reply Sebastiaan Koppe <mail skoppe.eu> writes:
After having had another day where my CI can't build because of 
gateway errors, this morning I had enough. (I setup an uptime 
robot so we can see how bad it actually is [1], it checks every 5 
min and I just started it.)

I know, I can setup my CI to cache dub packages. But that is just 
an optimisation.

Things ought to just work.

So this morning I started on a quick fix. I took a look at the 
source and concluded that for dub packages hosted on gitlab and 
github you can simply redirect to gitlab's and github's 
respective archives. That is what the dub registry does anyway.

To that end I setup that logic in a cloudflare worker [2]. So for 
metadata it redirects to the api endpoint on code.dlang.org 
(which is pretty stable), and for to actual packages directly to 
gitlab or github.

Use it like so:

`dub build --registry=https://dub.bytecraft.nl`

Of course, this is a bandaid solution. Probably dub can be made 
to do this as well. And likely should. If it were up to me I 
would go serverless.

[1] https://stats.uptimerobot.com/6mQX4Crw2L
[2] https://github.com/skoppe/dub-registry-mirror
Nov 19 2019
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 19 November 2019 at 13:27:31 UTC, Sebastiaan Koppe 
wrote:
 After having had another day where my CI can't build because of 
 gateway errors, this morning I had enough. (I setup an uptime 
 robot so we can see how bad it actually is [1], it checks every 
 5 min and I just started it.)

 [...]

 Of course, this is a bandaid solution. Probably dub can be made 
 to do this as well. And likely should. If it were up to me I 
 would go serverless.

 [1] https://stats.uptimerobot.com/6mQX4Crw2L
 [2] https://github.com/skoppe/dub-registry-mirror
I have updated the script to cache all routes. This means the registry mirror at dub.bytecraft.nl will also work when the registry at code.dlang.org is completely down, **if** your dependencies have been cached previously. The cache works in the following manner: 1. if a cache entry exists for a given route and it is no longer than 5 minutes old: return that 2. if a cache entry is older than 5 minutes: fetch from upstream, cache it and return that 3. if a cache entry is older than 5 minutes and upstream is down: reset cache expiry and return from cache 4. if a cache entry doesn't exist: fetch from upstream, cache it and return that 5. if a cache entry doesn't exist and upstream is down: return error On top of that there is a 5 second timeout on fetches from upstream. Essentially a cached entry is kept up-to-date (5 minute lag) whenever upstream is alive, otherwise stale data is returned if available. Use it like so: `dub build --registry=https://dub.bytecraft.nl` Code at https://github.com/skoppe/dub-registry-mirror/blob/master/worker.js
Jan 30
prev sibling parent Robert Schadek <rschadek symmetryinvestments.com> writes:
very nice, thank you
Jan 30