www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - code.dlang.org reliability update

reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
As of yesterday, code.dlang.org now points to a more powerful dedicated 
server that can host the DUB registry without the danger of freezing due 
to excessive swapping - this is what happened on the 26th last month [1].

In addition to that, the server that previously hosted the registry is 
now used to run an official mirror, reachable at codemirror.dlang.org. 
This will be configured as a built-in fallback server starting with DMD 
2.091.0/DUB 1.12.0 and, at least in theory, will lead to an uptime of 
virtually 100%.

To make use of the mirror today, it is also possible to configure it in 
DUB's settings.json as a custom registry:

{
    "registryUrls": ["https://codemirror.dlang.org/"]
}

settings.json is found/needs to be created in %APPDATA%\dub\ on Windows 
and in ~/.dub/ on all other systems. The custom entry should be removed 
once DUB 1.12.0 is used, to avoid redundant requests in certain situations.


[1]: https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org
Mar 02
next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].

 In addition to that, the server that previously hosted the 
 registry is now used to run an official mirror, reachable at 
 codemirror.dlang.org. This will be configured as a built-in 
 fallback server starting with DMD 2.091.0/DUB 1.12.0 and, at 
 least in theory, will lead to an uptime of virtually 100%.
Excellent, thank you! Bastiaan.
Mar 02
prev sibling next sibling parent reply MrSmith <mrsmith33 yandex.ru> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:

 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows
I guess that should be %APPDATA%\Local\dub\
Mar 02
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 02.03.2020 um 23:50 schrieb MrSmith:
 On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 
 settings.json is found/needs to be created in %APPDATA%\dub\ on Windows
I guess that should be %APPDATA%\Local\dub\
Nope, should be right. At least on my machines, %APPDATA% points to AppData\Roaming, which is the right base folder.
Mar 03
parent MrSmith <mrsmith33 yandex.ru> writes:
On Tuesday, 3 March 2020 at 08:26:19 UTC, Sönke Ludwig wrote:
 Nope, should be right. At least on my machines, %APPDATA% 
 points to AppData\Roaming, which is the right base folder.
My bad, confused %APPDATA% with AppData. It is indeed AppData\Roaming\dub\
Mar 03
prev sibling next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].
Sönke, thanks a lot for all the effort you are putting into this over the years. Very much appreciated!
Mar 03
prev sibling next sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the 
 danger of freezing due to excessive swapping - this is what 
 happened on the 26th last month [1].

 In addition to that, the server that previously hosted the 
 registry is now used to run an official mirror, reachable at 
 codemirror.dlang.org. This will be configured as a built-in 
 fallback server starting with DMD 2.091.0/DUB 1.12.0 and, at 
 least in theory, will lead to an uptime of virtually 100%.

 To make use of the mirror today, it is also possible to 
 configure it in DUB's settings.json as a custom registry:

 {
    "registryUrls": ["https://codemirror.dlang.org/"]
 }

 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows and in ~/.dub/ on all other systems. The custom entry 
 should be removed once DUB 1.12.0 is used, to avoid redundant 
 requests in certain situations.


 [1]: 
 https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org
thank you very much for this Sönke! Is throwing so much more RAM (= money) and power (= more money) at it going to be a good solution in the long run though? Is there maybe a plan for remaking the registry architecture like adding better supported mirrors and load balancing to it?
Mar 04
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 04.03.2020 um 20:06 schrieb WebFreak001:
 On Monday, 2 March 2020 at 19:17:59 UTC, Sönke Ludwig wrote:
 As of yesterday, code.dlang.org now points to a more powerful 
 dedicated server that can host the DUB registry without the danger of 
 freezing due to excessive swapping - this is what happened on the 26th 
 last month [1].

 In addition to that, the server that previously hosted the registry is 
 now used to run an official mirror, reachable at codemirror.dlang.org. 
 This will be configured as a built-in fallback server starting with 
 DMD 2.091.0/DUB 1.12.0 and, at least in theory, will lead to an uptime 
 of virtually 100%.

 To make use of the mirror today, it is also possible to configure it 
 in DUB's settings.json as a custom registry:

 {
    "registryUrls": ["https://codemirror.dlang.org/"]
 }

 settings.json is found/needs to be created in %APPDATA%\dub\ on 
 Windows and in ~/.dub/ on all other systems. The custom entry should 
 be removed once DUB 1.12.0 is used, to avoid redundant requests in 
 certain situations.


 [1]: https://forum.dlang.org/thread/ontwwoxuhnoczcokacwm forum.dlang.org
thank you very much for this Sönke! Is throwing so much more RAM (= money) and power (= more money) at it going to be a good solution in the long run though? Is there maybe a plan for remaking the registry architecture like adding better supported mirrors and load balancing to it?
The server has a different main purpose, so this is just a nice benefit for the registry in that regard. The main issue is that there is some kind of GC related memory leak that is hard to debug with the basically non-existent tooling. Decoupling the package scanning from the web frontend into separate processes will play a major role in making that much less of an issue, though. There is at least one fully supported mirror now and the cache based mirror by Sebastiaan could also be added as a reliable (albeit partial) fallback. So in that regard, at least the pressure for taking more involved steps has been mostly relieved now. But making changes to the general architecture - be it in the form of adding load-balancing, or more deeply rooted, such as using GIT for storing and distributing the package index - is something that still is definitely desirable to push forward.
Mar 05
next sibling parent mork <mork aloha.com> writes:
On Thursday, 5 March 2020 at 12:41:41 UTC, Sönke Ludwig wrote:
 The main issue is that there is some kind of GC related memory 
 leak that is hard to debug with the basically non-existent 
 tooling.
What a shame for D, how is this possible?
Mar 05
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 5 March 2020 at 12:41:41 UTC, Sönke Ludwig wrote:
 But making changes to the general architecture - be it in the 
 form of adding load-balancing, or more deeply rooted, such as 
 using GIT for storing and distributing the package index - is 
 something that still is definitely desirable to push forward.
A week ago I created a git dub package index at https://github.com/skoppe/dub-packages-index it gets updated every 15 minutes by scraping code.dlang.org I use it to feed my registry-mirror, which is now fully standalone A possible next step is creating a PackageSupplier class in dub that clones and uses this repo for resolution. If I was aware of Sönke's work I might have skipped it, but it was a good exercise regardless.
Mar 05
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 3/2/2020 11:17 AM, Sönke Ludwig wrote:
 As of yesterday, code.dlang.org now points to a more powerful dedicated server 
 that can host the DUB registry without the danger of freezing due to excessive 
 swapping - this is what happened on the 26th last month [1].
Wonderful news!
Mar 04