www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How hard would it be to create a dub2deb tool?

reply deadalnix <deadalnix gmail.com> writes:
For reference, this is what I'm referring to: 
https://github.com/LeoIannacone/npm2deb

I don't think that would be too hard, but, at the same time, I'm 
not super familiar with the dub ecosystem to be honest so maybe 
there is something I'm overlooking.

I think such a tool would be super valuable as it would increase 
options to access d libs. This seems to be a low hanging fruit.
Feb 17
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:
 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.
There's a tool called FPM [1], which is somewhat of a Swiss army knife when it comes to building and converting packages of various kinds. If support for Dub is added to that, then it would be possible to output Deb, RPM, tar, macOS .pkg and a couple of other packages as well. [1] https://github.com/jordansissel/fpm -- /Jacob Carlborg
Feb 17
parent James Lu <jamtlu gmail.com> writes:
On Thursday, 18 February 2021 at 07:01:19 UTC, Jacob Carlborg 
wrote:
 various kinds. If support for Dub is added to that, then it 
 would be possible to output Deb, RPM, tar, macOS .pkg and a 
 couple of other packages as well.

 [1] https://github.com/jordansissel/fpm

 --
 /Jacob Carlborg
This would make D my instant choice for several types of small applications.
Feb 23
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:
 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.
Not .deb, but converting DUB to a general package format nonetheless: https://github.com/lionello/dub2nix
Feb 18
parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 18 February 2021 at 14:15:15 UTC, Dukc wrote:
 Not .deb, but converting DUB to a general package format 
 nonetheless:

 https://github.com/lionello/dub2nix
That's an interesting project, and certainly contains the seeds of what I'm looking for, but some of the design decision are IMO misguided. Nothign against the author per se, these error seems to be pervasive in the npm, pip, ruby gem ecosystems and many more. You simply can't download a bunch of crap from the internet and deploy it this way. First, this is very insecure (see https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the latest iteration of the madness) but it also a reproducibility problems (the source may change from under your feets) and availability (someone pulling leftpad can bring down your whole deployment capability). This is why you want to be able to package things and deploy them as deb/rpm/dmg/whatever
Feb 18
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Feb 18, 2021 at 07:31:10PM +0000, deadalnix via Digitalmars-d wrote:
[...]
 Nothign against the author per se, these error seems to be pervasive
 in the npm, pip, ruby gem ecosystems and many more.
 
 You simply can't download a bunch of crap from the internet and deploy
 it this way. First, this is very insecure (see
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for
 the latest iteration of the madness) but it also a reproducibility
 problems (the source may change from under your feets) and
 availability (someone pulling leftpad can bring down your whole
 deployment capability).
 
 This is why you want to be able to package things and deploy them as
 deb/rpm/dmg/whatever
+1. I have always been skeptical of having my ability to build/deploy depend on some random 3rd party provider somewhere out there in the wild internet whose availability/continued existence is not under my control. But it seems nobody wants to acknowledge that the king has no clothes. Maybe the latest dependency confusion madness will finally pull away the wool. But I'm not holding my breath. T -- Жил-был король когда-то, при нём блоха жила.
Feb 18
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Thursday, 18 February 2021 at 19:56:51 UTC, H. S. Teoh wrote:

 You simply can't download a bunch of crap from the internet 
 and deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).
 
 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever
+1. I have always been skeptical of having my ability to build/deploy depend on some random 3rd party provider somewhere out there in the wild internet whose availability/continued existence is not under my control.
It seems, Meson build system solves this problem by referencing to strictly defined commits. If we will have something like dub2meson then we can use it for build Debian packages: dub2meson && dh build --buildsystem=meson
Feb 21
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Thursday, 18 February 2021 at 19:31:10 UTC, deadalnix wrote:
 You simply can't download a bunch of crap from the internet and 
 deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).

 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever
Good news - none of these are problems for dub2nix. The final Nix derivation (Nix install script) won't use dub2nix program directly, it uses `dub.selections.nix` file the package maintainer has pregenerated with the tool. `dub selections.nix` links directly to github projects, and to specific versions of them - newer version of the DUB package won't be used unless the maintainer regenerates `dub.selections.nix`. Nix forces this design - Internet downloads are verified with a sha256 provided in the dub derivation precisely because of the issues you mentioned. As for the availibility issue, Nix caches Internet downloads done by the derivations, and if you get a package published at Nixpkgs repository, it'll be cached in their servers too.
Feb 19
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 19 February 2021 at 08:07:29 UTC, Dukc wrote:
 On Thursday, 18 February 2021 at 19:31:10 UTC, deadalnix wrote:
 You simply can't download a bunch of crap from the internet 
 and deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).

 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever
Good news - none of these are problems for dub2nix. The final Nix derivation (Nix install script) won't use dub2nix program directly, it uses `dub.selections.nix` file the package maintainer has pregenerated with the tool. `dub selections.nix` links directly to github projects, and to specific versions of them - newer version of the DUB package won't be used unless the maintainer regenerates `dub.selections.nix`. Nix forces this design - Internet downloads are verified with a sha256 provided in the dub derivation precisely because of the issues you mentioned. As for the availibility issue, Nix caches Internet downloads done by the derivations, and if you get a package published at Nixpkgs repository, it'll be cached in their servers too.
deadalnix Required reading: https://edolstra.github.io/pubs/phd-thesis.pdf :P
Feb 19
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 19 February 2021 at 08:43:21 UTC, Petar Kirov 
[ZombineDev] wrote:
  deadalnix Required reading: 
 https://edolstra.github.io/pubs/phd-thesis.pdf
 :P
Here's a more approachable and up to date introduction to Nix: https://www.youtube.com/watch?v=D5Gq2wkRXpU
Feb 19
parent FaithChurchill <FaithOVayChurchill yahoo.com> writes:
On Friday, 19 February 2021 at 08:51:47 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Friday, 19 February 2021 at 08:43:21 UTC, Petar Kirov 
 [ZombineDev] wrote:
  deadalnix Required reading: 
 https://edolstra.github.io/pubs/phd-thesis.pdf
 :P
Here's a more approachable and up to date introduction to Nix: https://www.youtube.com/watch?v=D5Gq2wkRXpU
Thx for your answer.
Feb 24
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Friday, 19 February 2021 at 08:07:29 UTC, Dukc wrote:
 Good news - none of these are problems for dub2nix. The final 
 Nix derivation (Nix install script) won't use dub2nix program 
 directly, it uses `dub.selections.nix` file the package 
 maintainer has pregenerated with the tool. `dub selections.nix` 
 links directly to github projects, and to specific versions of 
 them - newer version of the DUB package won't be used unless 
 the maintainer regenerates `dub.selections.nix`. Nix forces 
 this design - Internet downloads are verified with a sha256 
 provided in the dub derivation precisely because of the issues 
 you mentioned.

 As for the availibility issue, Nix caches Internet downloads 
 done by the derivations, and if you get a package published at 
 Nixpkgs repository, it'll be cached in their servers too.
I'm kind of dumb because I was actually aware of NixOS, and didn't made the link at all. Still not quite what I need, but indeed much better than what I assumed in my initial assessment.
Feb 19
prev sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:
 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.
My gut feeling is, the real problem we have to solve here is to provide something similar like python wheels [1]. Many python packages like e.g. pyarrow are wrappers for C libraries. The python wheel has pre build (shared objects) for various operation systems. Python package manager decides based on the host OS which python wheel to download. No need to install additional system packages via apt-get and getting mad while solving build issues. If we could add s.th. similar to the dub infrastructure (registry and client), this would remove a lot work and headaches for dub package users. [1] https://realpython.com/python-wheels/ Kind regards Andre
Feb 18