www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How hard would it be to create a dub2deb tool?

reply deadalnix <deadalnix gmail.com> writes:
For reference, this is what I'm referring to: 
https://github.com/LeoIannacone/npm2deb

I don't think that would be too hard, but, at the same time, I'm 
not super familiar with the dub ecosystem to be honest so maybe 
there is something I'm overlooking.

I think such a tool would be super valuable as it would increase 
options to access d libs. This seems to be a low hanging fruit.
Feb 17 2021
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:

 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.


There's a tool called FPM [1], which is somewhat of a Swiss army 
knife when it comes to building and converting packages of 
various kinds. If support for Dub is added to that, then it would 
be possible to output Deb, RPM, tar, macOS .pkg and a couple of 
other packages as well.

[1] https://github.com/jordansissel/fpm

--
/Jacob Carlborg
Feb 17 2021
parent James Lu <jamtlu gmail.com> writes:
On Thursday, 18 February 2021 at 07:01:19 UTC, Jacob Carlborg 
wrote:

 various kinds. If support for Dub is added to that, then it 
 would be possible to output Deb, RPM, tar, macOS .pkg and a 
 couple of other packages as well.

 [1] https://github.com/jordansissel/fpm

 --
 /Jacob Carlborg


This would make D my instant choice for several types of small
applications.
Feb 23 2021
prev sibling next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:

 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.


Not .deb, but converting DUB to a general package format 
nonetheless:

https://github.com/lionello/dub2nix
Feb 18 2021
parent reply deadalnix <deadalnix gmail.com> writes:
On Thursday, 18 February 2021 at 14:15:15 UTC, Dukc wrote:

 Not .deb, but converting DUB to a general package format 
 nonetheless:

 https://github.com/lionello/dub2nix


That's an interesting project, and certainly contains the seeds 
of what I'm looking for, but some of the design decision are IMO 
misguided. Nothign against the author per se, these error seems 
to be pervasive in the npm, pip, ruby gem ecosystems and many 
more.

You simply can't download a bunch of crap from the internet and 
deploy it this way. First, this is very insecure (see 
https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 
for the latest iteration of the madness) but it also a 
reproducibility problems (the source may change from under your 
feets) and availability (someone pulling leftpad can bring down 
your whole deployment capability).

This is why you want to be able to package things and deploy them 
as deb/rpm/dmg/whatever
Feb 18 2021
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Feb 18, 2021 at 07:31:10PM +0000, deadalnix via Digitalmars-d wrote:
[...]

 Nothign against the author per se, these error seems to be pervasive
 in the npm, pip, ruby gem ecosystems and many more.
 
 You simply can't download a bunch of crap from the internet and deploy
 it this way. First, this is very insecure (see
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for
 the latest iteration of the madness) but it also a reproducibility
 problems (the source may change from under your feets) and
 availability (someone pulling leftpad can bring down your whole
 deployment capability).
 
 This is why you want to be able to package things and deploy them as
 deb/rpm/dmg/whatever


+1.  I have always been skeptical of having my ability to build/deploy
depend on some random 3rd party provider somewhere out there in the wild
internet whose availability/continued existence is not under my control.

But it seems nobody wants to acknowledge that the king has no clothes.

Maybe the latest dependency confusion madness will finally pull away the
wool.  But I'm not holding my breath.


T

-- 
Жил-был король когда-то, при нём блоха жила.
Feb 18 2021
parent Denis Feklushkin <feklushkin.denis gmail.com> writes:
On Thursday, 18 February 2021 at 19:56:51 UTC, H. S. Teoh wrote:


 You simply can't download a bunch of crap from the internet 
 and deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).
 
 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever


 +1.  I have always been skeptical of having my ability to 
 build/deploy depend on some random 3rd party provider somewhere 
 out there in the wild internet whose availability/continued 
 existence is not under my control.


It seems, Meson build system solves this problem by referencing 
to strictly defined commits.

If we will have something like dub2meson then we can use it for 
build Debian packages:

dub2meson && dh build --buildsystem=meson
Feb 21 2021
prev sibling parent reply Dukc <ajieskola gmail.com> writes:
On Thursday, 18 February 2021 at 19:31:10 UTC, deadalnix wrote:

 You simply can't download a bunch of crap from the internet and 
 deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).

 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever


Good news - none of these are problems for dub2nix. The final Nix 
derivation (Nix install script) won't use dub2nix program 
directly, it uses `dub.selections.nix` file the package 
maintainer has pregenerated with the tool. `dub selections.nix` 
links directly to github projects, and to specific versions of 
them - newer version of the DUB package won't be used unless the 
maintainer regenerates `dub.selections.nix`. Nix forces this 
design - Internet downloads are verified with a sha256 provided 
in the dub derivation precisely because of the issues you 
mentioned.

As for the availibility issue, Nix caches Internet downloads done 
by the derivations, and if you get a package published at Nixpkgs 
repository, it'll be cached in their servers too.
Feb 19 2021
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 19 February 2021 at 08:07:29 UTC, Dukc wrote:

 On Thursday, 18 February 2021 at 19:31:10 UTC, deadalnix wrote:

 You simply can't download a bunch of crap from the internet 
 and deploy it this way. First, this is very insecure (see 
 https://medium.com/ alex.birsan/dependency-confusion-4a5d60fec610 for the
latest iteration of the madness) but it also a reproducibility problems (the
source may change from under your feets) and availability (someone pulling
leftpad can bring down your whole deployment capability).

 This is why you want to be able to package things and deploy 
 them as deb/rpm/dmg/whatever


 Good news - none of these are problems for dub2nix. The final 
 Nix derivation (Nix install script) won't use dub2nix program 
 directly, it uses `dub.selections.nix` file the package 
 maintainer has pregenerated with the tool. `dub selections.nix` 
 links directly to github projects, and to specific versions of 
 them - newer version of the DUB package won't be used unless 
 the maintainer regenerates `dub.selections.nix`. Nix forces 
 this design - Internet downloads are verified with a sha256 
 provided in the dub derivation precisely because of the issues 
 you mentioned.

 As for the availibility issue, Nix caches Internet downloads 
 done by the derivations, and if you get a package published at 
 Nixpkgs repository, it'll be cached in their servers too.


 deadalnix Required reading: 
https://edolstra.github.io/pubs/phd-thesis.pdf
:P
Feb 19 2021
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 19 February 2021 at 08:43:21 UTC, Petar Kirov 
[ZombineDev] wrote:

  deadalnix Required reading: 
 https://edolstra.github.io/pubs/phd-thesis.pdf
 :P


Here's a more approachable and up to date introduction to Nix:
https://www.youtube.com/watch?v=D5Gq2wkRXpU
Feb 19 2021
parent FaithChurchill <FaithOVayChurchill yahoo.com> writes:
On Friday, 19 February 2021 at 08:51:47 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Friday, 19 February 2021 at 08:43:21 UTC, Petar Kirov 
 [ZombineDev] wrote:

  deadalnix Required reading: 
 https://edolstra.github.io/pubs/phd-thesis.pdf
 :P


 Here's a more approachable and up to date introduction to Nix:
 https://www.youtube.com/watch?v=D5Gq2wkRXpU


Thx for your answer.
Feb 24 2021
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Friday, 19 February 2021 at 08:07:29 UTC, Dukc wrote:

 Good news - none of these are problems for dub2nix. The final 
 Nix derivation (Nix install script) won't use dub2nix program 
 directly, it uses `dub.selections.nix` file the package 
 maintainer has pregenerated with the tool. `dub selections.nix` 
 links directly to github projects, and to specific versions of 
 them - newer version of the DUB package won't be used unless 
 the maintainer regenerates `dub.selections.nix`. Nix forces 
 this design - Internet downloads are verified with a sha256 
 provided in the dub derivation precisely because of the issues 
 you mentioned.

 As for the availibility issue, Nix caches Internet downloads 
 done by the derivations, and if you get a package published at 
 Nixpkgs repository, it'll be cached in their servers too.


I'm kind of dumb because I was actually aware of NixOS, and 
didn't made the link at all.

Still not quite what I need, but indeed much better than what I 
assumed in my initial assessment.
Feb 19 2021
prev sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 17 February 2021 at 17:14:35 UTC, deadalnix wrote:

 For reference, this is what I'm referring to: 
 https://github.com/LeoIannacone/npm2deb

 I don't think that would be too hard, but, at the same time, 
 I'm not super familiar with the dub ecosystem to be honest so 
 maybe there is something I'm overlooking.

 I think such a tool would be super valuable as it would 
 increase options to access d libs. This seems to be a low 
 hanging fruit.


My gut feeling is, the real problem we have to solve here is to 
provide something similar like python wheels [1].
Many python packages like e.g. pyarrow are wrappers for C 
libraries. The python wheel has pre build (shared objects) for 
various operation systems. Python package manager decides based 
on the host OS which python wheel to download.
No need to install additional system packages via apt-get and 
getting mad while solving build issues.

If we could add s.th. similar to the dub infrastructure (registry 
and client), this would remove a lot work and headaches for dub 
package users.

[1] https://realpython.com/python-wheels/

Kind regards
Andre
Feb 18 2021