www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Static linking D apps for distribution to the cloud, etc.

reply David J Kordsmeier <dkords gmail.com> writes:
Hi forums, I have what should be a simple question.  I have an 
application which I would like to distribute to the cloud onto a 
linux system which I will not have root access.  I thought it 
should be easier for me to perform static linking of this D app 
instead of packing the application with library dependencies.

If I try to do this, using my most simple example below, with 
ldc2 (1.15.0) using the --static flag, I get various warnings: 
warning: Using 'dlopen' in statically linked applications 
requires at runtime the shared libraries from the glibc version 
used for linking
warning: Using 'gethostbyaddr' in statically linked applications 
requires at runtime the shared libraries from the glibc version 
used for linking

If I use gdc (4.9.0 2.065) for this, I get linker errors, such 
as: undefined reference to `curl_easy_setopt'

I'll go back and review the best practices for using curl with 
phobos.  I still am struggling to make this work, as something 
that would be relatively simple in C to perform (statically link 
to any libraries I would like).  Without being an expert on 
phobos library construction, what is the recommended practice for 
app distribution?  Does anyone have a great sample project to 
point to?  I've looked at the dlang-tour 
https://github.com/dlang-tour/core/blob/master/dub.sdl , and this 
indeed seems like the right track, however, I still get the curl 
warnings and the app won't run.

I have also tried building and dynamically linking, which works 
fine, but I've had trouble pulling the dependent libraries for 
distribution, and namely having trouble with finding curl at 
runtime.

```
import std.net.curl, std.stdio;
int main() {
   auto http = HTTP("dlang.org");
   http.onReceiveHeader = (in char[] key, in char[] value) { 
writeln(key, ": ", value);  };
   http.onReceive = (ubyte[] data) { /+ drop +/ return 
data.length;  };
   http.perform();
   return 0;
}
```
Oct 08 2019
next sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 I'll go back and review the best practices for using curl with 
 phobos.  I still am struggling to make this work, as something 
 that would be relatively simple in C to perform (statically 
 link to any libraries I would like).


Phobos 2.069 switched to loading libcurl at runtime, see 
https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. It 
mentions that it's still possible to link it statically into the 
executable.
Oct 08 2019
parent reply Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 8 October 2019 at 19:35:03 UTC, kinke wrote:

 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:

 I'll go back and review the best practices for using curl with 
 phobos.  I still am struggling to make this work, as something 
 that would be relatively simple in C to perform (statically 
 link to any libraries I would like).


 Phobos 2.069 switched to loading libcurl at runtime, see 
 https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. 
 It mentions that it's still possible to link it statically into 
 the executable.


The TSV Utilities have used static linking for Linux prebuilt 
release binaries for a couple years now without any problems. The 
tools do not use curl or gethostbyaddr, but there are still 
plenty of warning messages. (Example: 
https://travis-ci.org/eBay/tsv-utils/jobs/593432965)

Static linking was suggested by Jacob Carlborg, who pointed out 
that the pre-built binaries wouldn't run on all Linux 
distributions. Static linking fixed it, with no subsequent 
problems reported (https://github.com/eBay/tsv-utils/issues/67).
Oct 08 2019
parent reply David J Kordsmeier <dkords gmail.com> writes:
On Tuesday, 8 October 2019 at 22:23:39 UTC, Jon Degenhardt wrote:

 On Tuesday, 8 October 2019 at 19:35:03 UTC, kinke wrote:

 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:

 I'll go back and review the best practices for using curl 
 with phobos.  I still am struggling to make this work, as 
 something that would be relatively simple in C to perform 
 (statically link to any libraries I would like).


 Phobos 2.069 switched to loading libcurl at runtime, see 
 https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading. 
 It mentions that it's still possible to link it statically 
 into the executable.


 The TSV Utilities have used static linking for Linux prebuilt 
 release binaries for a couple years now without any problems. 
 The tools do not use curl or gethostbyaddr, but there are still 
 plenty of warning messages. (Example: 
 https://travis-ci.org/eBay/tsv-utils/jobs/593432965)

 Static linking was suggested by Jacob Carlborg, who pointed out 
 that the pre-built binaries wouldn't run on all Linux 
 distributions. Static linking fixed it, with no subsequent 
 problems reported (https://github.com/eBay/tsv-utils/issues/67).


Thank you all forum folks for the replies.  My use case for curl 
is simply HTTP/HTTPS, not libcurl specifically.  That said, I was 
making a best effort to not use external dependencies per my 
requirements.  My feeling is that a platform should provide needs 
for modern computing, and web is a part of that.  I hope this 
question "goes away" in the future and there becomes a modern 
library alternative to the std.net for http as standard part of 
phobos.

So a number of great responses here, and I am going to explore 
the export option as well as the http2 (from arsd) option.  John, 
I'll check out the tsv-utils build as well, your CI server set up 
looks really great.  I am very interested to learn the best 
practices of distribution of prebuilt D apps and libraries.
Oct 08 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 9 October 2019 at 01:10:23 UTC, David J Kordsmeier 
wrote:

 HTTP/HTTPS

 as well as the http2 (from arsd) option.



So let me warn you ahead of time that my http2.d still has an 
outside dependency: OpenSSL. And that can sometimes be tricky as 
openssl broke binary compatibility between 1.1 and 1.0. I was 
planning on making it dynamically load that to work around it :S

so your -static build MIGHT work just I'm not sure. TLS/SSL is 
one of the few things I don't DIY and feel a third-party 
dependency is justified on, I just try to make it seamless and... 
it usually works on Linux but not always.

Helper switches: -version=older_openssl if you get linker errors 
and have 1.0 on the system or -version=newer_openssl if you get 
linker errors and have 1.1 on.

I *have* statically linked OpenSSL with this library before so I 
know it works, just I also had to jump through a few hoops to 
make it work. However I was on Windows then so Linux might be 
easier... or might be harder. I don't know. but if it does work 
that also gives a nice way around the 1.0/1.1 problem!

if you do try it lemme know how it goes plz.
Oct 08 2019
parent David J Kordsmeier <dkords gmail.com> writes:
On Wednesday, 9 October 2019 at 01:29:59 UTC, Adam D. Ruppe wrote:

 On Wednesday, 9 October 2019 at 01:10:23 UTC, David J 
 Kordsmeier wrote:

 HTTP/HTTPS

 as well as the http2 (from arsd) option.



 So let me warn you ahead of time that my http2.d still has an 
 outside dependency: OpenSSL. And that can sometimes be tricky 
 as openssl broke binary compatibility between 1.1 and 1.0. I 
 was planning on making it dynamically load that to work around 
 it :S

 so your -static build MIGHT work just I'm not sure. TLS/SSL is 
 one of the few things I don't DIY and feel a third-party 
 dependency is justified on, I just try to make it seamless 
 and... it usually works on Linux but not always.

 Helper switches: -version=older_openssl if you get linker 
 errors and have 1.0 on the system or -version=newer_openssl if 
 you get linker errors and have 1.1 on.

 I *have* statically linked OpenSSL with this library before so 
 I know it works, just I also had to jump through a few hoops to 
 make it work. However I was on Windows then so Linux might be 
 easier... or might be harder. I don't know. but if it does work 
 that also gives a nice way around the 1.0/1.1 problem!

 if you do try it lemme know how it goes plz.



Thank you Adam for your work in the D community.  Thanks for the 
heads up.  Yes, definitely OpenSSL (or any security library) 
isn't had for free in a secure Internet and I will need to make 
some effort to ensure I get that working.  In my limited use case 
(does not need to run ANYWHERE, but in a specific environment) it 
should be possible.  Will update as soon as I know.
Oct 08 2019
prev sibling next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.

 [...]


What do you need from Curl? There is also the option to not use 
Curl but e.g. the http client from here 
https://github.com/adamdruppe/arsd/blob/master/http2.d

There is also vibe-d and several other http implementations 
available.

Kind regards
Andre
Oct 08 2019
prev sibling next sibling parent mipri <mipri minimaltype.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 warning: Using 'gethostbyaddr' in statically linked 
 applications requires at runtime the shared libraries from the 
 glibc version used for linking


This is a well-known problem with GNU libc. Your best bets are

1. accepting dynamic linking on the target platform, and setting 
up a
build system that's similar enough to it that deployment is easy.
Maybe setting LD_LIBRARY_PATH to point to deployed libraries that 
you
can't get installed on the taget.

2. using something like flatpak or appimage.

3. using DNS resolution from a library outside of libc, and 
hacking
phobos to use that instead of gethostbyaddr.
Oct 08 2019
prev sibling next sibling parent Arun Chandrasekaran <aruncxy gmail.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.

 [...]


Take a look at https://github.com/ikod/dlang-requests

It offers more features than the libcurl wrapper in std.net.curl.
Oct 08 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.


Alternatively you can build inside a Docker container with the 
same distribution and version as the machine running in the 
cloud. Then you should be able to do dynamic linking and run the 
binary in the cloud without any problem.

--
/Jacob Carlborg
Oct 09 2019
parent Laeeth Isharc <Laeeth laeeth.com> writes:
On Wednesday, 9 October 2019 at 09:33:18 UTC, Jacob Carlborg 
wrote:

 On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
 wrote:

 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought 
 it should be easier for me to perform static linking of this D 
 app instead of packing the application with library 
 dependencies.


 Alternatively you can build inside a Docker container with the 
 same distribution and version as the machine running in the 
 cloud. Then you should be able to do dynamic linking and run 
 the binary in the cloud without any problem.

 --
 /Jacob Carlborg


And of course what seems to be merely a statically linked binary 
can be a container with its own filesystem and carry dependencies 
with it.

https://blog.jessfraz.com/post/nerd-sniped-by-binfmt_misc/

https://github.com/genuinetools/binctr
Oct 09 2019
prev sibling parent Jacob Carlborg <doob me.com> writes:
On Tuesday, 8 October 2019 at 18:28:14 UTC, David J Kordsmeier 
wrote:

 Hi forums, I have what should be a simple question.  I have an 
 application which I would like to distribute to the cloud onto 
 a linux system which I will not have root access.  I thought it 
 should be easier for me to perform static linking of this D app 
 instead of packing the application with library dependencies.


You can also try to build the binary on Alpine and do static 
linking. Alpine uses musl as the C standard library instead of 
GNU. musl is designed for static linking.

--
/Jacob Carlborg
Oct 09 2019