www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - std.net.curl and libcurl.so

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

As some have you may have followed, I've been working on 
snap-packaging LDC.  However, I've run into an issue when it 
comes to programs that use std.net.curl.

Here's a simple example:

------------------------
void main ()
{
     import std.net.curl : get;
     auto website = "http://dlang.org/".get;
}
------------------------

When I compile with my snap-packaged LDC, the program builds, but 
segfaults when I run it.  Here's the gdb backtrace:

Thread 1 "curlget" received signal SIGSEGV, Segmentation fault.
0x000000000046b048 in gc.gc.Gcx.smallAlloc(ubyte, ref ulong, 
uint) ()
(gdb) bt
#0  0x000000000046b048 in gc.gc.Gcx.smallAlloc(ubyte, ref ulong, 
uint) ()
#1  0x000000000046918a in gc.gc.GC.malloc(ulong, uint, ulong*, 
const(TypeInfo)) ()
#2  0x00000000004682bc in gc_qalloc ()
#3  0x000000000046293b in core.memory.GC.qalloc(ulong, uint, 
const(TypeInfo)) ()
#4  0x00000000004534f2 in 
std.array.Appender!(char[]).Appender.ensureAddable(ulong) ()
#5  0x00000000004530c5 in 
std.uni.toCase!(std.uni.toLowerIndex(dchar), 1043, 
std.uni.toLowerTab(ulong), 
char[]).toCase(char[]).__foreachbody2(ref ulong, ref dchar) ()
#6  0x000000000046dc96 in _aApplycd2 ()
#7  0x000000000044d997 in 
std.net.curl.HTTP.Impl.onReceiveHeader(void(const(char[]), 
const(char[])) delegate).__lambda2(const(char[])) ()
#8  0x0000000000451d78 in 
std.net.curl.Curl._receiveHeaderCallback(const(char*), ulong, 
ulong, void*) ()
#9  0x00007ffff6f7695d in ?? () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#10 0x00007ffff6f74ed0 in ?? () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#11 0x00007ffff6f8cce0 in ?? () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#12 0x00007ffff6f96b22 in ?? () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#13 0x00007ffff6f97986 in curl_multi_perform () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#14 0x00007ffff6f8e65b in curl_easy_perform () from 
/usr/lib/x86_64-linux-gnu/libcurl.so
#15 0x000000000044f971 in 
std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) ()
#16 0x000000000040db01 in 
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], 
const(void)[], std.net.curl.HTTP) ()
#17 0x00000000004031ff in std.net.curl.get!(std.net.curl.HTTP, 
char).get(const(char)[], std.net.curl.HTTP) ()
#18 0x0000000000403020 in 
std.net.curl.get!(std.net.curl.AutoProtocol, 
char).get(const(char)[], std.net.curl.AutoProtocol) ()
#19 0x0000000000402f98 in D main ()

Since I'm not super-familiar with libcurl, it's a bit un-obvious 
what's going on here.  If I understand right, libcurl is lazily 
loaded at runtime rather than explicitly linked.  
libcurl4-gnutls-dev is installed on my machine, so it can't just 
be lack of availability.  (I've also tried uninstalling 
libcurl4-gnutls-dev and installing libcurl4-openssl-dev instead, 
with no difference in the result.)

The segfault would suggest to me that either the loading of the 
library fails or that there's some resource phobos expects to 
find which it can't access.  Can anyone advise what could be 
going on here?

In particular, is there anything that happens at compile-time 
which could affect the ability of the resulting executable to 
locate libcurl.so or any other resources it needs?  My suspicion 
is that the snap-package containerization of LDC could mean that 
expectations about system-path locations are interfered with.

Can anyone advise?

Thanks & best wishes,

     -- Joe
Sep 22 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 22 September 2016 at 23:19:27 UTC, Joseph Rushton 
Wakeling wrote:
 The segfault would suggest to me that either the loading of the 
 library fails or that there's some resource phobos expects to 
 find which it can't access.  Can anyone advise what could be 
 going on here?
     -- Joe
 /usr/lib/x86_64-linux-gnu/libcurl.so
 #13 0x00007ffff6f97986 in curl_multi_perform () from 
 /usr/lib/x86_64-linux-gnu/libcurl.so
 #14 0x00007ffff6f8e65b in curl_easy_perform () from 
 /usr/lib/x86_64-linux-gnu/libcurl.so
This suggests that libcurl is loaded. could you compile with -g ? and then post the output ? Also the call seems to fail during allocation.
Sep 22 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 23 September 2016 at 00:55:43 UTC, Stefan Koch wrote:
 This suggests that libcurl is loaded.
 could you compile with -g ?
 and then post the output ?
Thanks Stefan! It was compiled with -g, but I was missing the libcurl3-dbg package. Here's the results: #0 0x000000000046b048 in gc.gc.Gcx.smallAlloc(ubyte, ref ulong, uint) () #1 0x000000000046918a in gc.gc.GC.malloc(ulong, uint, ulong*, const(TypeInfo)) () #2 0x00000000004682bc in gc_qalloc () #3 0x000000000046293b in core.memory.GC.qalloc(ulong, uint, const(TypeInfo)) () #4 0x00000000004534f2 in std.array.Appender!(char[]).Appender.ensureAddable(ulong) () #5 0x00000000004530c5 in std.uni.toCase!(std.uni.toLowerIndex(dchar), 1043, std.uni.toLowerTab(ulong), char[]).toCase(char[]).__foreachbody2(ref ulong, ref dchar) () #6 0x000000000046dc96 in _aApplycd2 () #7 0x000000000044d997 in std.net.curl.HTTP.Impl.onReceiveHeader(void(const(char[]), const(char[])) delegate).__lambda2(const(char[])) () #8 0x0000000000451d78 in std.net.curl.Curl._receiveHeaderCallback(const(char*), ulong, ulong, void*) () #9 0x00007ffff6f75a1d in Curl_client_chop_write (conn=conn entry=0x741fd0, type=type entry=2, ptr=0x72d3a0 "Server: Apache/2.4.17 (FreeBSD) OpenSSL/1.0.2d PHP/5.6.16\r\n", len=59) at sendf.c:454 #10 0x00007ffff6f75b9d in Curl_client_write (conn=conn entry=0x741fd0, type=type entry=2, ptr=<optimised out>, len=<optimised out>) at sendf.c:511 #11 0x00007ffff6f73f90 in Curl_http_readwrite_headers (data=data entry=0x72fb60, conn=conn entry=0x741fd0, nread=nread entry=0x7fffffffd370, stop_reading=stop_reading entry=0x7fffffffd36f) at http.c:3739 #12 0x00007ffff6f8be30 in readwrite_data (done=0x7fffffffd40b, didwhat=<synthetic pointer>, k=0x72fbd8, conn=0x741fd0, data=0x72fb60) at transfer.c:492 #13 Curl_readwrite (conn=0x741fd0, data=data entry=0x72fb60, done=done entry=0x7fffffffd40b) at transfer.c:1074 #14 0x00007ffff6f95f42 in multi_runsingle (multi=multi entry=0x7388b0, now=..., data=data entry=0x72fb60) at multi.c:1544 #15 0x00007ffff6f96ddd in curl_multi_perform (multi_handle=multi_handle entry=0x7388b0, running_handles=running_handles entry=0x7fffffffd5a4) at multi.c:1821 #16 0x00007ffff6f8d84b in easy_transfer (multi=0x7388b0) at easy.c:724 #17 easy_perform (events=false, data=0x72fb60) at easy.c:812 #18 curl_easy_perform (easy=0x72fb60) at easy.c:831 #19 0x000000000044f971 in std.net.curl.HTTP.perform(std.typecons.Flag!("throwOnError").Flag) () #20 0x000000000040db01 in std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[], std.net.curl.HTTP) () #21 0x00000000004031ff in std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[], std.net.curl.HTTP) () #22 0x0000000000403020 in std.net.curl.get!(std.net.curl.AutoProtocol, char).get(const(char)[], std.net.curl.AutoProtocol) () #23 0x0000000000402f98 in D main ()
 Also the call seems to fail during allocation.
Yup. And that allocation seems to occur during use of an Appender while converting unicode text to lowercase ... ?
Sep 24 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:11:52 UTC, Joseph Rushton 
Wakeling wrote:
 On Friday, 23 September 2016 at 00:55:43 UTC, Stefan Koch wrote:
 This suggests that libcurl is loaded.
 could you compile with -g ?
 and then post the output ?
Further to earlier remarks: I now think this may be a general problem of LDC 1.0.0 and not a problem of the snap package. I tried building my simple curl-using program using an LDC 1.0.0 build and installed from source in the standard cmake && make && make install fashion. The same segfault occurs.
Sep 24 2016
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:27:31 UTC, Joseph Rushton 
Wakeling wrote:
 Further to earlier remarks: I now think this may be a general 
 problem of LDC 1.0.0 and not a problem of the snap package.

 I tried building my simple curl-using program using an LDC 
 1.0.0 build and installed from source in the standard cmake && 
 make && make install fashion.  The same segfault occurs.
More on this: ldc 0.17.1 (the version packaged with Ubuntu 16.04) is based on dmd 2.068.2, which still linked against libcurl. It's only from v2.069.0+ that libcurl is loaded dynamically: https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading ldc 1.0.0 is based on 2.070.2.
Sep 24 2016
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 24 September 2016 at 19:42:11 UTC, Joseph Rushton 
Wakeling wrote:
 On Saturday, 24 September 2016 at 19:27:31 UTC, Joseph Rushton 
 Wakeling wrote:
 Further to earlier remarks: I now think this may be a general 
 problem of LDC 1.0.0 and not a problem of the snap package.

 I tried building my simple curl-using program using an LDC 
 1.0.0 build and installed from source in the standard cmake && 
 make && make install fashion.  The same segfault occurs.
More on this: ldc 0.17.1 (the version packaged with Ubuntu 16.04) is based on dmd 2.068.2, which still linked against libcurl. It's only from v2.069.0+ that libcurl is loaded dynamically: https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading ldc 1.0.0 is based on 2.070.2.
Downloaded a pre-built copy of ldc 1.0.0 from here (I used the Linux x86_64 tarball): https://github.com/ldc-developers/ldc/releases/tag/v1.0.0 ... and this runs my little curl-based program without any trouble. Maybe something about how the build was carried out?
Sep 24 2016