www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should "std.net.curl" be moved from Phobos to Deimos?

reply Jordi Sayol <g.sayol yahoo.es> writes:
As Jonathan M Davis said:
---
Several of the main devs (including Walter) have stated that having
std.net.curl on Phobos was a mistake given all of the problems that we've had
with it on Windows and Linux as well, and at least some of them have expressed
a desire for it to be removed. I expect that there's a good chance that it can
and will be removed from Phobos if brought up for discussion.

Certainly, I think that it's clear that we will not add any further external
dependencies like curl, because curl has proven to be a big problem. It's very
desirable to have bindings and wrappers for C libraries - but putting them in
the standard library when it's not guaranteed that the appropriate library is
on the system by default has proven to be too problematic, so they should be
left to external projects.
---

"std.net.curl" can also be moved from Phobos to Deimos.
Deimos can be rethink, i.e. new build master can add a make building script on
Deimos in order to compile all projects included on it, generating
"libdeimos.a" and "libdeimos.so", documentation, etc.

-- 
Jordi Sayol
Nov 24 2013
next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.


Deimos currently only hosts bindings, not wrappers, as part of
its objective. So, etc.c.curl would be appropriate, but
std.net.curl would be unprecedented.

I think both etc.c.curl and std.net.curl should just be ripped
out and moved into a CurlD (or whatever) project, hosted by
someone on Github (such as Jonas Drewsen or a maintainer), on the
grounds that I don't think Phobos is a place for wrapper
libraries.

I think std.net.curl is a well designed wrapper library, and I
would happily continue to use it in my projects, but I don't
think it's appropriate for Phobos.

As an aside, personally I think Deimos is pointless; and worse,
horribly executed (A Github organization? Really?). A simple wiki
page like this one[1] can do the job much better and without the
ridiculous self-imposed disadvantages we are putting up with by
using a Github organization.

[1] http://wiki.dlang.org/Bindings
Nov 25 2013
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 25.11.2013 09:32, schrieb Jakob Ovrum:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.


 
 Deimos currently only hosts bindings, not wrappers, as part of
 its objective. So, etc.c.curl would be appropriate, but
 std.net.curl would be unprecedented.
 
 I think both etc.c.curl and std.net.curl should just be ripped
 out and moved into a CurlD (or whatever) project, hosted by
 someone on Github (such as Jonas Drewsen or a maintainer), on the
 grounds that I don't think Phobos is a place for wrapper
 libraries.
 
 I think std.net.curl is a well designed wrapper library, and I
 would happily continue to use it in my projects, but I don't
 think it's appropriate for Phobos.
 
 As an aside, personally I think Deimos is pointless; and worse,
 horribly executed (A Github organization? Really?). A simple wiki
 page like this one[1] can do the job much better and without the
 ridiculous self-imposed disadvantages we are putting up with by
 using a Github organization.
 
 [1] http://wiki.dlang.org/Bindings


Or, even better (because it doesn't get out of date), an automatically
generated list like this one:
http://code.dlang.org/?sort=name&category=library.binding
Nov 25 2013
parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 25 November 2013 at 10:08:12 UTC, Sönke Ludwig wrote:

 Or, even better (because it doesn't get out of date), an 
 automatically
 generated list like this one:
 http://code.dlang.org/?sort=name&category=library.binding


Yeah, absolutely :)
Nov 25 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.


I have to second this. It has been a major pain to get D to work 
in some environment because of the dependency on curl.
Nov 25 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 1:34 AM, deadalnix wrote:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.


 I have to second this. It has been a major pain to get D to work in some
 environment because of the dependency on curl.


The continuing stream of issues associated with curl is puzzling me. We 
chose curl _exactly_ because (a) it was available or trivial to install 
everywhere, (b) it had accreted through experience a bunch of tricks 
that made little sense to rediscover the hard way.

Now it seems curl is failing at least the first premise, so we should 
consider various alternatives. First I'd like to gather an understanding 
on why we seem to have this problem (far as I understand, the likes of 
php and python are doing fine with curl, but maybe I'm wrong). So is 
there a short and simple list of issues that anyone can put together?

If we do decide to do away with libcurl, one possible solution would be 
to embed its source code within our build. That way we wouldn't break 
code that already uses it.


Andrei
Nov 25 2013
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:


 The continuing stream of issues associated with curl is 
 puzzling me. We chose curl _exactly_ because (a) it was 
 available or trivial to install everywhere, (b) it had accreted 
 through experience a bunch of tricks that made little sense to 
 rediscover the hard way.

 Now it seems curl is failing at least the first premise, so we 
 should consider various alternatives. First I'd like to gather 
 an understanding on why we seem to have this problem (far as I 
 understand, the likes of php and python are doing fine with 
 curl, but maybe I'm wrong). So is there a short and simple list 
 of issues that anyone can put together?

 If we do decide to do away with libcurl, one possible solution 
 would be to embed its source code within our build. That way we 
 wouldn't break code that already uses it.


 Andrei


A solution could be to ship curl with phobos, if license allow to 
do so.
Nov 25 2013
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:

 First I'd like to gather an understanding on why we seem to 
 have this problem (far as I understand, the likes of php and 
 python are doing fine with curl, but maybe I'm wrong).


A major difference is there's only one php/python binary, usually 
build on the same system that uses it. Phobos, on the other hand, 
is generally built on the packager's computer, which isn't always 
binary compatible with the deployment box.

On Windows, the problem is simply that curl isn't packaged with 
dmd, for some weird reason, meaning people have to get curl.lib 
separately. That's idiotic. But then again, so are a lot of the 
dmd Windows deficiencies.


 If we do decide to do away with libcurl, one possible solution 
 would be to embed its source code within our build. That way we 
 wouldn't break code that already uses it.


Yes, that would be ideal, we should just statically link curl 
right into the phobos build so it just works everywhere.
Nov 25 2013
next sibling parent "Tavi Cacina" <octavian.cacina outlook.com> writes:
On Tuesday, 26 November 2013 at 00:36:22 UTC, Adam D. Ruppe wrote:

 On Windows, the problem is simply that curl isn't packaged with 
 dmd


I see the required DLLs for win32 are in dmd2\windows\bin 
(libcurl.dll, libeay32.dll, libssl32.dll) and in dmd2\Windows the 
needed stuff for SSL (certificate bundle). At least, the docs 
should be improved so that the user knows he has to deliver these 
files with his EXE and has to enable the certificate bundle.
Nov 25 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 4:36 PM, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:

 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).


 A major difference is there's only one php/python binary, usually build
 on the same system that uses it. Phobos, on the other hand, is generally
 built on the packager's computer, which isn't always binary compatible
 with the deployment box.

 On Windows, the problem is simply that curl isn't packaged with dmd, for
 some weird reason, meaning people have to get curl.lib separately.
 That's idiotic. But then again, so are a lot of the dmd Windows
 deficiencies.


Please let me know whether my understanding of the situation is correct:

1. If people have a working installation of libcurl on their machine, we 
work with it.

2. Otherwise, phobos works fine but attempting to use std.net.curl will 
fail.

Is this correct? If not, please explain exactly why. If yes, this setup 
seems entirely appropriate to me.


 If we do decide to do away with libcurl, one possible solution would
 be to embed its source code within our build. That way we wouldn't
 break code that already uses it.


 Yes, that would be ideal, we should just statically link curl right into
 the phobos build so it just works everywhere.


There are several questions associated with this.

1. Does the author of libcurl agree with such? My understanding is he would.

2. Would we need to actually build libcurl from source, or just 
distribute it? In the former case, it would be quite odd that we'd need 
to do that (and presumably we'd need to take some maintenance burden, too).

3. What if people already have some working libcurl version and would 
want to use that?


Andrei
Nov 26 2013
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 10:44:07 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 11/25/13 4:36 PM, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:

 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).


 A major difference is there's only one php/python binary, usually build
 on the same system that uses it. Phobos, on the other hand, is generally
 built on the packager's computer, which isn't always binary compatible
 with the deployment box.

 On Windows, the problem is simply that curl isn't packaged with dmd, for
 some weird reason, meaning people have to get curl.lib separately.
 That's idiotic. But then again, so are a lot of the dmd Windows
 deficiencies.


 
 Please let me know whether my understanding of the situation is correct:
 
 1. If people have a working installation of libcurl on their machine, we 
 work with it.
 
 2. Otherwise, phobos works fine but attempting to use std.net.curl will 
 fail.
 
 Is this correct? If not, please explain exactly why. If yes, this setup 
 seems entirely appropriate to me.
 

 If we do decide to do away with libcurl, one possible solution would
 be to embed its source code within our build. That way we wouldn't
 break code that already uses it.


 Yes, that would be ideal, we should just statically link curl right into
 the phobos build so it just works everywhere.


 
 There are several questions associated with this.
 
 1. Does the author of libcurl agree with such? My understanding is he would.
 
 2. Would we need to actually build libcurl from source, or just 
 distribute it? In the former case, it would be quite odd that we'd need 
 to do that (and presumably we'd need to take some maintenance burden, too).
 
 3. What if people already have some working libcurl version and would 
 want to use that?
 
 
 Andrei


Point 3 is a bigger issue than it may seem at first. A reason
to use shared libraries is that the modularity makes it
trivial to get security updates, whereas a monolithic Phobos
is not necessarily updated when a security fix for cURL becomes
available.

This page shows that sometimes security holes are found on a
daily basis: http://curl.haxx.se/docs/security.html

-- 
Marco
Nov 26 2013
prev sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 19:44, Andrei Alexandrescu ha escrit:

 On 11/25/13 4:36 PM, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:

 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).


 A major difference is there's only one php/python binary, usually build
 on the same system that uses it. Phobos, on the other hand, is generally
 built on the packager's computer, which isn't always binary compatible
 with the deployment box.

 On Windows, the problem is simply that curl isn't packaged with dmd, for
 some weird reason, meaning people have to get curl.lib separately.
 That's idiotic. But then again, so are a lot of the dmd Windows
 deficiencies.


 
 Please let me know whether my understanding of the situation is correct:
 
 1. If people have a working installation of libcurl on their machine, we work
with it.
 
 2. Otherwise, phobos works fine but attempting to use std.net.curl will fail.
 
 Is this correct? If not, please explain exactly why. If yes, this setup seems
entirely appropriate to me.


Point 2. is correct only with phobos static library but not with shared one. A
simple "Hello World" program cannot be linked against shared phobos library if
libcurl is not available on system, at least on Linux. It cannot run either.

-- 
Jordi Sayol
Nov 28 2013
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, November 25, 2013 16:13:57 Andrei Alexandrescu wrote:

 The continuing stream of issues associated with curl is puzzling me. We
 chose curl _exactly_ because (a) it was available or trivial to install
 everywhere, (b) it had accreted through experience a bunch of tricks
 that made little sense to rediscover the hard way.
 
 Now it seems curl is failing at least the first premise, so we should
 consider various alternatives. First I'd like to gather an understanding
 on why we seem to have this problem (far as I understand, the likes of
 php and python are doing fine with curl, but maybe I'm wrong). So is
 there a short and simple list of issues that anyone can put together?
 
 If we do decide to do away with libcurl, one possible solution would be
 to embed its source code within our build. That way we wouldn't break
 code that already uses it.


The number one problem on Windows is the fact that libcurl does not come with 
Windows and that you have to get a version of it build which is compatible 
with dmd has proven to be a huge hurdle for Windows developers. There may be 
other problems, but that's the biggest. Distributing libcurl with dmd could fix 
that, but IIRC, there was some reason why we couldn't do that (or maybe it was 
just that Walter didn't want to - I can't remember).

On Linux, I believe that the biggest problem is that dmd then has to be built 
on your specific distro for it to work with your distro, so the libraries in 
the zip file were only working on debian-based systems. Regardless of why 
libcurl is not distributed with dmd on Windows, we couldn't distribute for 
Linux to fix this problem, because that would conflict with the system's 
libcurl.

There may be other problems, but I think that those are the two bigs ones, and 
they've come up several times (particularly the issues with libcurl and 
Windows).

There's also Don's recent rant on why curl was ever supported in the first 
place, but I guess that he just wasn't paying enough attention to it given 
that he only complained about it recently rather than during the review:

http://forum.dlang.org/thread/CADpwU1cu=0r+NJ+4GN9P9Fd_XisK1dsduYrrkjpGAvM0YJ_tUA mail.gmail.com

At this point, I'm inclined to argue that Phobos should not depend on any 
libraries other than the system libraries for each OS so that we can avoid 
further dependencies that may not be available - especially when it comes to 
Windows, since Windows comes with no 3rd party development libraries at all. 
So, I would very much be against std.net.curl's inclusion at this point and 
would argue that it should be in a library separate from Phobos and that if we 
want similar functionality in Phobos, we need to implement it without relying 
on anything other than system libraries. And from previous discussions on the 
topic, I believe that Walter and several other dmd/Phobos devs agree with 
that.

So, I'm all for removing std.net.curl, but if we do that, we'd obviously need 
to deprecate it first rather than simply removing it, and we'd need to have a 
separate library with std.net.curl in it (preferably one which could be pulled 
in via dub) which people could switch to using instead. Whether that library 
would be maintained by us, by Jonas (since he created it), or someone else, I 
don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I 
think that we should avoid putting any more 3rd party dependencies in Phobos 
in the future.

- Jonathan M Davis
Nov 25 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:


 At minimum, I think that we should avoid putting any
 more 3rd party dependencies in Phobos in the future.


I am not expert/qualified enough to discuss about all this, but 
having some "batteries included" in D is nice. I am using curl in 
D. I'd even like in the standard D distribution some mean to 
produce a little graphics on the screen, shared on all systems D 
compiles on (like the simplegraphics.d module).

Bye,
bearophile
Nov 25 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 4:36 PM, Jonathan M Davis wrote:

 The number one problem on Windows is the fact that libcurl does not come with
 Windows and that you have to get a version of it build which is compatible
 with dmd has proven to be a huge hurdle for Windows developers.


Why is "compatible with dmd" an issue? Do we make it any harder for 
libcurl than other clients? Could you please explain?


 There may be
 other problems, but that's the biggest. Distributing libcurl with dmd could fix
 that, but IIRC, there was some reason why we couldn't do that (or maybe it was
 just that Walter didn't want to - I can't remember).


Walter is opposed only to building libcurl from source for distributing 
it, which is an understandable stance.


 On Linux, I believe that the biggest problem is that dmd then has to be built
 on your specific distro for it to work with your distro, so the libraries in
 the zip file were only working on debian-based systems. Regardless of why
 libcurl is not distributed with dmd on Windows, we couldn't distribute for
 Linux to fix this problem, because that would conflict with the system's
 libcurl.


So what is the classic recommended solution for such cases? We can't be 
the first people who are having this issue on Unix.


 There may be other problems, but I think that those are the two bigs ones, and
 they've come up several times (particularly the issues with libcurl and
 Windows).


I've asked the question "what exactly is the problem" several times and 
you came closest to answering it, but a bunch of stuff is still unclear 
to me.


 There's also Don's recent rant on why curl was ever supported in the first
 place, but I guess that he just wasn't paying enough attention to it given
 that he only complained about it recently rather than during the review:

 http://forum.dlang.org/thread/CADpwU1cu=0r+NJ+4GN9P9Fd_XisK1dsduYrrkjpGAvM0YJ_tUA mail.gmail.com


First, it's not like we slipped curl in while nobody was looking. 
Second, I'm sure curl has its shortcomings, but I'd be hard pressed to 
find better alternatives.

As to just dumping curl and doing our own: I just looked and libcurl has 
160K lines. Those include examples and probably a fair amount of header 
chaff etc. that's unneeded in D, but I'd assess there are on the order 
of 100K lines of "pulp" code that does real work. Simply dumping 
std.net.curl would not only break existing code that depends on it in 
unthinkable ways (no replacement and no recourse), but would put us some 
100KLOC behind in terms of a working replacement.


 At this point, I'm inclined to argue that Phobos should not depend on any
 libraries other than the system libraries for each OS so that we can avoid
 further dependencies that may not be available - especially when it comes to
 Windows, since Windows comes with no 3rd party development libraries at all.
 So, I would very much be against std.net.curl's inclusion at this point and
 would argue that it should be in a library separate from Phobos and that if we
 want similar functionality in Phobos, we need to implement it without relying
 on anything other than system libraries. And from previous discussions on the
 topic, I believe that Walter and several other dmd/Phobos devs agree with
 that.


I don't agree with that. Or I don't understand what you're asking. If 
what you're asking is we break all existing code using std.net.curl with 
no suggested replacement, that would be pretty bad, no?

If I understand correctly the complaint is: "people who don't have curl 
can't use std.net.curl". That's to be expected. It's in the name!

A more reasonable complaint is: "people who don't have curl can't use 
high-level HTTP/FTP client networking, which is expected of a modern 
language's standard library". For those we should NOT throw away 
libcurl. We should instead focus on providing e.g. std.net.durl which is 
an in-house library for doing great stuff. Then people can choose. In 
the long term we may even recommend moving away from curl because durl 
is mature and better etc.


 So, I'm all for removing std.net.curl, but if we do that, we'd obviously need
 to deprecate it first rather than simply removing it, and we'd need to have a
 separate library with std.net.curl in it (preferably one which could be pulled
 in via dub) which people could switch to using instead. Whether that library
 would be maintained by us, by Jonas (since he created it), or someone else, I
 don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I
 think that we should avoid putting any more 3rd party dependencies in Phobos
 in the future.


This is probably going to come again. We can't build everything in house 
in due time. Do we have an openssl layer written in D?


Andrei
Nov 26 2013
next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:00:35 UTC, Andrei Alexandrescu 
wrote:

 On 11/25/13 4:36 PM, Jonathan M Davis wrote:

 The number one problem on Windows is the fact that libcurl 
 does not come with
 Windows and that you have to get a version of it build which 
 is compatible
 with dmd has proven to be a huge hurdle for Windows developers.


 Why is "compatible with dmd" an issue? Do we make it any harder 
 for libcurl than other clients? Could you please explain?


To use 32-bit curl you need to generate an OMF import library 
from the libcurl DLL. It's not terribly hard. I did it originally 
for when I added downloading of libcurl to the Windows installer. 
It's not at all obvious to users that they need to distribute 
libcurl.dll et al when they distribute their application but 
that's a documentation issue.

If I remember correctly Walter didn't want to distribute in the 
dmd zip anything that wasn't boost licensed which is why curl on 
Windows comes as a separate download. zlib is a notable exception 
but I seem to recall Walter regretted including zlib.

I can't remember off the top of my head if libcurl's license 
allows it to be statically linked with a boost licensed library 
but that would certainly simplify things for end users (but 
complicate the building of phobos a bit).
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:13 AM, Brad Anderson wrote:

 To use 32-bit curl you need to generate an OMF import library from the
 libcurl DLL. It's not terribly hard. I did it originally for when I
 added downloading of libcurl to the Windows installer.


When is the OMF library needed?

1. While building dmd

2. While building druntime

3. While building phobos

4. While building client code that does NOT use std.net.curl

5. While building client code that DOES use std.net curl


 It's not at all
 obvious to users that they need to distribute libcurl.dll et al when
 they distribute their application but that's a documentation issue.


Or they could kick the can down the road by requiring their customers to 
have it.


 If I remember correctly Walter didn't want to distribute in the dmd zip
 anything that wasn't boost licensed which is why curl on Windows comes
 as a separate download. zlib is a notable exception but I seem to recall
 Walter regretted including zlib.


On the other hand we've hardly heard one peep from anyone in literally 
years that their zip code has trouble building.

We can ask the libcurl author if it comes down to redistributing a 
libcurl binary. But we want to make sure this is the solution of choice.


 I can't remember off the top of my head if libcurl's license allows it
 to be statically linked with a boost licensed library but that would
 certainly simplify things for end users (but complicate the building of
 phobos a bit).


My reading of http://curl.haxx.se/docs/copyright.html is that the author 
does his best to open up his library.


Andrei
Nov 26 2013
next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei Alexandrescu 
wrote:

 On 11/26/13 11:13 AM, Brad Anderson wrote:

 To use 32-bit curl you need to generate an OMF import library 
 from the
 libcurl DLL. It's not terribly hard. I did it originally for 
 when I
 added downloading of libcurl to the Windows installer.


 When is the OMF library needed?

 1. While building dmd

 2. While building druntime

 3. While building phobos

 4. While building client code that does NOT use std.net.curl

 5. While building client code that DOES use std.net curl


5. std.net.curl actually existed for several releases of dmd 
before I went to use it one day and got linker errors and decided 
to fix it with the installer change.


 It's not at all
 obvious to users that they need to distribute libcurl.dll et 
 al when
 they distribute their application but that's a documentation 
 issue.


 Or they could kick the can down the road by requiring their 
 customers to have it.


The MSVCRT.dll hell of yore has shown this is a terrible option. 
Windows applications almost universally ship with everything 
needed to run. There is no package manager on Windows to handle 
dependencies like there is on Linux.  You always include the DLLs 
or installers in your application (or instruct the user to 
download themselves if licensing forbids redistribution but this 
is avoided as much as possible).


 If I remember correctly Walter didn't want to distribute in 
 the dmd zip
 anything that wasn't boost licensed which is why curl on 
 Windows comes
 as a separate download. zlib is a notable exception but I seem 
 to recall
 Walter regretted including zlib.


 On the other hand we've hardly heard one peep from anyone in 
 literally years that their zip code has trouble building.

 We can ask the libcurl author if it comes down to 
 redistributing a libcurl binary. But we want to make sure this 
 is the solution of choice.


Walter actually already did that and the author of libcurl said 
redistributing it was fine.

Here's an old thread discussing the licensing: 
http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:56 AM, Brad Anderson wrote:

 On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei Alexandrescu wrote:

 On 11/26/13 11:13 AM, Brad Anderson wrote:

 To use 32-bit curl you need to generate an OMF import library from the
 libcurl DLL. It's not terribly hard. I did it originally for when I
 added downloading of libcurl to the Windows installer.


 When is the OMF library needed?

 1. While building dmd

 2. While building druntime

 3. While building phobos

 4. While building client code that does NOT use std.net.curl

 5. While building client code that DOES use std.net curl


 5. std.net.curl actually existed for several releases of dmd before I
 went to use it one day and got linker errors and decided to fix it with
 the installer change.


Awesome. So can we assume 
https://github.com/D-Programming-Language/installer/pull/27 takes care 
of Windows for good?

Thanks for your work!!! I'll make sure I'll get a word of thanks to 
 broccolis as well.


 We can ask the libcurl author if it comes down to redistributing a
 libcurl binary. But we want to make sure this is the solution of choice.


 Walter actually already did that and the author of libcurl said
 redistributing it was fine.

 Here's an old thread discussing the licensing:
 http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org


Terrific, thanks.


Andrei
Nov 26 2013
parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 20:02:29 UTC, Andrei Alexandrescu 
wrote:

 On 11/26/13 11:56 AM, Brad Anderson wrote:

 On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei 
 Alexandrescu wrote:

 On 11/26/13 11:13 AM, Brad Anderson wrote:

 To use 32-bit curl you need to generate an OMF import 
 library from the
 libcurl DLL. It's not terribly hard. I did it originally for 
 when I
 added downloading of libcurl to the Windows installer.


 When is the OMF library needed?

 1. While building dmd

 2. While building druntime

 3. While building phobos

 4. While building client code that does NOT use std.net.curl

 5. While building client code that DOES use std.net curl


 5. std.net.curl actually existed for several releases of dmd 
 before I
 went to use it one day and got linker errors and decided to 
 fix it with
 the installer change.


 Awesome. So can we assume 
 https://github.com/D-Programming-Language/installer/pull/27 
 takes care of Windows for good?


It still needs one of the guys that can upload to the downloads 
page to upload the new binary but yeah, it's more or less ready 
for all scenarios now.


 Thanks for your work!!! I'll make sure I'll get a word of 
 thanks to  broccolis as well.


Will do.


 We can ask the libcurl author if it comes down to 
 redistributing a
 libcurl binary. But we want to make sure this is the solution 
 of choice.


 Walter actually already did that and the author of libcurl said
 redistributing it was fine.

 Here's an old thread discussing the licensing:
 http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org


 Terrific, thanks.


 Andrei
Nov 26 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:27, Andrei Alexandrescu wrote:


 When is the OMF library needed?

 1. While building dmd

 2. While building druntime

 3. While building phobos

 4. While building client code that does NOT use std.net.curl

 5. While building client code that DOES use std.net curl


Not that I have tried it but I'm pretty sure it is 5.


 My reading of http://curl.haxx.se/docs/copyright.html is that the author
 does his best to open up his library.


See this post:

http://forum.dlang.org/thread/i0dlde$59$1 digitalmars.com#post-i0eaoc:2410bs:241:40digitalmars.com

-- 
/Jacob Carlborg
Nov 26 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:13, Brad Anderson wrote:


 If I remember correctly Walter didn't want to distribute in the dmd zip
 anything that wasn't boost licensed which is why curl on Windows comes
 as a separate download. zlib is a notable exception but I seem to recall
 Walter regretted including zlib.


The zip already contains tools which the source code isn't available. 
Like "shell", "dumpobj" and "obj2asm". But I guess Walter, DigitalMars 
or Symantec has created those.


 I can't remember off the top of my head if libcurl's license allows it
 to be statically linked with a boost licensed library but that would
 certainly simplify things for end users (but complicate the building of
 phobos a bit).


Walter contacted the author and asked about that, here's the post:

http://forum.dlang.org/thread/i0dlde$59$1 digitalmars.com#post-i0eaoc:2410bs:241:40digitalmars.com

-- 
/Jacob Carlborg
Nov 26 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Nov-2013 23:00, Andrei Alexandrescu пишет:

 There may be other problems, but I think that those are the two bigs
 ones, and
 they've come up several times (particularly the issues with libcurl and
 Windows).


 I've asked the question "what exactly is the problem" several times and
 you came closest to answering it, but a bunch of stuff is still unclear
 to me.


I thought I'd chime in on this.

I find that if we actually worked on getting curl integrated better it 
may just as well be an all right network client lib.

For the moment it looks as if half people are pissed off because of the 
work they would need to do on it (workaround, shipping, maintenance, 
etc.), and the other half (using it) are having usability problems on 
all fronts.

Now to my limited experience with it.
When I work with it on Linux I have to always jump through hoops because of:
https://d.puremagic.com/issues/show_bug.cgi?id=7044

Because of that in the end I tend to dodge it completely and roll some 
shell script scaffolding that calls `curl`, as in the cmd-line client.

See some hilarious things I had to do to link it properly:
https://github.com/blackwhale/tools/blob/master/posix.mak#L35

Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT 
shipped with dmd zip or whatever. The better thing is that last time I 
checked there is no proper which means that for Win32 one had to get the 
rare OMF object file format (a custom compilation no less).

You may or may not find them at the VARY BOTTOM of downloads page, and 
figure it out how to fit them, what flags to use etc. Last but not least 
it's Win32 only. No links for Win64.

All in all, it feels as if somebody actually wanted to sabotage the 
thing, he could score an A+ job as it stands already.

P.S. And before I forgot - there are specifics problems with curl Linux 
distros that are not Debian derived and with shared phobos in 
particular, but there are fokls that must known this side of coin better 
then I.
-- 
Dmitry Olshansky
Nov 26 2013
next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:23:54 UTC, Dmitry Olshansky 
wrote:

 26-Nov-2013 23:00, Andrei Alexandrescu пишет:

 There may be other problems, but I think that those are the 
 two bigs
 ones, and
 they've come up several times (particularly the issues with 
 libcurl and
 Windows).


 I've asked the question "what exactly is the problem" several 
 times and
 you came closest to answering it, but a bunch of stuff is 
 still unclear
 to me.


 I thought I'd chime in on this.

 I find that if we actually worked on getting curl integrated 
 better it may just as well be an all right network client lib.

 For the moment it looks as if half people are pissed off 
 because of the work they would need to do on it (workaround, 
 shipping, maintenance, etc.), and the other half (using it) are 
 having usability problems on all fronts.

 Now to my limited experience with it.
 When I work with it on Linux I have to always jump through 
 hoops because of:
 https://d.puremagic.com/issues/show_bug.cgi?id=7044


This is a constant annoyance of mine using std.net.curl on Linux.


 Now if that doesn't set off the alarms. Win32/64 curl libraries 
 are NOT shipped with dmd zip or whatever. The better thing is 
 that last time I checked there is no proper which means that 
 for Win32 one had to get the rare OMF object file format (a 
 custom compilation no less).


Nah, it was as simple as generating an import library from the 
DLL. There's no need to build libcurl yourself to do that. The 
installer automatically downloads one I prepared a long time ago 
already.

 brocolis recently prepared an updated libcurl zip with the OMF 
import library as well as 64-bit support. It also uses WinSSL 
instead of OpenSSL so using it is even easier than it was before. 
The installer has already been updated for it: 
https://github.com/D-Programming-Language/installer/pull/27

With that libcurl should just work on Windows for both 32 and 64 
bits. Linux problems are basically all that remains of the 
libcurl issues.


 You may or may not find them at the VARY BOTTOM of downloads 
 page, and figure it out how to fit them, what flags to use etc. 
 Last but not least it's Win32 only. No links for Win64.


The installer handles downloading it.  End users don't need to 
find it themselves unless they download the dmd zip file (I still 
don't understand why anyone does that).

 brocolis wrote up a guide for how he made prepared the updated 
curl zip and put it up on the wiki. End users shouldn't need to 
do that though. It's just documented for the phobos developers. 
http://wiki.dlang.org/Curl_on_Windows
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:39 AM, Brad Anderson wrote:

 On Tuesday, 26 November 2013 at 19:23:54 UTC, Dmitry Olshansky wrote:

 https://d.puremagic.com/issues/show_bug.cgi?id=7044


 This is a constant annoyance of mine using std.net.curl on Linux.


 Now if that doesn't set off the alarms. Win32/64 curl libraries are
 NOT shipped with dmd zip or whatever. The better thing is that last
 time I checked there is no proper which means that for Win32 one had
 to get the rare OMF object file format (a custom compilation no less).


 Nah, it was as simple as generating an import library from the DLL.
 There's no need to build libcurl yourself to do that. The installer
 automatically downloads one I prepared a long time ago already.

  brocolis recently prepared an updated libcurl zip with the OMF import
 library as well as 64-bit support. It also uses WinSSL instead of
 OpenSSL so using it is even easier than it was before. The installer has
 already been updated for it:
 https://github.com/D-Programming-Language/installer/pull/27

 With that libcurl should just work on Windows for both 32 and 64 bits.
 Linux problems are basically all that remains of the libcurl issues.


This is pretty awesome. Is  brocolis around on this forum?

Andrei
Nov 26 2013
parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:43:02 UTC, Andrei Alexandrescu 
wrote:

 This is pretty awesome. Is  brocolis around on this forum?

 Andrei


I'm not sure. He goes by dnewbie/dnwbie on IRC.
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:23 AM, Dmitry Olshansky wrote:

 I find that if we actually worked on getting curl integrated better it
 may just as well be an all right network client lib.


Great.


 For the moment it looks as if half people are pissed off because of the
 work they would need to do on it (workaround, shipping, maintenance,
 etc.), and the other half (using it) are having usability problems on
 all fronts.


I don't think that's the whole set. Don't forget the forum participation 
is a very small fraction of people who actually use D out there. So 
there are a bunch of people who don't use curl and aren't hurt, and 
presumably quite a few people who do use std.net.curl without troubles 
and don't know we plan to pull the plug on it.


 Now to my limited experience with it.
 When I work with it on Linux I have to always jump through hoops because
 of:
 https://d.puremagic.com/issues/show_bug.cgi?id=7044


Interesting, I ran into that too with ldc. That, fortunately, is fixable 
on our end.


 Because of that in the end I tend to dodge it completely and roll some
 shell script scaffolding that calls `curl`, as in the cmd-line client.

 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35


Interesting. Never knew of the pattern syntax in there.


 Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT
 shipped with dmd zip or whatever. The better thing is that last time I
 checked there is no proper which means that for Win32 one had to get the
 rare OMF object file format (a custom compilation no less).


Is that file needed if one does NOT use std.net.curl? Would it be a bear 
for us to distributed that particular file?


 You may or may not find them at the VARY BOTTOM of downloads page, and
 figure it out how to fit them, what flags to use etc. Last but not least
 it's Win32 only. No links for Win64.


Which downloads page? Ours or others'?


 All in all, it feels as if somebody actually wanted to sabotage the
 thing, he could score an A+ job as it stands already.


What I see above is it's in our power to fix 
https://d.puremagic.com/issues/show_bug.cgi?id=7044 and that would 
benefit not only curl but other libs as well.



Andrei
Nov 26 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Nov-2013 23:40, Andrei Alexandrescu пишет:

 On 11/26/13 11:23 AM, Dmitry Olshansky wrote:

 For the moment it looks as if half people are pissed off because of the
 work they would need to do on it (workaround, shipping, maintenance,
 etc.), and the other half (using it) are having usability problems on
 all fronts.


 I don't think that's the whole set. Don't forget the forum participation
 is a very small fraction of people who actually use D out there. So
 there are a bunch of people who don't use curl and aren't hurt, and
 presumably quite a few people who do use std.net.curl without troubles
 and don't know we plan to pull the plug on it.


Yeah, there is a big part of people how don't use it and are just fine. 
And some folks that may as well use it (with a custom link step on 
Linux) and be OK. I don't think we should deprecate or remove 
std.net.curl unless a strong alternative comes along.

It seems that radically inclined folks want to get rid of it now, 
presumably for immediate maintenance benefits. The thing is, the best we 
can pull off with deprecation is no less then a year. Then there is 
basically no point in it, we just as well bite the bullet and fix what 
there is to fix in the said 1 year time frame.


 Because of that in the end I tend to dodge it completely and roll some
 shell script scaffolding that calls `curl`, as in the cmd-line client.

 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35


 Interesting. Never knew of the pattern syntax in there.


Me neither, must be some google-foo of mine.


 Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT
 shipped with dmd zip or whatever. The better thing is that last time I
 checked there is no proper which means that for Win32 one had to get the
 rare OMF object file format (a custom compilation no less).


 Is that file needed if one does NOT use std.net.curl? Would it be a bear
 for us to distributed that particular file?


To build phobos - most likely yes. I think we must distribute it, no 
matter how much of LICENSE notices it takes.


 You may or may not find them at the VARY BOTTOM of downloads page, and
 figure it out how to fit them, what flags to use etc. Last but not least
 it's Win32 only. No links for Win64.


 Which downloads page? Ours or others'?


http://dlang.org/download.html



 All in all, it feels as if somebody actually wanted to sabotage the
 thing, he could score an A+ job as it stands already.


 What I see above is it's in our power to fix
 https://d.puremagic.com/issues/show_bug.cgi?id=7044 and that would
 benefit not only curl but other libs as well.


Would be freaking amazing. Walter seemed surprisingly adamant on not 
fixing this (there was a pull I believe), I really have no idea what his 
the reservations are.



-- 
Dmitry Olshansky
Nov 26 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:24 PM, Dmitry Olshansky wrote:

 26-Nov-2013 23:40, Andrei Alexandrescu пишет:

 On 11/26/13 11:23 AM, Dmitry Olshansky wrote:

 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35


 Interesting. Never knew of the pattern syntax in there.


 Me neither, must be some google-foo of mine.


Found it! 
http://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage

Thanks,

Andrei
Nov 26 2013
prev sibling parent reply "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 26 November 2013 at 19:40:10 UTC, Andrei Alexandrescu 
wrote:


 [...] and presumably quite a few people who do use std.net.curl 
 without troubles and don't know we plan to pull the plug on it. 
 [...]


That's me! It works for me and my company.
Nov 26 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 26 November 2013 at 21:12:23 UTC, Andrea Fontana 
wrote:

 On Tuesday, 26 November 2013 at 19:40:10 UTC, Andrei 
 Alexandrescu wrote:


 [...] and presumably quite a few people who do use 
 std.net.curl without troubles and don't know we plan to pull 
 the plug on it. [...]


 That's me! It works for me and my company.


It's worked fine for me as well.

I also don't quite see why the same backwards compatibility 
requirements as for the rest of Phobos shouldn't apply here. If 
libcurl is deemed to be an unacceptable dependency, then it 
should be possible to provide an API-compatible reimplementation 
for at least the parts not directly exposing the libcurl 
interface.

David
Nov 26 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:00, Andrei Alexandrescu wrote:


 So what is the classic recommended solution for such cases? We can't be
 the first people who are having this issue on Unix.


As many have mentioned before. Everything (in this case Phobos) should 
be built on the same platform as it is shipping. So we needed specific 
releases for each Linux distribution we want to support.


 I've asked the question "what exactly is the problem" several times and
 you came closest to answering it, but a bunch of stuff is still unclear
 to me.


On Linux the problem is that Phobos is built on a Debian based system 
where libcurl has symbol names not present in any other libcurl built 
for other distributions.

-- 
/Jacob Carlborg
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:06 PM, Jacob Carlborg wrote:

 On 2013-11-26 20:00, Andrei Alexandrescu wrote:


 So what is the classic recommended solution for such cases? We can't be
 the first people who are having this issue on Unix.


 As many have mentioned before. Everything (in this case Phobos) should
 be built on the same platform as it is shipping. So we needed specific
 releases for each Linux distribution we want to support.


OK, thanks. That seems like something approachable from our end. Is this 
a common approach among other language distributions, e.g. python, ruby, 
go, rust etc. etc? What is a list of platforms we need to support?


 I've asked the question "what exactly is the problem" several times and
 you came closest to answering it, but a bunch of stuff is still unclear
 to me.


 On Linux the problem is that Phobos is built on a Debian based system
 where libcurl has symbol names not present in any other libcurl built
 for other distributions.


I see. So we need more OSs in as many virtual machines. We need to start 
with a list. Looks like something we could dump on the build czar. 
Andrew, are you there? :o)


Andrei
Nov 26 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 21:10, Andrei Alexandrescu wrote:


 OK, thanks. That seems like something approachable from our end. Is this
 a common approach among other language distributions, e.g. python, ruby,
 go, rust etc. etc? What is a list of platforms we need to support?


Yes, that's always the preferred choice to use dist specific packages if 
available. Python and Ruby are available the system package manager in 
all distributions, don't know about Go an Rust.


 I see. So we need more OSs in as many virtual machines. We need to start
 with a list. Looks like something we could dump on the build czar.
 Andrew, are you there? :o)


Yes, that's what we need. I think we can cover most of the distributions 
with a Fedora based and a Debian based system.

-- 
/Jacob Carlborg
Nov 26 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu 
wrote:

 As many have mentioned before. Everything (in this case 
 Phobos) should
 be built on the same platform as it is shipping. So we needed 
 specific
 releases for each Linux distribution we want to support.


 OK, thanks. That seems like something approachable from our 
 end. Is this a common approach among other language 
 distributions, e.g. python, ruby, go, rust etc. etc? What is a 
 list of platforms we need to support?


I have already linked Python example - they provide only source 
tarballs for Linux, no binary distribution. It is assumed that 
Linux way of getting stuff is either using native package manager 
or compiling manually.

One can try to prepare archives for every distro separately but 
it hardly scales. First, you need to decide how many you would 
like to support (few? dozens? hundreds?) Second, those will 
always be inferior to packages supplied from native repository 
(assuming active maintainer) - it is a pretty time-consuming job 
to do this right and no build czar can do it for dozens of 
platforms at once.

I really think providing just source + single additional .deb 
package as an example is the best way to go.
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:21 PM, Dicebot wrote:

 On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu wrote:

 As many have mentioned before. Everything (in this case Phobos) should
 be built on the same platform as it is shipping. So we needed specific
 releases for each Linux distribution we want to support.


 OK, thanks. That seems like something approachable from our end. Is
 this a common approach among other language distributions, e.g.
 python, ruby, go, rust etc. etc? What is a list of platforms we need
 to support?


 I have already linked Python example - they provide only source tarballs
 for Linux, no binary distribution. It is assumed that Linux way of
 getting stuff is either using native package manager or compiling manually.

 One can try to prepare archives for every distro separately but it
 hardly scales. First, you need to decide how many you would like to
 support (few? dozens? hundreds?) Second, those will always be inferior
 to packages supplied from native repository (assuming active maintainer)
 - it is a pretty time-consuming job to do this right and no build czar
 can do it for dozens of platforms at once.

 I really think providing just source + single additional .deb package as
 an example is the best way to go.


Well we kind of do that already. No?

Andrei
Nov 26 2013
next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu 
wrote:

 On 11/26/13 12:21 PM, Dicebot wrote:

 I really think providing just source + single additional .deb 
 package as
 an example is the best way to go.


 Well we kind of do that already. No?


The issue is (mainly) with people expecting the binaries from the 
official .zip file to work on all Linux platforms.

David
Nov 26 2013
parent reply Mike Parker <aldacron gmail.com> writes:
On 11/27/2013 6:12 AM, David Nadlinger wrote:

 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:

 On 11/26/13 12:21 PM, Dicebot wrote:

 I really think providing just source + single additional .deb package as
 an example is the best way to go.


 Well we kind of do that already. No?


 The issue is (mainly) with people expecting the binaries from the
 official .zip file to work on all Linux platforms.


This problem would be solved by going with a dynamic binding for libcurl 
rather than a static binding. Then std.net.curl could load libcurl on 
demand and it doesn't matter how it was compiled (unless, of course, the 
libcurl API changes with compiler flags).
Nov 26 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 27 November 2013 at 01:40:15 UTC, Mike Parker wrote:

 This problem would be solved by going with a dynamic binding 
 for libcurl rather than a static binding. Then std.net.curl 
 could load libcurl on demand and it doesn't matter how it was 
 compiled (unless, of course, the libcurl API changes with 
 compiler flags).


In addition, even with Phobos as a shared library libcurl 
wouldn't necessarily be loaded if it's not actually used then.

David
Nov 26 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu 
wrote:

 I really think providing just source + single additional .deb 
 package as
 an example is the best way to go.


 Well we kind of do that already. No?

 Andrei


No. We provide Linux binaries in .zip and pretend they are 
expected to work for all distros which simply can't happen in 
general. Fedora bug report would not have been valid if not 
implication that there is a supported binary distribution for 
Fedora.

I suggest to simply stop pretending any kind of such support 
exist and drop Linux binaries from .zip at all. Motivating people 
to create dmd packages for their native distros scales much 
better (and solves most/all curl issues for Linux)
Nov 27 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 4:56 AM, Dicebot wrote:

 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:

 I really think providing just source + single additional .deb package as
 an example is the best way to go.


 Well we kind of do that already. No?

 Andrei


 No. We provide Linux binaries in .zip and pretend they are expected to
 work for all distros which simply can't happen in general. Fedora bug
 report would not have been valid if not implication that there is a
 supported binary distribution for Fedora.

 I suggest to simply stop pretending any kind of such support exist and
 drop Linux binaries from .zip at all. Motivating people to create dmd
 packages for their native distros scales much better (and solves
 most/all curl issues for Linux)


I agree the business with distributing all binaries for all platforms in 
the .zip is just weird. We should drop it for the New Year's release.

Andrei
Nov 27 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 27, 2013 at 10:00:38AM -0800, Andrei Alexandrescu wrote:

 On 11/27/13 4:56 AM, Dicebot wrote:


[...]

No. We provide Linux binaries in .zip and pretend they are expected
to work for all distros which simply can't happen in general. Fedora
bug report would not have been valid if not implication that there is
a supported binary distribution for Fedora.

I suggest to simply stop pretending any kind of such support exist
and drop Linux binaries from .zip at all. Motivating people to create
dmd packages for their native distros scales much better (and solves
most/all curl issues for Linux)


 
 I agree the business with distributing all binaries for all
 platforms in the .zip is just weird. We should drop it for the New
 Year's release.


[...]

Didn't Nick already write a script that would generate separate tarballs
/ zipballs / whatnot for each supported platform?

I agree that putting *all* platforms in a *single* zip is very strange.
It makes people wonder, "why did they make me download the binaries for
every single platform out there when I only need support for one
platform?"


T

-- 
Doubt is a self-fulfilling prophecy.
Nov 27 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:15, H. S. Teoh wrote:


 Didn't Nick already write a script that would generate separate tarballs
 / zipballs / whatnot for each supported platform?


Yes, https://github.com/D-Programming-Language/installer/pull/24

-- 
/Jacob Carlborg
Nov 27 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 18:16:34 UTC, H. S. Teoh wrote:

 Didn't Nick already write a script that would generate separate 
 tarballs
 / zipballs / whatnot for each supported platform?


It still implies that "Linux xx-bit" is a single platform which 
is not true.
Nov 27 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:00, Andrei Alexandrescu wrote:


 I agree the business with distributing all binaries for all platforms in
 the .zip is just weird. We should drop it for the New Year's release.


I would hate to see the cross-platform zip gone. It would break DVM and 
probably other tools/scripts.

I don't mind additional packages but please, keep the current zip as well.

-- 
/Jacob Carlborg
Nov 27 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 10:57 AM, Jacob Carlborg wrote:

 On 2013-11-27 19:00, Andrei Alexandrescu wrote:


 I agree the business with distributing all binaries for all platforms in
 the .zip is just weird. We should drop it for the New Year's release.


 I would hate to see the cross-platform zip gone. It would break DVM and
 probably other tools/scripts.

 I don't mind additional packages but please, keep the current zip as well.


Wait, you mean with binaries and all?

Andrei
Nov 27 2013
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 18:58:42 UTC, Andrei 
Alexandrescu wrote:

 Wait, you mean with binaries and all?


Yeah, I like it the way it is too since I use the binaries too, 
building windows programs from my linux box calling wine dmd. My 
scripts currently assume the zip layout and it works pretty 
beautifully.

I hate installers and package managers, the zip rox.


Though, I've said this before, but I could actually live with a 
separate zip per platform if:

1) the folder layouts remains the same. So if I unzip 
dmd-windows.zip, unzip dmd-linux.zip in the same place, it would 
recreate the same directory layout we have now.

2) The Windows zip actually offers a compelling benefit: include 
up-to-date .lib files and more complete Win32 bindings out of the 
box!
Nov 27 2013
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 19:10:26 UTC, Adam D. Ruppe 
wrote:

 On Wednesday, 27 November 2013 at 18:58:42 UTC, Andrei 
 Alexandrescu wrote:

 Wait, you mean with binaries and all?


 Yeah, I like it the way it is too since I use the binaries too, 
 building windows programs from my linux box calling wine dmd. 
 My scripts currently assume the zip layout and it works pretty 
 beautifully.


I don't think Windows binaries will go anywhere as they do work 
on all supported Windows versions. Problem is with considering 
"Linux" a single platform while in practice each distro is more 
like separate operating system (despite being so similar in many 
things).

My vision is that we can keep current layout but to make it 
mandatory to run `make` on target Linux system to create actual 
binaries in the same places they are present right now. It should 
work quite reliably at only added cost of actual compilation time 
(which is relatively small for dmd + phobos)
Nov 27 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:58, Andrei Alexandrescu wrote:


 Wait, you mean with binaries and all?


I mean that I like the current zip to stay.

-- 
/Jacob Carlborg
Nov 27 2013
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 27 Nov 2013 21:08:20 +0100
schrieb Jacob Carlborg <doob me.com>:


 I mean that I like the current zip to stay.


Don't be so ignorant. The zip is broken for all Linux systems
bug Debian. I thought that with D Version Manager you tried to
support more than one Linux distribution...

-- 
Marco
Nov 28 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-28 19:25, Marco Leise wrote:


 Don't be so ignorant. The zip is broken for all Linux systems
 bug Debian. I thought that with D Version Manager you tried to
 support more than one Linux distribution...


Yes, I would like that. But if the the current zip is removed it will 
break _every_ platform.

I can't remember hearing anyone complain about the zip isn't working. 
The only complains I've heard of is with curl and perhaps some ancient 
libc version.

Removing the current zip doesn't fix the problem anyway, it only creates 
new ones. What's the harm in leaving it?

-- 
/Jacob Carlborg
Nov 28 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/13 12:21 PM, Jacob Carlborg wrote:

 On 2013-11-28 19:25, Marco Leise wrote:


 Don't be so ignorant. The zip is broken for all Linux systems
 bug Debian. I thought that with D Version Manager you tried to
 support more than one Linux distribution...


 Yes, I would like that. But if the the current zip is removed it will
 break _every_ platform.

 I can't remember hearing anyone complain about the zip isn't working.


There have been plenty of complaints that it's illogical to distribute 
all binaries in one archive.


 The only complains I've heard of is with curl and perhaps some ancient
 libc version.

 Removing the current zip doesn't fix the problem anyway, it only creates
 new ones. What's the harm in leaving it?


We'll probably keep it but not advertise it. But you're making a 
terrible argument - you want us to keep a bankrupt artifact just because 
you don't want to update your code.


Andrei
Nov 28 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-29 03:39, Andrei Alexandrescu wrote:


 We'll probably keep it but not advertise it. But you're making a
 terrible argument - you want us to keep a bankrupt artifact just because
 you don't want to update your code.


That's the standard argument, "can't remove/change this, will break 
backwards compatibility".

-- 
/Jacob Carlborg
Nov 29 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/13 5:47 AM, Jacob Carlborg wrote:

 On 2013-11-29 03:39, Andrei Alexandrescu wrote:


 We'll probably keep it but not advertise it. But you're making a
 terrible argument - you want us to keep a bankrupt artifact just because
 you don't want to update your code.


 That's the standard argument, "can't remove/change this, will break
 backwards compatibility".


I'd say alternative installers are in a different category than user 
code. Anyhow, you are making a good point that if your tool is out 
there, others may be too.

Andrei
Nov 29 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 18:57:23 UTC, Jacob Carlborg 
wrote:

 On 2013-11-27 19:00, Andrei Alexandrescu wrote:


 I agree the business with distributing all binaries for all 
 platforms in
 the .zip is just weird. We should drop it for the New Year's 
 release.


 I would hate to see the cross-platform zip gone. It would break 
 DVM and probably other tools/scripts.

 I don't mind additional packages but please, keep the current 
 zip as well.


DVM is already broken if it relies on binaries present in .zip
We can't claim that we support things that we don't actually 
support
Nov 27 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 20:13, Dicebot wrote:


 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually support


So we don't support the zip we release every time?

-- 
/Jacob Carlborg
Nov 27 2013
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 20:05:29 UTC, Jacob Carlborg 
wrote:

 On 2013-11-27 20:13, Dicebot wrote:


 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually 
 support


 So we don't support the zip we release every time?


Yes, this is exactly the very reason of this thread. Don't 
support and have never really supported. It does support 
Debian-based stuff and some other distros may happen to work so 
there were no real fuss about it. Does not make it less broken 
thugh.
Nov 27 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 20:13, Dicebot wrote:


 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually support


It works perfectly fine on Mac OS X and Windows. No need to break it on 
all platforms just because one platform is broken. I haven't had any 
problems on Linux either.

-- 
/Jacob Carlborg
Nov 27 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 20:06:52 UTC, Jacob Carlborg 
wrote:

 On 2013-11-27 20:13, Dicebot wrote:


 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually 
 support


 It works perfectly fine on Mac OS X and Windows. No need to 
 break it on all platforms just because one platform is broken. 
 I haven't had any problems on Linux either.


Distributing binaries for Mac and Windows is fine (though it is 
much better to keep those separate archives and patch DVM 
accordingly), this thread is about Linux ones. You did not have 
any problems with it because Debian derivatives take major 
percentage of users and you are likely to have one.
Nov 27 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 21:46, Dicebot wrote:


 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.


If we're removing the zip as it currently is, it's another prof that D 
isn't stable as it claims to be.

And I repeat, why should we break for every platform when it's only one 
platform that is broken.

-- 
/Jacob Carlborg
Nov 27 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 28 November 2013 at 07:36:09 UTC, Jacob Carlborg 
wrote:

 On 2013-11-27 21:46, Dicebot wrote:


 Distributing binaries for Mac and Windows is fine (though it 
 is much
 better to keep those separate archives and patch DVM 
 accordingly), this
 thread is about Linux ones. You did not have any problems with 
 it
 because Debian derivatives take major percentage of users and 
 you are
 likely to have one.


 If we're removing the zip as it currently is, it's another prof 
 that D isn't stable as it claims to be.

 And I repeat, why should we break for every platform when it's 
 only one platform that is broken.


We can leave old .zip as-is just make it explicit that it is 
legacy layout and only 100% expected to work on Debian-compatible 
distros. I really don't care how it is distributed as I am using 
git tag anyway - what I do care about are users having wrong 
expectations when they come to download page.
Nov 28 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-28 15:51, Dicebot wrote:


 We can leave old .zip as-is just make it explicit that it is legacy
 layout and only 100% expected to work on Debian-compatible distros. I
 really don't care how it is distributed as I am using git tag anyway -
 what I do care about are users having wrong expectations when they come
 to download page.


That's only what I wanted.

-- 
/Jacob Carlborg
Nov 28 2013
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 28, 2013 08:36:08 Jacob Carlborg wrote:

 On 2013-11-27 21:46, Dicebot wrote:

 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.


 
 If we're removing the zip as it currently is, it's another prof that D
 isn't stable as it claims to be.


The zip has nothing to do with the stability of D itself. It just has to do 
with the stalibity of how D is distributed, which is a completely different 
issue. I can see why you care about the zip, given that you wrote and maintain 
a tool which relies on it, but calling D unstable because we might change how 
we release it would be like calling KDE or gnome unstable, because they 
changed from distributing their code in tar.gz files to tar.bz2 files. How a 
program or library is distributed has nothing to do with the stability of the 
code itself.

- Jonathan M Davis
Nov 28 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/13 11:17 AM, Jonathan M Davis wrote:

 On Thursday, November 28, 2013 08:36:08 Jacob Carlborg wrote:

 On 2013-11-27 21:46, Dicebot wrote:

 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.


 If we're removing the zip as it currently is, it's another prof that D
 isn't stable as it claims to be.


 The zip has nothing to do with the stability of D itself. It just has to do
 with the stalibity of how D is distributed, which is a completely different
 issue. I can see why you care about the zip, given that you wrote and maintain
 a tool which relies on it, but calling D unstable because we might change how
 we release it would be like calling KDE or gnome unstable, because they
 changed from distributing their code in tar.gz files to tar.bz2 files. How a
 program or library is distributed has nothing to do with the stability of the
 code itself.


Besides the fundamental point remains. People don't need to download all 
platform files on one platform. It just doesn't follow. And the larger 
the distribution will become the more annoying the issue will be.

Andrei
Nov 28 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Nov 28, 2013 at 11:29:04AM -0800, Andrei Alexandrescu wrote:
[...]

 Besides the fundamental point remains. People don't need to download
 all platform files on one platform. It just doesn't follow. And the
 larger the distribution will become the more annoying the issue will
 be.


[...]

I don't see what's the big deal with providing one zip per supported
platform (and each Linux distro is to be considered a separate platform
here), *and* a dmd_everything.zip for distributors who *want* to have
everything in one (they are installing D onto a network of heterogenous
machines, they're making a distribution CD, etc.).

The default download *should* be the platform-specific zip / tarball /
package, but there's no need to kill the all-inclusive zip that contains
literally *everything*, for the people who want it for whatever reason.


T

-- 
First Rule of History: History doesn't repeat itself -- historians merely
repeat each other.
Nov 28 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-28 20:44, H. S. Teoh wrote:


 I don't see what's the big deal with providing one zip per supported
 platform (and each Linux distro is to be considered a separate platform
 here), *and* a dmd_everything.zip for distributors who *want* to have
 everything in one (they are installing D onto a network of heterogenous
 machines, they're making a distribution CD, etc.).

 The default download *should* be the platform-specific zip / tarball /
 package, but there's no need to kill the all-inclusive zip that contains
 literally *everything*, for the people who want it for whatever reason.


Thank you. I agree.

-- 
/Jacob Carlborg
Nov 28 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-28 20:17, Jonathan M Davis wrote:


 The zip has nothing to do with the stability of D itself. It just has to do
 with the stalibity of how D is distributed, which is a completely different
 issue. I can see why you care about the zip, given that you wrote and maintain
 a tool which relies on it, but calling D unstable because we might change how
 we release it would be like calling KDE or gnome unstable, because they
 changed from distributing their code in tar.gz files to tar.bz2 files. How a
 program or library is distributed has nothing to do with the stability of the
 code itself.


So where do we draw the line, what's considered to be part of D, if the 
releases aren't.

-- 
/Jacob Carlborg
Nov 28 2013
parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 28, 2013 21:23:33 Jacob Carlborg wrote:

 On 2013-11-28 20:17, Jonathan M Davis wrote:

 The zip has nothing to do with the stability of D itself. It just has to
 do
 with the stalibity of how D is distributed, which is a completely
 different
 issue. I can see why you care about the zip, given that you wrote and
 maintain a tool which relies on it, but calling D unstable because we
 might change how we release it would be like calling KDE or gnome
 unstable, because they changed from distributing their code in tar.gz
 files to tar.bz2 files. How a program or library is distributed has
 nothing to do with the stability of the code itself.


 
 So where do we draw the line, what's considered to be part of D, if the
 releases aren't.


D is the language specification, dmd is the reference implementation of the 
language, druntime is its standard runtime library, and Phobos is its standard 
library. I don't see how the format that dmd and the standard runtime and 
library are distributed in is at all "part of D." It's just how it's currently 
distributed.

You have a legitimate concern in that you have written a program which relies 
on how dmd is currently distributed, but I don't see how you can argue that 
that has anything to do with what D is.

And while I don't think that we should just willy-nilly change how we 
distribute dmd, I would point out that we never actually promised that 
anything about how dmd is distributed would always stay that way. dmd has been 
distributed as a zip, because that happens to be how Walter decided to do it, 
and there's technically no guarantee that we will continue to do so.

Now, the fact that a useful tool written by one of this community's well-known 
members and which is used by a number of folks in the community relies on the 
zip being put together the way that it's put together and named the way that 
it's name should be taken into consideration when we look at changing how dmd 
is distributed. But I think that you have to acknowledge that as useful as DVM 
is, it's relying on something that was never actually guaranteed. It just 
happens to be how we've done it up to now.

And a number of people have been clamoring for some time that we change how do 
it and that the zip makes no sense, so I find it unlikely that we will continue 
to distribute the zip as-is. Maybe we leave it as-is temporarily, but I think 
that there's a good chance that at some point, you're just going to have to 
change how DVM works if you want it to continue to function - particularly 
since the zip really doesn't make sense and makes less and less sense as time 
goes on (e.g. the shared library situation doesn't work at all with the zip, 
and soon enough that will be more than just Linux).

- Jonathan M Davis
Nov 28 2013
prev sibling parent reply Jordi Sayol <g.sayol yahoo.es> writes:
El 27/11/13 13:56, Dicebot ha escrit:

 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:

 I really think providing just source + single additional .deb package as
 an example is the best way to go.


 Well we kind of do that already. No?

 Andrei


 
 No. We provide Linux binaries in .zip and pretend they are expected to work
for all distros which simply can't happen in general. Fedora bug report would
not have been valid if not implication that there is a supported binary
distribution for Fedora.
 
 I suggest to simply stop pretending any kind of such support exist and drop
Linux binaries from .zip at all. Motivating people to create dmd packages for
their native distros scales much better (and solves most/all curl issues for
Linux)
 


We should then generate dmd packages for every distro release because they can
fail between releases too.

The only thing that breaks the current dmd zip system is libcurl dependency.
All other thinks properly works on all Linux distros/releases.

Should we change all this well working system due to a single problem with an
external library dependency?

-- 
Jordi Sayol
Nov 28 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 28 November 2013 at 15:59:12 UTC, Jordi Sayol wrote:

 We should then generate dmd packages for every distro release 
 because they can fail between releases too.


Or just let specific distro package maintainers do their job.


 The only thing that breaks the current dmd zip system is 
 libcurl dependency. All other thinks properly works on all 
 Linux distros/releases.


Are you actually sure? _all_ stuff? _all_ distros? I doubt it.


 Should we change all this well working system due to a single 
 problem with an external library dependency?


It is not working, it just happens to not work for a very tiny 
user base which is more likely to workaround the issue by 
building from source other than reporting it.

Really, I don't care what the solution is. You can leave current 
archive as-is and close that Fedora issue as invalid, it is not 
that important. Just stop pretending it is supported for all 
platforms and that it can _possibly_ be supported for all 
platforms.
Nov 28 2013
parent reply Jordi Sayol <g.sayol yahoo.es> writes:
El 28/11/13 17:12, Dicebot ha escrit:

 On Thursday, 28 November 2013 at 15:59:12 UTC, Jordi Sayol wrote:

 We should then generate dmd packages for every distro release because they can
fail between releases too.


 
 Or just let specific distro package maintainers do their job.
 

 The only thing that breaks the current dmd zip system is libcurl dependency.
All other thinks properly works on all Linux distros/releases.


 
 Are you actually sure? _all_ stuff? _all_ distros? I doubt it.


Please can you show something on dmd/phobos that fails on any actively
maintained Linux distro that properly works on another one?

Just libcurl dependency.


 

 Should we change all this well working system due to a single problem with an
external library dependency?


 
 It is not working, it just happens to not work for a very tiny user base which
is more likely to workaround the issue by building from source other than
reporting it.
 
 Really, I don't care what the solution is. You can leave current archive as-is
and close that Fedora issue as invalid, it is not that important. Just stop
pretending it is supported for all platforms and that it can _possibly_ be
supported for all platforms.
 


Things, if simpler, much better.

I think is better to keep a single dmd/phobos binary/libray for Linux if they
are capable to work on all active distros, and this is the case without libcurl
dependency.

If a new libcurl D wrapper project is created, I'll be the first to generate
deb packages for it, but out of phobos.

-- 
Jordi Sayol
Nov 28 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 28 November 2013 at 17:12:25 UTC, Jordi Sayol wrote:

 Please can you show something on dmd/phobos that fails on any 
 actively maintained Linux distro that properly works on another 
 one?


The dmd zip doesn't work on CentOS 5 because of a libc version 
mismatch, so it has to be recompiled from source, and then 
sometimes dmd.conf has to be edited too because it passes an 
option that older ld's didn't understand.

CentOS 5 may be old, but it is still in use in a lot of places.


BTW I'm in favor of keeping the zip, it is perfect for me on my 
slackware+wine setup. But adding other options is cool too.
Nov 28 2013
parent Jordi Sayol <g.sayol yahoo.es> writes:
El 28/11/13 18:17, Adam D. Ruppe ha escrit:

 On Thursday, 28 November 2013 at 17:12:25 UTC, Jordi Sayol wrote:

 Please can you show something on dmd/phobos that fails on any actively
maintained Linux distro that properly works on another one?


 
 The dmd zip doesn't work on CentOS 5 because of a libc version mismatch, so it
has to be recompiled from source, and then sometimes dmd.conf has to be edited
too because it passes an option that older ld's didn't understand.
 
 CentOS 5 may be old, but it is still in use in a lot of places.


It properly runs on Centos 6 (6.4).

Which Centos version? 5.0? 5.10?


 
 
 BTW I'm in favor of keeping the zip, it is perfect for me on my slackware+wine
setup. But adding other options is cool too.


Sure, I'm agree.

-- 
Jordi Sayol
Nov 28 2013
prev sibling parent "Parke" <parke.nexus gmail.com> writes:
 On 11/26/13 12:06 PM, Jacob Carlborg wrote:

 As many have mentioned before. Everything (in this case 
 Phobos) should
 be built on the same platform as it is shipping. So we needed 
 specific
 releases for each Linux distribution we want to support.




On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu 
wrote:

 OK, thanks. That seems like something approachable from our 
 end. Is this a common approach among other language 
 distributions, e.g. python, ruby, go, rust etc. etc?


(Greetings all.  I just started using D yesterday.)

Languages that are entirely open source and popular just publish 
their source code, and the Linux distributions compile, package 
and distribute those languages.

For example, if you look at Debian's Python3 package, you can see 
who the package maintainers are (right column, part way down):
http://packages.debian.org/wheezy/python3

The Debian Python3 package maintainers may (or may not) work on 
Python itself.

I believe that parts of DMD are closed source, which may create 
some level of barrier (both technological and ideological) to 
distributing DMD (and its libraries) in the standard Linux 
fashion.
Nov 26 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 01:36, Jonathan M Davis ha escrit:
[...]

 
 At this point, I'm inclined to argue that Phobos should not depend on any 
 libraries other than the system libraries for each OS so that we can avoid 
 further dependencies that may not be available - especially when it comes to 
 Windows, since Windows comes with no 3rd party development libraries at all. 
 So, I would very much be against std.net.curl's inclusion at this point and 
 would argue that it should be in a library separate from Phobos and that if we 
 want similar functionality in Phobos, we need to implement it without relying 
 on anything other than system libraries. And from previous discussions on the 
 topic, I believe that Walter and several other dmd/Phobos devs agree with 
 that.
 
 So, I'm all for removing std.net.curl, but if we do that, we'd obviously need 
 to deprecate it first rather than simply removing it, and we'd need to have a 
 separate library with std.net.curl in it (preferably one which could be pulled 
 in via dub) which people could switch to using instead. Whether that library 
 would be maintained by us, by Jonas (since he created it), or someone else, I 
 don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I 
 think that we should avoid putting any more 3rd party dependencies in Phobos 
 in the future.
 
 - Jonathan M Davis
 


+1

-- 
Jordi Sayol
Nov 25 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:

 (far as I understand, the likes of php and python are doing 
 fine with curl, but maybe I'm wrong).


If you have a look at Python download page 
(http://www.python.org/download/) they don't provide binaries for 
Linux, only source tarball. And Windows installer includes curl 
AFAIK.
Nov 25 2013
prev sibling next sibling parent reply Martin Krejcirik <mk-junk i-line.cz> writes:
On 25.11.2013 8:38, Jordi Sayol wrote:

 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux


I'm for removing as I always compile Phobos without curl anyway (I don't
want to install all dependencies).

-- 
mk
Nov 25 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 5:51 PM, Martin Krejcirik wrote:

 On 25.11.2013 8:38, Jordi Sayol wrote:

 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux


 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).


Does curl in turn depend on many other libraries?

Andrei
Nov 25 2013
parent reply Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 03:11, Andrei Alexandrescu ha escrit:

 On 11/25/13 5:51 PM, Martin Krejcirik wrote:

 On 25.11.2013 8:38, Jordi Sayol wrote:

 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux


 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).


 
 Does curl in turn depend on many other libraries?
 


On Debian testing (64-bit):

$ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
	linux-vdso.so.1 (0x00007fff36519000)
	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f28cc724000)
	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f28cc332000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f28cc0f2000)
	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
(0x00007f28cbee3000)
	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
(0x00007f28cbc92000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26
(0x00007f28cb1f1000)
	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f28ca870000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f28ca462000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f28ca25e000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
(0x00007f28c99fa000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f28c97f6000)

-- 
Jordi Sayol
Nov 25 2013
next sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 26 November 2013 at 02:22:50 UTC, Jordi Sayol wrote:

 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
 	linux-vdso.so.1 (0x00007fff36519000)


/snip

Evidently, Debian builds curl with 
--enable-everything-under-the-sun.  Here's mine, with just ssl 
(via OpenSSL) support:

wyatt Yue ~ $ ldd /usr/lib/libcurl.so
         linux-vdso.so.1 (0x00007fff7f7ff000)
         libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 
(0x00007ff835865000)
         libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 
(0x00007ff83547c000)
         libz.so.1 => /lib64/libz.so.1 (0x00007ff835265000)
         librt.so.1 => /lib64/librt.so.1 (0x00007ff83505c000)
         libc.so.6 => /lib64/libc.so.6 (0x00007ff834cb2000)
         libdl.so.2 => /lib64/libdl.so.2 (0x00007ff834aad000)
         libpthread.so.0 => /lib64/libpthread.so.0 
(0x00007ff834890000)
         /lib64/ld-linux-x86-64.so.2 (0x00007ff835d73000)

I'll be generous and allow that zlib might somehow be a sticking 
point, but there's really nothing unexpected here, IMO.

-Wyatt
Nov 26 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 6:46 AM, Wyatt wrote:

 On Tuesday, 26 November 2013 at 02:22:50 UTC, Jordi Sayol wrote:

 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
     linux-vdso.so.1 (0x00007fff36519000)


 /snip

 Evidently, Debian builds curl with --enable-everything-under-the-sun.
 Here's mine, with just ssl (via OpenSSL) support:

 wyatt Yue ~ $ ldd /usr/lib/libcurl.so
          linux-vdso.so.1 (0x00007fff7f7ff000)
          libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007ff835865000)
          libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0
 (0x00007ff83547c000)
          libz.so.1 => /lib64/libz.so.1 (0x00007ff835265000)
          librt.so.1 => /lib64/librt.so.1 (0x00007ff83505c000)
          libc.so.6 => /lib64/libc.so.6 (0x00007ff834cb2000)
          libdl.so.2 => /lib64/libdl.so.2 (0x00007ff834aad000)
          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff834890000)
          /lib64/ld-linux-x86-64.so.2 (0x00007ff835d73000)

 I'll be generous and allow that zlib might somehow be a sticking point,
 but there's really nothing unexpected here, IMO.


Great. All this is fine - whatever libcurl is installed will be used. Right?


Andrei
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 6:22 PM, Jordi Sayol wrote:

 El 26/11/13 03:11, Andrei Alexandrescu ha escrit:

 On 11/25/13 5:51 PM, Martin Krejcirik wrote:

 On 25.11.2013 8:38, Jordi Sayol wrote:

 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux


 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).


 Does curl in turn depend on many other libraries?


 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
 	linux-vdso.so.1 (0x00007fff36519000)
 	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
 	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
 	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
 	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f28cc724000)
 	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f28cc332000)
 	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f28cc0f2000)
 	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
(0x00007f28cbee3000)
 	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
(0x00007f28cbc92000)
 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
 	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26
(0x00007f28cb1f1000)
 	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
 	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
 	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f28ca870000)
 	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
 	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f28ca462000)
 	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007f28ca25e000)
 	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
 	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
 	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
 	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
 	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
(0x00007f28c99fa000)
 	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f28c97f6000)


But those are already installed libraries, right?

Andrei
Nov 26 2013
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 11:02:06 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 11/25/13 6:22 PM, Jordi Sayol wrote:

 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
 	linux-vdso.so.1 (0x00007fff36519000)
 	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
 	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
 	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
 	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f28cc724000)
 	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f28cc332000)
 	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f28cc0f2000)
 	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
(0x00007f28cbee3000)
 	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
(0x00007f28cbc92000)
 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
 	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26
(0x00007f28cb1f1000)
 	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
 	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
 	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f28ca870000)
 	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
 	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f28ca462000)
 	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007f28ca25e000)
 	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
 	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
 	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
 	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
 	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
(0x00007f28c99fa000)
 	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f28c97f6000)


 
 But those are already installed libraries, right?
 
 Andrei
 


Yes, the curl package on this Linux distribution depends on
all the listed libraries and when curl is installed, they are
installed with it. Often these dependencies are added on a per
case basis. E.g. when the distribution adds a program using
Kerberos network authentication through curl, it is enabled in
the curl package by the maintainer.
A notable exception is with source only distributions like
Gentoo, where packages can depend on curl WITH certain options
enabled. In this case curl is recompiled whenever an
additional requirement occurs and ideally stays as big as
necessary and as small as possible ;)

-- 
Marco
Nov 26 2013
prev sibling parent reply Martin Krejcirik <mk-junk i-line.cz> writes:
On 26.11.2013 20:02, Andrei Alexandrescu wrote:

 (0x00007f28ca047000)
     libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
     libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
     libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
     libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)


 
 But those are already installed libraries, right?


The problem (in my case) is, that you also need to install the -dev
packages of these libraries just to compile Phobos (even if you don't
use curl in your programs at all).

Maybe we just need something like --without-curl for Phobos makefile.

-- 
mk
Nov 27 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 11:26, Martin Krejcirik wrote:


 The problem (in my case) is, that you also need to install the -dev
 packages of these libraries just to compile Phobos (even if you don't
 use curl in your programs at all).

 Maybe we just need something like --without-curl for Phobos makefile.


That's only needed when compiling as a shared library.

-- 
/Jacob Carlborg
Nov 27 2013
prev sibling next sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Wednesday, 27 November 2013 at 10:26:50 UTC, Martin Krejcirik 
wrote:

 The problem (in my case) is, that you also need to install the 
 -dev
 packages of these libraries just to compile Phobos (even if you 
 don't
 use curl in your programs at all).


As usual, -dev packages just make life worse for people and offer 
no real benefit. :/


 Maybe we just need something like --without-curl for Phobos 
 makefile.


If it's _really_ important that a curl binary is distributed with 
D, it should be built statically.  But I doubt it's actually THAT 
important.

Make it a proper dep and let the distro packagers do what they do 
best.  The --without-curl switch is a good idea, btw.

-Wyatt
Nov 27 2013
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 14:07:02 UTC, Wyatt wrote:

 Make it a proper dep and let the distro packagers do what they 
 do best.  The --without-curl switch is a good idea, btw.


Distro packagers don't have any issues even now (assuming they 
build stuff locally). It is all about "official" .zip
Nov 27 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 2:26 AM, Martin Krejcirik wrote:

 On 26.11.2013 20:02, Andrei Alexandrescu wrote:

 (0x00007f28ca047000)
      libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
      libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
      libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
      libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)


 But those are already installed libraries, right?


 The problem (in my case) is, that you also need to install the -dev
 packages of these libraries just to compile Phobos (even if you don't
 use curl in your programs at all).

 Maybe we just need something like --without-curl for Phobos makefile.


We could add that. I'll note that building Phobos is something for the 
experts only.

Andrei
Nov 27 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 27 November 2013 at 17:58:48 UTC, Andrei 
Alexandrescu wrote:

 On 11/27/13 2:26 AM, Martin Krejcirik wrote:

 On 26.11.2013 20:02, Andrei Alexandrescu wrote:

 (0x00007f28ca047000)
     libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
     libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
     libp11-kit.so.0 => 
 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
     libgpg-error.so.0 => 
 /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)


 But those are already installed libraries, right?


 The problem (in my case) is, that you also need to install the 
 -dev
 packages of these libraries just to compile Phobos (even if 
 you don't
 use curl in your programs at all).

 Maybe we just need something like --without-curl for Phobos 
 makefile.


 We could add that. I'll note that building Phobos is something 
 for the experts only.

 Andrei


I'm not sure I would consider myself an expert but I compile 
dmd/druntime/phobos once a day on 3 different machines. Hah, now 
I type that it seems a little excessive!
Nov 27 2013
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 27, 2013 at 07:45:07PM +0100, John Colvin wrote:

 On Wednesday, 27 November 2013 at 17:58:48 UTC, Andrei Alexandrescu
 wrote:


[...]

We could add that. I'll note that building Phobos is something for
the experts only.

Andrei


 
 I'm not sure I would consider myself an expert but I compile
 dmd/druntime/phobos once a day on 3 different machines. Hah, now I
 type that it seems a little excessive!


I don't have 3 different machines, but I find myself compiling git HEAD
so often, that I've written a script that automatically pulls upstream
master, pushes to my github fork, runs make clean, and rebuilds
everything from scratch. Sometimes, I run that script multiple times a
day. :)

But yeah, if you can build the D toolchain by hand, I'd consider you as
an expert. Perhaps not as expert as the phobos devs, maybe, but
definitely not among the lay people. ;-)

To D's credit, though, building the toolchain really *is* as simple as
just running the makefiles in the source code in the right order (dmd,
druntime, then phobos), perhaps with an option or two (like MODEL=64).

Try doing that with, say, the gcc toolchain, and you're in for a journey
of pain that will last, if you're lucky, days, or if not, weeks and
maybe even months. (Unless you're just one of those geniuses that seem
to have no problem whatsoever in taming the gcc / glibc build process.
I'm certainly not among them. The gcc build scripts hate me, and hate me
a lot, for some reason, and they just refuse to work in my environment.
I flinch every time the thought of rebuilding gcc comes to mind. The dmd
toolchain is, in comparison, a pleasure to build.)


T

-- 
A mathematician is a device for turning coffee into theorems. -- P. Erdos
Nov 27 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 5:51 PM, Martin Krejcirik wrote:

 On 25.11.2013 8:38, Jordi Sayol wrote:

 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux


 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).


So then why are you bothered? Are you willing to break a lot of people's 
code because you don't need this feature?

Andrei
Nov 26 2013
parent Martin Krejcirik <mk-junk i-line.cz> writes:
On 26.11.2013 20:01, Andrei Alexandrescu wrote:

 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).


 
 So then why are you bothered? Are you willing to break a lot of people's
 code because you don't need this feature?


I'm not that bothered really, not a major point for me. Just seems more
flexible having the curl wrapper as an independent library. Or some
system for enabling/disabling components (assuming other wrappers will
be added in the future).

-- 
mk
Nov 27 2013
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
I don't use curl (at this point), I use D on Debian and Windows. 
I have had no issues with curl dependencies.

Reading the issues at hand it looks they all come from people 
actually trying to use curl. It sounds like this suggest isn't a 
solution but a means to clean ones hands of the problem.

I don't think I understand having to build dmd just because you 
have a different distribution, Phobos maybe but dmd... Even for 
Phobos I'm skeptical but am not going to install a different 
distro to experiment myself.

If it is moved to a third party library, the problems will still 
remain. If the Phobos team is expected to support it, then the 
burden hasn't shifted either.
Nov 25 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
[...]

 I don't think I understand having to build dmd just because you have
 a different distribution, Phobos maybe but dmd... Even for Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.


If you have used enough Linux distros (or even different versions of the
same distro!), you will realize that all it takes is for a single shared
library on the system to be a different version, missing, or just the
same version built with different compiler flags, and the executable
will not work. Not to mention that filesystem layout can be different
across distros (and versions of the same distro), which will break
things.

Basically, to guarantee a program runs on distro X, the only way is it
has to be built from source. If you're lucky, somebody else has already
done that, and you can just download the binary. (Or better yet,
somebody packaged it for your distro, then you can just install it via
your system's package system.) But if not, you'll just have to do it
yourself.

Copying an executable from one Linux box to another is generally a bad
idea. If it runs at all, there's a chance that it is actually subtly
malfunctioning. I've seen heisenbugs that have no representation in
source code, but show up when you run an executable in an environment
that's close to, but not quite, the environment it was built on.
Rebuilding from source (often with exactly the same commands, same
compile flags, etc.) makes the problems go away.



 If it is moved to a third party library, the problems will still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.


This I agree with. However, it misses the point. The point is that the
core D toolchain (dmd/phobos/druntime) would have one less dependency,
which is generally a good thing, because you want to make it as easy as
possible for people to get a working compiler up and running.  Requiring
this library and that library and that other tool, just to get a bare
bones D toolchain working, is a good way to turn people off D. If a bare
bones D compiler is actually working, then they'd have more incentive to
actually put in more effort to make the frills work too. Having a hello
world program be uncompilable just because libcurl isn't available, is a
very bad first impression ("why does this stupid language need libcurl
just to compile a hello world program?!").


T

-- 
EMACS = Extremely Massive And Cumbersome System
Nov 25 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 07:53, H. S. Teoh wrote:

 Having a hello world program be uncompilable just because libcurl isn't
available, is a
 very bad first impression ("why does this stupid language need libcurl
 just to compile a hello world program?!").


Isn't curl only needed if std.net.curl is actually used? I have never 
used std.net.curl and I never had any problems with it tough. Although I 
use Mac OS X where curl is pre-installed.

-- 
/Jacob Carlborg
Nov 25 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 06:54:51 UTC, H. S. Teoh wrote:

 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]

 I don't think I understand having to build dmd just because 
 you have
 a different distribution, Phobos maybe but dmd... Even for 
 Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.


 If you have used enough Linux distros (or even different 
 versions of the
 same distro!), you will realize that all it takes is for a 
 single shared
 library on the system to be a different version, missing, or 
 just the
 same version built with different compiler flags, and the 
 executable
 will not work.


Yes, the most common being libc. Usually coming from compiling 
against a newer version and trying to use it with an older 
version (but not all projects are as disciplined as libc).


 Not to mention that filesystem layout can be different
 across distros (and versions of the same distro), which will 
 break
 things.


It can, but that isn't relevant to the discussion of linking we 
have here.


 Basically, to guarantee a program runs on distro X, the only 
 way is it
 has to be built from source. If you're lucky, somebody else has 
 already
 done that, and you can just download the binary. (Or better yet,
 somebody packaged it for your distro, then you can just install 
 it via
 your system's package system.) But if not, you'll just have to 
 do it
 yourself.


All you've said is that we are distributing DMD wrong and should 
be building it on every platform, I still don't know what 
building DMD has to do with libcurl though (DMD doesn't use it).


 This I agree with. However, it misses the point. The point is 
 that the
 core D toolchain (dmd/phobos/druntime) would have one less 
 dependency,
 which is generally a good thing, because you want to make it as 
 easy as
 possible for people to get a working compiler up and running.


I didn't miss this point, I started with explaining that I've 
never thought about curl before (other than wondering if I had a 
problem which I needed to solve with it).

Though, it seems libcurl is installed due to other dependencies. 
So I'm less likely to see this issue.

I guess I should mention I don't use the Windows installer, just 
extract the zip. So doubt libcurl is on any of the Windows 
machines I use.
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 10:53 PM, H. S. Teoh wrote:

 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]

 I don't think I understand having to build dmd just because you have
 a different distribution, Phobos maybe but dmd... Even for Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.


 If you have used enough Linux distros (or even different versions of the
 same distro!), you will realize that all it takes is for a single shared
 library on the system to be a different version, missing, or just the
 same version built with different compiler flags, and the executable
 will not work. Not to mention that filesystem layout can be different
 across distros (and versions of the same distro), which will break
 things.

 Basically, to guarantee a program runs on distro X, the only way is it
 has to be built from source. If you're lucky, somebody else has already
 done that, and you can just download the binary. (Or better yet,
 somebody packaged it for your distro, then you can just install it via
 your system's package system.) But if not, you'll just have to do it
 yourself.


Do we with our std.net.curl make this matter any worse?


 If it is moved to a third party library, the problems will still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.


 This I agree with. However, it misses the point. The point is that the
 core D toolchain (dmd/phobos/druntime) would have one less dependency,
 which is generally a good thing, because you want to make it as easy as
 possible for people to get a working compiler up and running.


But my understanding is that the dependency is conditional, i.e. you 
have no dependency on libcurl unless you attempt to use std.net.curl. 
Which strikes me as entirely reasonable. No?


 Requiring
 this library and that library and that other tool, just to get a bare
 bones D toolchain working, is a good way to turn people off D.


Bare bones == no std.net.curl. No?


 If a bare
 bones D compiler is actually working, then they'd have more incentive to
 actually put in more effort to make the frills work too. Having a hello
 world program be uncompilable just because libcurl isn't available, is a
 very bad first impression ("why does this stupid language need libcurl
 just to compile a hello world program?!").


My understanding is that a missing or broken libcurl does not break 
hello world. Is that correct?

As a note regarding my many messages in this thread: I decided to get to 
the bottom of this given that I've asked "what is exactly the problem 
with libcurl" and all I got was unclear answers.


Thanks,

Andrei
Nov 26 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu 
wrote:

 But my understanding is that the dependency is conditional, 
 i.e. you have no dependency on libcurl unless you attempt to 
 use std.net.curl. Which strikes me as entirely reasonable. No?


There is no such thing as _The_ curl library. Different distros 
may have versions with incompatible symbol versioning installed - 
this is the very issue with Fedora which was actively discussed 
in bugzilla.

So yes, dependency is conditional but on some distros it doesn't 
work _at all_, even if libcurl is installed.

There are only 2 possible ways out of it - distributing only 
source tarballs or creating distro-specific distribution archives 
/ packages. Nothing else is guaranteed to work. And latter is 
rarely a good idea.
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:14 AM, Dicebot wrote:

 On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu wrote:

 But my understanding is that the dependency is conditional, i.e. you
 have no dependency on libcurl unless you attempt to use std.net.curl.
 Which strikes me as entirely reasonable. No?


 There is no such thing as _The_ curl library. Different distros may have
 versions with incompatible symbol versioning installed - this is the
 very issue with Fedora which was actively discussed in bugzilla.


But can't we say "to use std.net.curl with dmd 2.xxx you must have 
libcurl 7.10 or higher"?

Andrei
Nov 26 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 19:29:02 UTC, Andrei Alexandrescu 
wrote:

 There is no such thing as _The_ curl library. Different 
 distros may have
 versions with incompatible symbol versioning installed - this 
 is the
 very issue with Fedora which was actively discussed in 
 bugzilla.


 But can't we say "to use std.net.curl with dmd 2.xxx you must 
 have libcurl 7.10 or higher"?

 Andrei


It is not that simple. Well, I remember you taking part in 
https://d.puremagic.com/issues/show_bug.cgi?id=10710 so you 
should be aware of it. It is not about version number of curl but 
about transitive dependency - openssl vs gnutls.

We can say "you must have this version of libcurl built with this 
flags" but it is not much different from saying "use Debian-likes 
or get out".
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:57 AM, Dicebot wrote:

 On Tuesday, 26 November 2013 at 19:29:02 UTC, Andrei Alexandrescu wrote:

 There is no such thing as _The_ curl library. Different distros may have
 versions with incompatible symbol versioning installed - this is the
 very issue with Fedora which was actively discussed in bugzilla.


 But can't we say "to use std.net.curl with dmd 2.xxx you must have
 libcurl 7.10 or higher"?

 Andrei


 It is not that simple. Well, I remember you taking part in
 https://d.puremagic.com/issues/show_bug.cgi?id=10710 so you should be
 aware of it. It is not about version number of curl but about transitive
 dependency - openssl vs gnutls.

 We can say "you must have this version of libcurl built with this flags"
 but it is not much different from saying "use Debian-likes or get out".


What flags are we requiring for use with std.net.curl? Are those 
depending on what specific calls are being used (e.g. using https etc)? 
We could document that stuff. All in all it doesn't seem unreasonable to 
me to require certain specifics for using std.net.curl.

Andrei
Nov 26 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 21:04, Andrei Alexandrescu wrote:


 What flags are we requiring for use with std.net.curl? Are those
 depending on what specific calls are being used (e.g. using https etc)?
 We could document that stuff. All in all it doesn't seem unreasonable to
 me to require certain specifics for using std.net.curl.


Preferably std.net.curl should work with any variant of libcurl. 
Regardless if it's using OpenSSL or GNU TLS.

-- 
/Jacob Carlborg
Nov 26 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 12:04:59 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 What flags are we requiring for use with std.net.curl? Are those 
 depending on what specific calls are being used (e.g. using https etc)? 
 We could document that stuff. All in all it doesn't seem unreasonable to 
 me to require certain specifics for using std.net.curl.
 
 Andrei


Yes, listing requirements is always good to do. But let's take
a look at some hard facts first. `ldd libphobos2.so' on the
linux amd64 version from the zip (for 2.064.2) reveals:

	linux-vdso.so.1 (0x00007fff2fd96000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f1294e66000)
	libcurl-gnutls.so.4 => not found
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1294c48000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f1294951000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f1294748000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f12943a0000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003989a00000)

libcurl-gnutls is not found on *my* installation of Linux.
You correctly made out https (SSL) as a culprit. It is not a
single flag though, but up to 6(!) different SSL "backends"
you can chose from. Typically it comes down to OpenSSL and
GnuTLS. The Debian system that this phobos.so was compiled on
obviously used a GnuTLS version that was suffixed with
"-gnutls" to allow a parallel installation of a curl
version with e.g. OpenSSL. On my Linux on the other hand, I
can only install one version of curl at a time and no suffix
is used. So:

1. curl library names differ on distributions

Next, when you issue:
`readelf --version-info libphobos2.so | less'
you will get a dump of the .gnu.version section that contains
entries like "CURL_GNUTLS_3". These correlate to the
respective "versioned symbols" from libcurl.so. Versioned
symbols are the regular C export names with an  -suffix. The
following command should reveal them:
`readelf -s /usr/lib64/libcurl-gnutls.so.4 | grep GNUTLS_3'
What this means, is that not only the library name phobos.so
uses is linked to the gnutls version, but also each of the SSL
specific functions! This makes for:

2. phobos.so uses GNUTLS version of curl functions

You can rely on the fact that curl is installed on Linux and
that it exposes SSL functionality, but the way phobos.so is
created it is Debian and GnuTLS specific at the moment.

-- 
Marco
Nov 26 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu 
wrote:

 On 11/25/13 10:53 PM, H. S. Teoh wrote:

 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]

 I don't think I understand having to build dmd just because 
 you have
 a different distribution, Phobos maybe but dmd... Even for 
 Phobos
 I'm skeptical but am not going to install a different distro 
 to
 experiment myself.


 If you have used enough Linux distros (or even different 
 versions of the
 same distro!), you will realize that all it takes is for a 
 single shared
 library on the system to be a different version, missing, or 
 just the
 same version built with different compiler flags, and the 
 executable
 will not work. Not to mention that filesystem layout can be 
 different
 across distros (and versions of the same distro), which will 
 break
 things.

 Basically, to guarantee a program runs on distro X, the only 
 way is it
 has to be built from source. If you're lucky, somebody else 
 has already
 done that, and you can just download the binary. (Or better 
 yet,
 somebody packaged it for your distro, then you can just 
 install it via
 your system's package system.) But if not, you'll just have to 
 do it
 yourself.


 Do we with our std.net.curl make this matter any worse?


 If it is moved to a third party library, the problems will 
 still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.


 This I agree with. However, it misses the point. The point is 
 that the
 core D toolchain (dmd/phobos/druntime) would have one less 
 dependency,
 which is generally a good thing, because you want to make it 
 as easy as
 possible for people to get a working compiler up and running.


 But my understanding is that the dependency is conditional, 
 i.e. you have no dependency on libcurl unless you attempt to 
 use std.net.curl. Which strikes me as entirely reasonable. No?


 Requiring
 this library and that library and that other tool, just to get 
 a bare
 bones D toolchain working, is a good way to turn people off D.


 Bare bones == no std.net.curl. No?


 If a bare
 bones D compiler is actually working, then they'd have more 
 incentive to
 actually put in more effort to make the frills work too. 
 Having a hello
 world program be uncompilable just because libcurl isn't 
 available, is a
 very bad first impression ("why does this stupid language need 
 libcurl
 just to compile a hello world program?!").


 My understanding is that a missing or broken libcurl does not 
 break hello world. Is that correct?

 As a note regarding my many messages in this thread: I decided 
 to get to the bottom of this given that I've asked "what is 
 exactly the problem with libcurl" and all I got was unclear 
 answers.


 Thanks,

 Andrei


In some cases, it can prevent the whole stuff to link. Otherwise, 
hello world should simply work.
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 3:03 PM, deadalnix wrote:

 In some cases, it can prevent the whole stuff to link.


This is very vague. Are there cases in which something NOT using 
std.net.curl fails to link?


 Otherwise, hello world should simply work.


I guess that's where we want it.

My conclusion after this whole discussion: in the short term we need to 
have both Fedora and Debian builds. Then we'll move from there.


Andrei
Nov 26 2013
parent Martin Nowak <code dawg.eu> writes:
On 11/27/2013 12:07 AM, Andrei Alexandrescu wrote:

 My conclusion after this whole discussion: in the short term we need to
 have both Fedora and Debian builds. Then we'll move from there.


Right, different distributions => different shared libraries.
Nov 29 2013
prev sibling next sibling parent reply "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.


+1
It should have been removed from Phobos long ago.
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 7:48 AM, Dejan Lekic wrote:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.


 +1
 It should have been removed from Phobos long ago.


The problem is that "long ago" adding libcurl seemed like a perfectly 
reasonable option, particularly considering that we didn't (and we 
don't) have anything to replace it.

Andrei
Nov 26 2013
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 16:39, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 On 11/26/13 7:48 AM, Dejan Lekic wrote:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.



 +1
 It should have been removed from Phobos long ago.



 The problem is that "long ago" adding libcurl seemed like a perfectly
 reasonable option, particularly considering that we didn't (and we don't)
 have anything to replace it.

 Andrei


Especially when you look at the "competitors"

http://curl.haxx.se/libcurl/competitors.html


The checkboxes that would need to be ticked for me:

1. Does it have any third party dependencies?  Preferably no.
2. Can it be built into libphobos easily - see: zlib in phobos, or
sqllite3.amalgamation.c which I thought *was* in phobos at some
point... maybe not?
Nov 26 2013
next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 17:58:46 UTC, Iain Buclaw wrote:

 sqllite3.amalgamation.c which I thought *was* in phobos at some
 point... maybe not?


I had done work in that regard, but it was decided that it 
shouldn't be added so Phobos didn't ship with old versions that 
had known security bugs.
Nov 26 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 18:58, Iain Buclaw wrote:


 Especially when you look at the "competitors"

 http://curl.haxx.se/libcurl/competitors.html


 The checkboxes that would need to be ticked for me:

 1. Does it have any third party dependencies?  Preferably no.
 2. Can it be built into libphobos easily - see: zlib in phobos, or
 sqllite3.amalgamation.c which I thought *was* in phobos at some
 point... maybe not?


What's nice about libcurl, if we're not building or own solution, is 
that is available on basically all platforms and comes pre-installed on 
many.

-- 
/Jacob Carlborg
Nov 26 2013
prev sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei Alexandrescu 
wrote:

 The problem is that "long ago" adding libcurl seemed like a 
 perfectly reasonable option, particularly considering that we 
 didn't (and we don't) have anything to replace it.


Meh, doing the http and smtp stuff isn't rocket science. I did a 
little http library in ~300 lines on my github that some D 
programmers who don't want curl use. Uses openssl for https. curl 
isn't exactly bug free either - I found one a few weeks ago on a 
random stack overflow question about php!

That said, I think curl is ok, we should just bundle it with 
phobos better.
Nov 26 2013
parent reply "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Tuesday, 26 November 2013 at 21:23:05 UTC, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei 
 Alexandrescu wrote:

 The problem is that "long ago" adding libcurl seemed like a 
 perfectly reasonable option, particularly considering that we 
 didn't (and we don't) have anything to replace it.


 Meh, doing the http and smtp stuff isn't rocket science. I did 
 a little http library in ~300 lines on my github that some D 
 programmers who don't want curl use.


Yes - basic http, smtp is easy. But have a look at the libcurl 
API docs and think about how many lines it would take to 
implement that in D.
Nov 26 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 1:33 PM, Jonas Drewsen wrote:

 On Tuesday, 26 November 2013 at 21:23:05 UTC, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei Alexandrescu wrote:

 The problem is that "long ago" adding libcurl seemed like a perfectly
 reasonable option, particularly considering that we didn't (and we
 don't) have anything to replace it.


 Meh, doing the http and smtp stuff isn't rocket science. I did a
 little http library in ~300 lines on my github that some D programmers
 who don't want curl use.


 Yes - basic http, smtp is easy. But have a look at the libcurl API docs
 and think about how many lines it would take to implement that in D.


It seems it's one of those cases in which from proof of concept to bells 
& whistles there's a long way.

Andrei
Nov 26 2013
prev sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 21:33:50 UTC, Jonas Drewsen wrote:

 But have a look at the libcurl API docs and think about how 
 many lines it would take to implement that in D.


Meh, I don't think I've ever needed any of it. Set header and 
request data is enough to do anything on http.
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 2:04 PM, Adam D. Ruppe wrote:

 On Tuesday, 26 November 2013 at 21:33:50 UTC, Jonas Drewsen wrote:

 But have a look at the libcurl API docs and think about how many lines
 it would take to implement that in D.


 Meh, I don't think I've ever needed any of it. Set header and request
 data is enough to do anything on http.


That's quite a claim to make. I know very little on the topic, but from 
what I see at http://curl.haxx.se/changes.html it seems to me like a 
hard working professional is responding to real bug reports and feature 
requests from the field (as opposed to just toying with some 
sophisticated useless artifact).


Andrei
Nov 26 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 23:05:08 UTC, Andrei Alexandrescu 
wrote:

 That's quite a claim to make.


If you've used HTTP, you'll know how true it is. Indeed, since a 
request consists solely of a request and a body, being able to 
change them by definition gives you full access!

The most complex things in http are proxying and caching. curl 
does proxying, wich my 300 line http.d doesn't do. As far as I 
know, curl doesn't do caching at all. (My curl.d offers a cache 
functionality on top of curl, but it ignores the http headers, 
instead opting simply for a max age provided by the programmer, 
e.g. "refresh cache if the file is more than four hours old or 
doesn't exist, otherwise just return the cache file's contents".)


 but from what I see at http://curl.haxx.se/changes.html it 
 seems to me like a hard working professional is responding to 
 real bug reports and feature requests from the field


Yet, very little of that stuff is actually exposed through 
std.net.curl anyway, except perhaps through the download() and 
upload() functions. Phobos doesn't directly expose tftp, imap, 
pop, globbing, ntlm, nor most of the other things mentioned on 
that changelog.

SSL is legitimately a bit of a pain, but curl doesn't actually 
implement ssl, it just uses it, with a few different backends - 
openssl, gnutls, and so forth. Indeed, this is one of the 
difficulties in using curl with D, it is compiled with openssl 
but the client machine uses gnutls or whatever. We'd have that 
same problem.

Async requests are similar to ssl: it should implemented in a 
generic way for all I/O, then the http layer can just use that. 
It's probably just a matter of time before there's demand for an 
async file and socket facility in there anyway.


Again though, my position is that curl is ok and I use it myself, 
but let's not assume we need the entire kitchen sink that curl 
offers, especially given the fact that std.net.curl barely 
scratches it anyway. If someone does need one of curl's less 
common features, it is still there for them, but for the 99% we 
can do with the std.net.curl api, I'm sure we could make that 
happen in ~1000 lines of curl replacement.
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 3:40 PM, Adam D. Ruppe wrote:

 Again though, my position is that curl is ok and I use it myself, but
 let's not assume we need the entire kitchen sink that curl offers,
 especially given the fact that std.net.curl barely scratches it anyway.
 If someone does need one of curl's less common features, it is still
 there for them, but for the 99% we can do with the std.net.curl api, I'm
 sure we could make that happen in ~1000 lines of curl replacement.


Care to make it interesting? I may have some funds for this. But we're 
looking for no less than a glorious 100% replacement.

Andrei
Nov 26 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2013 at 04:19:31PM -0800, Andrei Alexandrescu wrote:

 On 11/26/13 3:40 PM, Adam D. Ruppe wrote:

Again though, my position is that curl is ok and I use it myself, but
let's not assume we need the entire kitchen sink that curl offers,
especially given the fact that std.net.curl barely scratches it
anyway.  If someone does need one of curl's less common features, it
is still there for them, but for the 99% we can do with the
std.net.curl api, I'm sure we could make that happen in ~1000 lines
of curl replacement.


 
 Care to make it interesting? I may have some funds for this. But we're
 looking for no less than a glorious 100% replacement.


[...]

This piqued my interest.

Please excuse another of my TL;DR stories / rants. Some time earlier
this year, I wanted to write a simple utility to (1) login to a website
with a preconfigured username/password; (2) navigate to a specific page
on the site; (3) extract a specific HTML table from that page, and (4)
parse and format the table into an ASCII-graphics text format. Sounds
like the perfect job for a quick-n-dirty D program, right? In the old
days, I'd hack up a Perl script to do this in a couple o' hours, but
this time, I thought, since I have D at my disposal, let's do this the
Right Way(tm).  Where in the past I'd just run wget or curl (the CLI
utility) to fetch the web pages, I thought it'd be much cleaner to do
the web interaction directly so that I didn't have to mess around with
parsing temporary cookie files and what-not just to keep track of my
login session.

So I scrolled through the Phobos docs, and lo and behold, there was
std.net.curl sitting right there, looking all pretty and beckoning to
me. I glanced over its API, and found that it was just the right fit for
what I needed to do. Or so I thought...  About 30 minutes into the
coding, I ran into a blocker: std.net.curl didn't let me specify
Content-Type of POST requests!!! The server simply refused to accept
anything that wasn't explicitly stated to be x/www-urlencoded, but
std.net.curl only allows text/plain or application/octet-stream!  I
simply couldn't login.  Thinking that I must've missed something
obvious, I spent the rest of the day (and probably several more hours
the next day) combing through the docs to find out how to set the
Content-Type to something else. Finally I resorted to reading the
std.net.curl source code, only to discover, to my dismay, that there was
NO way to do this unless I bypassed all the nice high-level syntactic
sugar, and got my hands down and dirty with low-level libcurl API calls
(or their thin wrappers thereof).

At that point, I threw up my hands. The original task was far too simple
for something this heavy-handed, so I backed off and said, it was nice
knowing you, std.net.curl, but I think we're better off going our
separate ways.

TL;DR: I would *love* to see a replacement for std.net.curl that isn't
so straitjacketed that I can't even do something so basic as logging on
to a website. I'm sure there are very good reasons why the current API
is the way it is, but the fact of the matter is that, from an end-user's
POV, its limitations were a deal-breaker.

So, I'm all for a D-only replacement of std.net.curl with a superior
API. Getting rid of the external dependence on libcurl would be an added
bonus.


T

-- 
"I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you
already said that." -- User-Friendly
Nov 26 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 00:49:57 UTC, H. S. Teoh wrote:

 Sounds like the perfect job for a quick-n-dirty D program, 
 right?


Oh yeah, my http.d and dom.d can make short work of that. I had a 
similar problem last year and ended up getting it to work very 
easily (well, that one used my curl.d, but http.d does cookies 
too now).


 coding, I ran into a blocker: std.net.curl didn't let me specify
 Content-Type of POST requests!!!


.....holy crap, you're not kidding. I thought addRequestHeader 
can do it, but that duplicates the header. Unbelievable.

My curl.d is an ugly piece of garbage. Seriously, I hate it, just 
look at this signature:

string curlAuth(string url, string data = null, string username = 
null, string password = null, string contentType = 
"application/x-www-form-urlencoded", string methodOverride = 
null, string[] customHeaders = null, string cookieJar = null) {

but at least it gets the job done - I use it on my work projects 
to access all kinds of things from arbitrary website downloads 
(one of the work sites offers a link posting functionality 
similar to Facebook, it downloads the site and reads description, 
images, etc. to make a richer link preview) to a variety of APIs 
like bing search, Facebook, twitter, linkedin, youtube, 
Authorize.net, I've done a lot with it, and especially my helper 
oauth.d file for many of them, all built on top of this hideous 
do-it-all function.

If I was doing a whole new interface, I think I'd prefer to do an 
object that keeps this state internally. So it'd be like:

auto client = new HttpClient("host", port);
client.userAgent = "whatever";
// authorization would ideally be a UFCS function or at least do 
some template method check, so it can be extended by things like 
oauth
client.authorization = BasicAuth("username", "password");

client.host = "host override if you want.com";

auto response = client.post("/login", ["username":"lol", 
"password":"rofl"]);

// or
client.post("/login", "text/plain", "text");

// or
client.post("/login", "text/xml", someXmlInputRange);

or something like that. Url and method could be set via 
properties too; get, post, head, put, delete, and so on could 
perhaps be implemented in terms of opDispatch that sets the 
properties then calls .execute();


Anyway, the important thing would be if you do future changes to 
this object, it maintains a bit of state. Cookies and such. That 
could work with async too, since the response returned would be 
more like a handle, so you then do

response.wait();

or do callbacks or whatever.


But the point is it'd be nice and simple for most series of 
requests.
Nov 26 2013
prev sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei 
Alexandrescu wrote:

 Care to make it interesting? I may have some funds for this. 
 But we're looking for no less than a glorious 100% replacement.


Meh, my first choice is still to just bundle curl with phobos, 
like we do with zlib. It seems silly to me to want a glorious 
100% replacement of an *open source* library.

Regardless, I probably wouldn't be the ideal choice for changing 
the implementation because I barely use the interface: I wrote my 
own http modules, including one that uses curl and one that 
doesn't, before std.net.curl came around. The phobos thing had no 
compelling benefit to me, so I never switched most my code.

Thus, I'm not very familiar with std.net.curl's strengths and 
weaknesses and would likely break it somehow while changing the 
implementation.


...unless doing a new interface is on the table too. Then, we can 
leave std.net.curl exactly how it is, so people who use it don't 
have broken code, while a new std.net.http, std.net.smtp, 
std.net.ftp, and so on are phased in for people who want them. I 
could get behind that.
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 5:05 PM, Adam D. Ruppe wrote:

 On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei Alexandrescu wrote:

 Care to make it interesting? I may have some funds for this. But we're
 looking for no less than a glorious 100% replacement.


 Meh, my first choice is still to just bundle curl with phobos, like we
 do with zlib. It seems silly to me to want a glorious 100% replacement
 of an *open source* library.

 Regardless, I probably wouldn't be the ideal choice for changing the
 implementation because I barely use the interface: I wrote my own http
 modules, including one that uses curl and one that doesn't, before
 std.net.curl came around. The phobos thing had no compelling benefit to
 me, so I never switched most my code.

 Thus, I'm not very familiar with std.net.curl's strengths and weaknesses
 and would likely break it somehow while changing the implementation.


 ...unless doing a new interface is on the table too. Then, we can leave
 std.net.curl exactly how it is, so people who use it don't have broken
 code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are
 phased in for people who want them. I could get behind that.


A new interface is also on the table, but that brings the additional 
burden of defining your own design. std.net.curl has already been approved.

Andrei
Nov 26 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
Alexandrescu wrote:

 A new interface is also on the table, but that brings the 
 additional burden of defining your own design. std.net.curl has 
 already been approved.


Aye, but that'd be worth it. There's a few pieces we should get
together:

1) a url module. My cgi.d has one that may be usable with a
little testing:
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022

about 150 lines long, including some unittest. Also has some
functions that build on std.uri to make encoding multiple values
easier - working with associative arrays instead of just strings.

This is useful since we can then pass type-safe urls to various
functions and get consistent parsing and creation. The basedOn()
method in my code there is meant to do relative links in a
browser fetch situation. Regardless, all this is fairly useful
independent of network functions too.

2) Some kind of SSL sockets, including adding ssl to an already
open socket (needed for STARTTLS on smtp at least). I did a
partial implementation as a subclass of Phobos' Socket with
openssl. Should also do a Windows impl and probably a gnutls as
well.

I think this would be ideally picked at runtime, with dlopen().
If gnutls is available, make that work. Otherwise, use openssl,
and if none are there, throw upon ssl requests. On Windows, it
can just use the built in ssl facilities the operating system
provides.

I'd probably make this a new module that extends std.socket
rather than editing std/socket.d.

3) async (overlapped) I/O on those sockets. This might need
std.socket to be edited to add the functionality.



Then, we go into the curl replacement:


4) The http module. Offers simple functions (get(), post() that
just work), a reusable client object, and advanced functions, all
in sync and async versions.

My http.d can be a starting point, though I have a new api in
mind. I'll perhaps slap together a first draft over the next
week, if my other work leaves enough time. I have to write a book
chapter on Phobos soon too! lol maybe time to scratch the
std.net.curl portion.

5) The smtp module. Offers the low level send functions, and a
higher level MIME message builder class. I've written the latter
already.
https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/email.d#L27

though it uses my html to text module, which uses my dom module,
which uses my characterencodings module. That could be easily
pulled out though, it would just require the user to set the
plain text version of an html email themselves.

As you can see there too, it currently uses std.net.curl to
actually send the email, since my smtp function (written but it
isn't great) doesn't support the starttls command that Amazon SES
uses and i needed that for a work project...

My module can also read email, but I don't think that's needed in
the std.net.email.

6) The ftp module. I haven't written ftp code for a long time, it
would perhaps be better if someone else wrote this, though if
nobody else does, I'm sure I can refresh myself well enough with
a lil time. But the beauty of modules is we can divide and
conquer!

7) Any other protocols we care about. Personally, I'd kinda love
some ssh and scp code... but I'm afraid actually writing that is
something I don't want to do, perhaps we can do a wrapper for
libssh. Perhaps not appropriate for phobos due to dependency and
licensing but I'd still love someone to write it and post a link
here!

8) If we want to, there could be a combined module that reads a
runtime url and dispatches to the appropriate protocol, like the
std.net.curl.download and upload can do. But tbh I don't even
care much about this. It is trivial once the other modules are
written anyway.

9) Maybe my OAuth module's client code can be cleaned up and 
added too, as an add-on to the http functionality.


Each piece here is pretty independent, except the first three,
and I have enough written already for them to get started at
least.
Nov 26 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 05:09, Adam D. Ruppe wrote:


 2) Some kind of SSL sockets, including adding ssl to an already
 open socket (needed for STARTTLS on smtp at least). I did a
 partial implementation as a subclass of Phobos' Socket with
 openssl. Should also do a Windows impl and probably a gnutls as
 well.


Doesn't GnuTLS provide an OpenSSL API?

-- 
/Jacob Carlborg
Nov 26 2013
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 07:39:17 UTC, Jacob Carlborg 
wrote:

 Doesn't GnuTLS provide an OpenSSL API?


idk. If so, that's easy. I've just seen curl compiled separately 
for gnutls and openssl so I assumed they must be different.

Nevertheless, I really think dynamic linking is the way to go 
here since that can gracefully degrade when no ssl library is 
available at all too.
Nov 27 2013
prev sibling parent reply "Andrea Fontana" <nospam example.com> writes:
On Wednesday, 27 November 2013 at 04:09:53 UTC, Adam D. Ruppe 
wrote:

 On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
 Alexandrescu wrote:

 A new interface is also on the table, but that brings the 
 additional burden of defining your own design. std.net.curl 
 has already been approved.


 Aye, but that'd be worth it. There's a few pieces we should get
 together:

 1) a url module. My cgi.d has one that may be usable with a
 little testing:
 https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022


Should we consider interoperability with std.path too?


 4) The http module. Offers simple functions (get(), post() that
 just work), a reusable client object, and advanced functions, 
 all
 in sync and async versions.


I worked on http protocol, and I don't think that's will be a big 
issue. Http client side is not so complex if you need a working 
subset. I think we don't need gzip compression in first release, 
for example.


 9) Maybe my OAuth module's client code can be cleaned up and 
 added too, as an add-on to the http functionality.


+1 for an oauth module. It would be very useful to work with 
third party api (facebook, google, etc ...) and to write api too! 
However I think it depends on http(s) module.
Nov 27 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 08:45:37 UTC, Andrea Fontana 
wrote:

 Should we consider interoperability with std.path too?


Generally, I think no, though it would be easy enough to get 
uri.path as a decoded string and pass it to those functions.

But std.path is a bit different on Windows and Linux (I think; 
I've never actually used it), whereas urls have a slightly 
different and stricter interpretation (which may be treated like 
win or lin depending on the server).


 I worked on http protocol, and I don't think that's will be a 
 big issue. Http client side is not so complex if you need a 
 working subset. I think we don't need gzip compression in first 
 release, for example.


Yeah, http is pretty simple. Even gzip isn't really hard.


 +1 for an oauth module. It would be very useful to work with 
 third party api (facebook, google, etc ...) and to write api 
 too! However I think it depends on http(s) module.


Yup. I wasn't thinking of putting the server code in the 
std.net.http, though I have some in my oauth.d (isSignatureValid, 
isOauthRequest, etc., the functions that take a Cgi parameter) 
I've barely actually used it, so it is probably buggy, but it is 
there and wasn't horribly hard to write after getting the client 
working.

(The biggest pain with clients btw? Buggy servers! It has gotten 
a little better, but when I started with this, Twitter, LinkedIn, 
and AWeber, the three I had to use.... all interpreted the spec 
slightly differently, so what worked on one wouldn't work on the 
other two, and, of course, the error messages are  never helpful 
and the documentation sucks. Massive pain.)
Nov 27 2013
parent "Andrea Fontana" <nospam example.com> writes:
On Wednesday, 27 November 2013 at 14:16:33 UTC, Adam D. Ruppe 
wrote:

 (The biggest pain with clients btw? Buggy servers! It has 
 gotten a little better, but when I started with this, Twitter, 
 LinkedIn, and AWeber, the three I had to use.... all 
 interpreted the spec slightly differently, so what worked on 
 one wouldn't work on the other two, and, of course, the error 
 messages are  never helpful and the documentation sucks. 
 Massive pain.)


Because they started using oauth 2.0 when rfc draft changes every 
day. I remember the pain. Now it seems that there's a final rfc. 
I remember that google implemented draft #xx twitter draft #yy 
and so on...
Nov 27 2013
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, November 27, 2013 02:05:39 Adam D. Ruppe wrote:

 ...unless doing a new interface is on the table too. Then, we can
 leave std.net.curl exactly how it is, so people who use it don't
 have broken code, while a new std.net.http, std.net.smtp,
 std.net.ftp, and so on are phased in for people who want them. I
 could get behind that.


If we were going to replace std.net.curl, I would fully expect us to end up 
with a new API, because the current API is what is beacuse of what curl can 
and can't do rather than because it's the best API for what you're 
conceptually trying to do. The new API might end up being fairly close to the 
current one, but I would not consider it desirable to try and maintain a 100% 
compatible API if we were replacing the current implementation except in cases 
where the current API is the best choice even without curl underneath it. 
Maybe that would be most of the API, but I'd hate to see us put ourselves in a 
straightjacket based on how curl works when writing an API which didn't depend 
on curl - particularly as writing our own would give us the opportunity to 
make sure that we had something that made the most conceptual sense rather 
than necessarily having what curl has.

- Jonathan M Davis
Nov 26 2013
parent reply "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Wednesday, 27 November 2013 at 01:32:18 UTC, Jonathan M Davis 
wrote:

 On Wednesday, November 27, 2013 02:05:39 Adam D. Ruppe wrote:

 ...unless doing a new interface is on the table too. Then, we 
 can
 leave std.net.curl exactly how it is, so people who use it 
 don't
 have broken code, while a new std.net.http, std.net.smtp,
 std.net.ftp, and so on are phased in for people who want them. 
 I
 could get behind that.


 If we were going to replace std.net.curl, I would fully expect 
 us to end up
 with a new API, because the current API is what is beacuse of 
 what curl can
 and can't do rather than because it's the best API for what 
 you're
 conceptually trying to do. The new API might end up being 
 fairly close to the
 current one, but I would not consider it desirable to try and 
 maintain a 100%
 compatible API if we were replacing the current implementation 
 except in cases
 where the current API is the best choice even without curl 
 underneath it.
 Maybe that would be most of the API, but I'd hate to see us put 
 ourselves in a
 straightjacket based on how curl works when writing an API 
 which didn't depend
 on curl - particularly as writing our own would give us the 
 opportunity to
 make sure that we had something that made the most conceptual 
 sense rather
 than necessarily having what curl has.


I definitely agree. In addition D has obtained some nice features 
since the definition of the std.net.curl API that I would liked 
have used back then.

I do believe that people underestimate the work needed to make a 
proper http client implementation with the bells and whristles 
that people expect like cookie store, chuncked transfers, ssl, 
authentication... and handle wierd per server type bugs as well.

I definitely think that a native solution is the end goal here. 
But if that is the case then it should be based on a proper event 
based system (std.event?) to support async handling (peek at how 
vibe has done it very nicely). But that in turn depends somewhat 
on allocators being well defined and implemented as I see it.

-Jonas
Nov 28 2013
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 28 November 2013 at 11:37:45 UTC, Jonas Drewsen 
wrote:

 I definitely agree. In addition D has obtained some nice 
 features since the definition of the std.net.curl API that I 
 would liked have used back then.


What are you thinking of there? I think plain old classes and 
delegates are a good fit for http stuff.
Nov 28 2013
parent reply "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Thursday, 28 November 2013 at 16:24:34 UTC, Adam D. Ruppe 
wrote:

 On Thursday, 28 November 2013 at 11:37:45 UTC, Jonas Drewsen 
 wrote:

 I definitely agree. In addition D has obtained some nice 
 features since the definition of the std.net.curl API that I 
 would liked have used back then.


 What are you thinking of there? I think plain old classes and 
 delegates are a good fit for http stuff.


Indeed, the basic design for wrapping curl is not that bad I 
think. One thing that was not available was UFCS which I think 
could have made the API look a bit different.

Another thing that could have have affected the API is support 
for named parameters which is still not supported (and probably 
never will?).

/Jonas
Dec 02 2013
parent "Suliman" <evermind live.ru> writes:
Till we have such issues 
http://forum.dlang.org/thread/mailman.357.1384163617.9546.digital
ars-d puremagic.com 
I think we should have native replacement.
May 07 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 7:48 AM, Dejan Lekic wrote:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.


 +1
 It should have been removed from Phobos long ago.


Why?

Andrei
Nov 26 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 19:07, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 On 11/26/13 7:48 AM, Dejan Lekic wrote:

 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.



 +1
 It should have been removed from Phobos long ago.



 Why?

 Andrei


The problem I think is a distribution one.  Not a compiler/library problem.

Remembering to include '-lcurl' when importing std.net.curl is perhaps
one urk.  But this will be a no longer a problem if libphobos is built
shared.
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:13 AM, Iain Buclaw wrote:

 Remembering to include '-lcurl' when importing std.net.curl is perhaps
 one urk.  But this will be a no longer a problem if libphobos is built
 shared.


Interesting. Can we supply a pragma that automates the flag thingie?

Andrei
Nov 26 2013
next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 19:21, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

 On 11/26/13 11:13 AM, Iain Buclaw wrote:

 Remembering to include '-lcurl' when importing std.net.curl is perhaps
 one urk.  But this will be a no longer a problem if libphobos is built
 shared.



 Interesting. Can we supply a pragma that automates the flag thingie?



If pragma(lib) worked for me. (not talking dmd here :-)
Nov 26 2013
prev sibling parent "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 26 November 2013 at 19:21:31 UTC, Andrei Alexandrescu 
wrote:

 On 11/26/13 11:13 AM, Iain Buclaw wrote:

 Remembering to include '-lcurl' when importing std.net.curl is 
 perhaps
 one urk.  But this will be a no longer a problem if libphobos 
 is built
 shared.


 Interesting. Can we supply a pragma that automates the flag 
 thingie?

 Andrei



It doesn't work: 
http://forum.dlang.org/thread/yosxnqziavcnnxkfhtgm forum.dlang.org
Nov 29 2013
prev sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 19:13:40 UTC, Iain Buclaw wrote:

 Remembering to include '-lcurl' when importing std.net.curl is 
 perhaps
 one urk.  But this will be a no longer a problem if libphobos 
 is built
 shared.


Isn't that like -lm for C programs? Doesn't seem big to me.
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.


As others said, Deimos isn't really the correct place for it 
since it is a wrapper.

I'm fine with it being removed but only after dub has become the 
official package manager and there is a replacement for it on dub 
that is easy for users to switch to.
Nov 26 2013
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
Personally I have always thought of std.net.curl as a temporary 
solution until a native implementation in D could be created. But 
I was also fully aware that people were not standing in line to 
make a native one and that we needed something else in the 
meantime. That is why I implemented it in the first place.

So until there is a proper replacement in phobos I really think 
it should stay, since a standard library really should include 
networking support out of the box IMHO.

/Jonas
Nov 26 2013
prev sibling parent "nazriel" <spam dzfl.pl> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:

 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.


We need something to replace std.net.curl with in the first place.

The problem is that if want to support for example HTTPS or FTPS 
we would end with dependencie (on openssl for example) anyways.

Another thing is that making anything more than trivial requests, 
requires a bit of work - compressions, encodings, authentication, 
RFC (*Grins) etc - it all makes it complex task.

Removing std.net.curl at this point will probably break a bit of 
code.
About cases where people have problems with linking when using 
std.net.curl for http - advise them to use Vibe.d http client or 
Vladimir's ae.net.* stuff or Adam's Ruppe custom framework? 
Dunno. Merge any of those 3 projects work into Phobos? Bounty for 
D implementation of libCURL - like for phobos inclusion?
Nov 26 2013