www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should "std.net.curl" be moved from Phobos to Deimos?

reply Jordi Sayol <g.sayol yahoo.es> writes:
As Jonathan M Davis said:
---
Several of the main devs (including Walter) have stated that having
std.net.curl on Phobos was a mistake given all of the problems that we've had
with it on Windows and Linux as well, and at least some of them have expressed
a desire for it to be removed. I expect that there's a good chance that it can
and will be removed from Phobos if brought up for discussion.

Certainly, I think that it's clear that we will not add any further external
dependencies like curl, because curl has proven to be a big problem. It's very
desirable to have bindings and wrappers for C libraries - but putting them in
the standard library when it's not guaranteed that the appropriate library is
on the system by default has proven to be too problematic, so they should be
left to external projects.
---

"std.net.curl" can also be moved from Phobos to Deimos.
Deimos can be rethink, i.e. new build master can add a make building script on
Deimos in order to compile all projects included on it, generating
"libdeimos.a" and "libdeimos.so", documentation, etc.

-- 
Jordi Sayol
Nov 24 2013
next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.

Deimos currently only hosts bindings, not wrappers, as part of its objective. So, etc.c.curl would be appropriate, but std.net.curl would be unprecedented. I think both etc.c.curl and std.net.curl should just be ripped out and moved into a CurlD (or whatever) project, hosted by someone on Github (such as Jonas Drewsen or a maintainer), on the grounds that I don't think Phobos is a place for wrapper libraries. I think std.net.curl is a well designed wrapper library, and I would happily continue to use it in my projects, but I don't think it's appropriate for Phobos. As an aside, personally I think Deimos is pointless; and worse, horribly executed (A Github organization? Really?). A simple wiki page like this one[1] can do the job much better and without the ridiculous self-imposed disadvantages we are putting up with by using a Github organization. [1] http://wiki.dlang.org/Bindings
Nov 25 2013
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
Am 25.11.2013 09:32, schrieb Jakob Ovrum:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

Deimos currently only hosts bindings, not wrappers, as part of its objective. So, etc.c.curl would be appropriate, but std.net.curl would be unprecedented. I think both etc.c.curl and std.net.curl should just be ripped out and moved into a CurlD (or whatever) project, hosted by someone on Github (such as Jonas Drewsen or a maintainer), on the grounds that I don't think Phobos is a place for wrapper libraries. I think std.net.curl is a well designed wrapper library, and I would happily continue to use it in my projects, but I don't think it's appropriate for Phobos. As an aside, personally I think Deimos is pointless; and worse, horribly executed (A Github organization? Really?). A simple wiki page like this one[1] can do the job much better and without the ridiculous self-imposed disadvantages we are putting up with by using a Github organization. [1] http://wiki.dlang.org/Bindings

Or, even better (because it doesn't get out of date), an automatically generated list like this one: http://code.dlang.org/?sort=name&category=library.binding
Nov 25 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.

I have to second this. It has been a major pain to get D to work in some environment because of the dependency on curl.
Nov 25 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 1:34 AM, deadalnix wrote:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

I have to second this. It has been a major pain to get D to work in some environment because of the dependency on curl.

The continuing stream of issues associated with curl is puzzling me. We chose curl _exactly_ because (a) it was available or trivial to install everywhere, (b) it had accreted through experience a bunch of tricks that made little sense to rediscover the hard way. Now it seems curl is failing at least the first premise, so we should consider various alternatives. First I'd like to gather an understanding on why we seem to have this problem (far as I understand, the likes of php and python are doing fine with curl, but maybe I'm wrong). So is there a short and simple list of issues that anyone can put together? If we do decide to do away with libcurl, one possible solution would be to embed its source code within our build. That way we wouldn't break code that already uses it. Andrei
Nov 25 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 4:36 PM, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:
 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).

A major difference is there's only one php/python binary, usually build on the same system that uses it. Phobos, on the other hand, is generally built on the packager's computer, which isn't always binary compatible with the deployment box. On Windows, the problem is simply that curl isn't packaged with dmd, for some weird reason, meaning people have to get curl.lib separately. That's idiotic. But then again, so are a lot of the dmd Windows deficiencies.

Please let me know whether my understanding of the situation is correct: 1. If people have a working installation of libcurl on their machine, we work with it. 2. Otherwise, phobos works fine but attempting to use std.net.curl will fail. Is this correct? If not, please explain exactly why. If yes, this setup seems entirely appropriate to me.
 If we do decide to do away with libcurl, one possible solution would
 be to embed its source code within our build. That way we wouldn't
 break code that already uses it.

Yes, that would be ideal, we should just statically link curl right into the phobos build so it just works everywhere.

There are several questions associated with this. 1. Does the author of libcurl agree with such? My understanding is he would. 2. Would we need to actually build libcurl from source, or just distribute it? In the former case, it would be quite odd that we'd need to do that (and presumably we'd need to take some maintenance burden, too). 3. What if people already have some working libcurl version and would want to use that? Andrei
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 4:36 PM, Jonathan M Davis wrote:
 The number one problem on Windows is the fact that libcurl does not come with
 Windows and that you have to get a version of it build which is compatible
 with dmd has proven to be a huge hurdle for Windows developers.

Why is "compatible with dmd" an issue? Do we make it any harder for libcurl than other clients? Could you please explain?
 There may be
 other problems, but that's the biggest. Distributing libcurl with dmd could fix
 that, but IIRC, there was some reason why we couldn't do that (or maybe it was
 just that Walter didn't want to - I can't remember).

Walter is opposed only to building libcurl from source for distributing it, which is an understandable stance.
 On Linux, I believe that the biggest problem is that dmd then has to be built
 on your specific distro for it to work with your distro, so the libraries in
 the zip file were only working on debian-based systems. Regardless of why
 libcurl is not distributed with dmd on Windows, we couldn't distribute for
 Linux to fix this problem, because that would conflict with the system's
 libcurl.

So what is the classic recommended solution for such cases? We can't be the first people who are having this issue on Unix.
 There may be other problems, but I think that those are the two bigs ones, and
 they've come up several times (particularly the issues with libcurl and
 Windows).

I've asked the question "what exactly is the problem" several times and you came closest to answering it, but a bunch of stuff is still unclear to me.
 There's also Don's recent rant on why curl was ever supported in the first
 place, but I guess that he just wasn't paying enough attention to it given
 that he only complained about it recently rather than during the review:

 http://forum.dlang.org/thread/CADpwU1cu=0r+NJ+4GN9P9Fd_XisK1dsduYrrkjpGAvM0YJ_tUA mail.gmail.com

First, it's not like we slipped curl in while nobody was looking. Second, I'm sure curl has its shortcomings, but I'd be hard pressed to find better alternatives. As to just dumping curl and doing our own: I just looked and libcurl has 160K lines. Those include examples and probably a fair amount of header chaff etc. that's unneeded in D, but I'd assess there are on the order of 100K lines of "pulp" code that does real work. Simply dumping std.net.curl would not only break existing code that depends on it in unthinkable ways (no replacement and no recourse), but would put us some 100KLOC behind in terms of a working replacement.
 At this point, I'm inclined to argue that Phobos should not depend on any
 libraries other than the system libraries for each OS so that we can avoid
 further dependencies that may not be available - especially when it comes to
 Windows, since Windows comes with no 3rd party development libraries at all.
 So, I would very much be against std.net.curl's inclusion at this point and
 would argue that it should be in a library separate from Phobos and that if we
 want similar functionality in Phobos, we need to implement it without relying
 on anything other than system libraries. And from previous discussions on the
 topic, I believe that Walter and several other dmd/Phobos devs agree with
 that.

I don't agree with that. Or I don't understand what you're asking. If what you're asking is we break all existing code using std.net.curl with no suggested replacement, that would be pretty bad, no? If I understand correctly the complaint is: "people who don't have curl can't use std.net.curl". That's to be expected. It's in the name! A more reasonable complaint is: "people who don't have curl can't use high-level HTTP/FTP client networking, which is expected of a modern language's standard library". For those we should NOT throw away libcurl. We should instead focus on providing e.g. std.net.durl which is an in-house library for doing great stuff. Then people can choose. In the long term we may even recommend moving away from curl because durl is mature and better etc.
 So, I'm all for removing std.net.curl, but if we do that, we'd obviously need
 to deprecate it first rather than simply removing it, and we'd need to have a
 separate library with std.net.curl in it (preferably one which could be pulled
 in via dub) which people could switch to using instead. Whether that library
 would be maintained by us, by Jonas (since he created it), or someone else, I
 don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I
 think that we should avoid putting any more 3rd party dependencies in Phobos
 in the future.

This is probably going to come again. We can't build everything in house in due time. Do we have an openssl layer written in D? Andrei
Nov 26 2013
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Nov-2013 23:00, Andrei Alexandrescu пишет:
 There may be other problems, but I think that those are the two bigs
 ones, and
 they've come up several times (particularly the issues with libcurl and
 Windows).

I've asked the question "what exactly is the problem" several times and you came closest to answering it, but a bunch of stuff is still unclear to me.

I thought I'd chime in on this. I find that if we actually worked on getting curl integrated better it may just as well be an all right network client lib. For the moment it looks as if half people are pissed off because of the work they would need to do on it (workaround, shipping, maintenance, etc.), and the other half (using it) are having usability problems on all fronts. Now to my limited experience with it. When I work with it on Linux I have to always jump through hoops because of: https://d.puremagic.com/issues/show_bug.cgi?id=7044 Because of that in the end I tend to dodge it completely and roll some shell script scaffolding that calls `curl`, as in the cmd-line client. See some hilarious things I had to do to link it properly: https://github.com/blackwhale/tools/blob/master/posix.mak#L35 Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT shipped with dmd zip or whatever. The better thing is that last time I checked there is no proper which means that for Win32 one had to get the rare OMF object file format (a custom compilation no less). You may or may not find them at the VARY BOTTOM of downloads page, and figure it out how to fit them, what flags to use etc. Last but not least it's Win32 only. No links for Win64. All in all, it feels as if somebody actually wanted to sabotage the thing, he could score an A+ job as it stands already. P.S. And before I forgot - there are specifics problems with curl Linux distros that are not Debian derived and with shared phobos in particular, but there are fokls that must known this side of coin better then I. -- Dmitry Olshansky
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:23 AM, Dmitry Olshansky wrote:
 I find that if we actually worked on getting curl integrated better it
 may just as well be an all right network client lib.

Great.
 For the moment it looks as if half people are pissed off because of the
 work they would need to do on it (workaround, shipping, maintenance,
 etc.), and the other half (using it) are having usability problems on
 all fronts.

I don't think that's the whole set. Don't forget the forum participation is a very small fraction of people who actually use D out there. So there are a bunch of people who don't use curl and aren't hurt, and presumably quite a few people who do use std.net.curl without troubles and don't know we plan to pull the plug on it.
 Now to my limited experience with it.
 When I work with it on Linux I have to always jump through hoops because
 of:
 https://d.puremagic.com/issues/show_bug.cgi?id=7044

Interesting, I ran into that too with ldc. That, fortunately, is fixable on our end.
 Because of that in the end I tend to dodge it completely and roll some
 shell script scaffolding that calls `curl`, as in the cmd-line client.

 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35

Interesting. Never knew of the pattern syntax in there.
 Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT
 shipped with dmd zip or whatever. The better thing is that last time I
 checked there is no proper which means that for Win32 one had to get the
 rare OMF object file format (a custom compilation no less).

Is that file needed if one does NOT use std.net.curl? Would it be a bear for us to distributed that particular file?
 You may or may not find them at the VARY BOTTOM of downloads page, and
 figure it out how to fit them, what flags to use etc. Last but not least
 it's Win32 only. No links for Win64.

Which downloads page? Ours or others'?
 All in all, it feels as if somebody actually wanted to sabotage the
 thing, he could score an A+ job as it stands already.

What I see above is it's in our power to fix https://d.puremagic.com/issues/show_bug.cgi?id=7044 and that would benefit not only curl but other libs as well. Andrei
Nov 26 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Nov-2013 23:40, Andrei Alexandrescu пишет:
 On 11/26/13 11:23 AM, Dmitry Olshansky wrote:
 For the moment it looks as if half people are pissed off because of the
 work they would need to do on it (workaround, shipping, maintenance,
 etc.), and the other half (using it) are having usability problems on
 all fronts.

I don't think that's the whole set. Don't forget the forum participation is a very small fraction of people who actually use D out there. So there are a bunch of people who don't use curl and aren't hurt, and presumably quite a few people who do use std.net.curl without troubles and don't know we plan to pull the plug on it.

Yeah, there is a big part of people how don't use it and are just fine. And some folks that may as well use it (with a custom link step on Linux) and be OK. I don't think we should deprecate or remove std.net.curl unless a strong alternative comes along. It seems that radically inclined folks want to get rid of it now, presumably for immediate maintenance benefits. The thing is, the best we can pull off with deprecation is no less then a year. Then there is basically no point in it, we just as well bite the bullet and fix what there is to fix in the said 1 year time frame.
 Because of that in the end I tend to dodge it completely and roll some
 shell script scaffolding that calls `curl`, as in the cmd-line client.

 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35

Interesting. Never knew of the pattern syntax in there.

Me neither, must be some google-foo of mine.
 Now if that doesn't set off the alarms. Win32/64 curl libraries are NOT
 shipped with dmd zip or whatever. The better thing is that last time I
 checked there is no proper which means that for Win32 one had to get the
 rare OMF object file format (a custom compilation no less).

Is that file needed if one does NOT use std.net.curl? Would it be a bear for us to distributed that particular file?

To build phobos - most likely yes. I think we must distribute it, no matter how much of LICENSE notices it takes.
 You may or may not find them at the VARY BOTTOM of downloads page, and
 figure it out how to fit them, what flags to use etc. Last but not least
 it's Win32 only. No links for Win64.

Which downloads page? Ours or others'?

http://dlang.org/download.html
 All in all, it feels as if somebody actually wanted to sabotage the
 thing, he could score an A+ job as it stands already.

What I see above is it's in our power to fix https://d.puremagic.com/issues/show_bug.cgi?id=7044 and that would benefit not only curl but other libs as well.

Would be freaking amazing. Walter seemed surprisingly adamant on not fixing this (there was a pull I believe), I really have no idea what his the reservations are. -- Dmitry Olshansky
Nov 26 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:24 PM, Dmitry Olshansky wrote:
 26-Nov-2013 23:40, Andrei Alexandrescu пишет:
 On 11/26/13 11:23 AM, Dmitry Olshansky wrote:
 See some hilarious things I had to do to link it properly:
 https://github.com/blackwhale/tools/blob/master/posix.mak#L35

Interesting. Never knew of the pattern syntax in there.

Me neither, must be some google-foo of mine.

Found it! http://www.gnu.org/software/make/manual/html_node/Static-Usage.html#Static-Usage Thanks, Andrei
Nov 26 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:39 AM, Brad Anderson wrote:
 On Tuesday, 26 November 2013 at 19:23:54 UTC, Dmitry Olshansky wrote:
 https://d.puremagic.com/issues/show_bug.cgi?id=7044

This is a constant annoyance of mine using std.net.curl on Linux.
 Now if that doesn't set off the alarms. Win32/64 curl libraries are
 NOT shipped with dmd zip or whatever. The better thing is that last
 time I checked there is no proper which means that for Win32 one had
 to get the rare OMF object file format (a custom compilation no less).

Nah, it was as simple as generating an import library from the DLL. There's no need to build libcurl yourself to do that. The installer automatically downloads one I prepared a long time ago already. brocolis recently prepared an updated libcurl zip with the OMF import library as well as 64-bit support. It also uses WinSSL instead of OpenSSL so using it is even easier than it was before. The installer has already been updated for it: https://github.com/D-Programming-Language/installer/pull/27 With that libcurl should just work on Windows for both 32 and 64 bits. Linux problems are basically all that remains of the libcurl issues.

This is pretty awesome. Is brocolis around on this forum? Andrei
Nov 26 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:13 AM, Brad Anderson wrote:
 To use 32-bit curl you need to generate an OMF import library from the
 libcurl DLL. It's not terribly hard. I did it originally for when I
 added downloading of libcurl to the Windows installer.

When is the OMF library needed? 1. While building dmd 2. While building druntime 3. While building phobos 4. While building client code that does NOT use std.net.curl 5. While building client code that DOES use std.net curl
 It's not at all
 obvious to users that they need to distribute libcurl.dll et al when
 they distribute their application but that's a documentation issue.

Or they could kick the can down the road by requiring their customers to have it.
 If I remember correctly Walter didn't want to distribute in the dmd zip
 anything that wasn't boost licensed which is why curl on Windows comes
 as a separate download. zlib is a notable exception but I seem to recall
 Walter regretted including zlib.

On the other hand we've hardly heard one peep from anyone in literally years that their zip code has trouble building. We can ask the libcurl author if it comes down to redistributing a libcurl binary. But we want to make sure this is the solution of choice.
 I can't remember off the top of my head if libcurl's license allows it
 to be statically linked with a boost licensed library but that would
 certainly simplify things for end users (but complicate the building of
 phobos a bit).

My reading of http://curl.haxx.se/docs/copyright.html is that the author does his best to open up his library. Andrei
Nov 26 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:27, Andrei Alexandrescu wrote:

 When is the OMF library needed?

 1. While building dmd

 2. While building druntime

 3. While building phobos

 4. While building client code that does NOT use std.net.curl

 5. While building client code that DOES use std.net curl

Not that I have tried it but I'm pretty sure it is 5.
 My reading of http://curl.haxx.se/docs/copyright.html is that the author
 does his best to open up his library.

See this post: http://forum.dlang.org/thread/i0dlde$59$1 digitalmars.com#post-i0eaoc:2410bs:241:40digitalmars.com -- /Jacob Carlborg
Nov 26 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:56 AM, Brad Anderson wrote:
 On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei Alexandrescu wrote:
 On 11/26/13 11:13 AM, Brad Anderson wrote:
 To use 32-bit curl you need to generate an OMF import library from the
 libcurl DLL. It's not terribly hard. I did it originally for when I
 added downloading of libcurl to the Windows installer.

When is the OMF library needed? 1. While building dmd 2. While building druntime 3. While building phobos 4. While building client code that does NOT use std.net.curl 5. While building client code that DOES use std.net curl

5. std.net.curl actually existed for several releases of dmd before I went to use it one day and got linker errors and decided to fix it with the installer change.

Awesome. So can we assume https://github.com/D-Programming-Language/installer/pull/27 takes care of Windows for good? Thanks for your work!!! I'll make sure I'll get a word of thanks to broccolis as well.
 We can ask the libcurl author if it comes down to redistributing a
 libcurl binary. But we want to make sure this is the solution of choice.

Walter actually already did that and the author of libcurl said redistributing it was fine. Here's an old thread discussing the licensing: http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org

Terrific, thanks. Andrei
Nov 26 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:13, Brad Anderson wrote:

 If I remember correctly Walter didn't want to distribute in the dmd zip
 anything that wasn't boost licensed which is why curl on Windows comes
 as a separate download. zlib is a notable exception but I seem to recall
 Walter regretted including zlib.

The zip already contains tools which the source code isn't available. Like "shell", "dumpobj" and "obj2asm". But I guess Walter, DigitalMars or Symantec has created those.
 I can't remember off the top of my head if libcurl's license allows it
 to be statically linked with a boost licensed library but that would
 certainly simplify things for end users (but complicate the building of
 phobos a bit).

Walter contacted the author and asked about that, here's the post: http://forum.dlang.org/thread/i0dlde$59$1 digitalmars.com#post-i0eaoc:2410bs:241:40digitalmars.com -- /Jacob Carlborg
Nov 26 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-26 20:00, Andrei Alexandrescu wrote:

 So what is the classic recommended solution for such cases? We can't be
 the first people who are having this issue on Unix.

As many have mentioned before. Everything (in this case Phobos) should be built on the same platform as it is shipping. So we needed specific releases for each Linux distribution we want to support.
 I've asked the question "what exactly is the problem" several times and
 you came closest to answering it, but a bunch of stuff is still unclear
 to me.

On Linux the problem is that Phobos is built on a Debian based system where libcurl has symbol names not present in any other libcurl built for other distributions. -- /Jacob Carlborg
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:06 PM, Jacob Carlborg wrote:
 On 2013-11-26 20:00, Andrei Alexandrescu wrote:

 So what is the classic recommended solution for such cases? We can't be
 the first people who are having this issue on Unix.

As many have mentioned before. Everything (in this case Phobos) should be built on the same platform as it is shipping. So we needed specific releases for each Linux distribution we want to support.

OK, thanks. That seems like something approachable from our end. Is this a common approach among other language distributions, e.g. python, ruby, go, rust etc. etc? What is a list of platforms we need to support?
 I've asked the question "what exactly is the problem" several times and
 you came closest to answering it, but a bunch of stuff is still unclear
 to me.

On Linux the problem is that Phobos is built on a Debian based system where libcurl has symbol names not present in any other libcurl built for other distributions.

I see. So we need more OSs in as many virtual machines. We need to start with a list. Looks like something we could dump on the build czar. Andrew, are you there? :o) Andrei
Nov 26 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 21:10, Andrei Alexandrescu wrote:

 OK, thanks. That seems like something approachable from our end. Is this
 a common approach among other language distributions, e.g. python, ruby,
 go, rust etc. etc? What is a list of platforms we need to support?

Yes, that's always the preferred choice to use dist specific packages if available. Python and Ruby are available the system package manager in all distributions, don't know about Go an Rust.
 I see. So we need more OSs in as many virtual machines. We need to start
 with a list. Looks like something we could dump on the build czar.
 Andrew, are you there? :o)

Yes, that's what we need. I think we can cover most of the distributions with a Fedora based and a Debian based system. -- /Jacob Carlborg
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 12:21 PM, Dicebot wrote:
 On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu wrote:
 As many have mentioned before. Everything (in this case Phobos) should
 be built on the same platform as it is shipping. So we needed specific
 releases for each Linux distribution we want to support.

OK, thanks. That seems like something approachable from our end. Is this a common approach among other language distributions, e.g. python, ruby, go, rust etc. etc? What is a list of platforms we need to support?

I have already linked Python example - they provide only source tarballs for Linux, no binary distribution. It is assumed that Linux way of getting stuff is either using native package manager or compiling manually. One can try to prepare archives for every distro separately but it hardly scales. First, you need to decide how many you would like to support (few? dozens? hundreds?) Second, those will always be inferior to packages supplied from native repository (assuming active maintainer) - it is a pretty time-consuming job to do this right and no build czar can do it for dozens of platforms at once. I really think providing just source + single additional .deb package as an example is the best way to go.

Well we kind of do that already. No? Andrei
Nov 26 2013
next sibling parent Mike Parker <aldacron gmail.com> writes:
On 11/27/2013 6:12 AM, David Nadlinger wrote:
 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:
 On 11/26/13 12:21 PM, Dicebot wrote:
 I really think providing just source + single additional .deb package as
 an example is the best way to go.

Well we kind of do that already. No?

The issue is (mainly) with people expecting the binaries from the official .zip file to work on all Linux platforms.

This problem would be solved by going with a dynamic binding for libcurl rather than a static binding. Then std.net.curl could load libcurl on demand and it doesn't matter how it was compiled (unless, of course, the libcurl API changes with compiler flags).
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 4:56 AM, Dicebot wrote:
 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:
 I really think providing just source + single additional .deb package as
 an example is the best way to go.

Well we kind of do that already. No? Andrei

No. We provide Linux binaries in .zip and pretend they are expected to work for all distros which simply can't happen in general. Fedora bug report would not have been valid if not implication that there is a supported binary distribution for Fedora. I suggest to simply stop pretending any kind of such support exist and drop Linux binaries from .zip at all. Motivating people to create dmd packages for their native distros scales much better (and solves most/all curl issues for Linux)

I agree the business with distributing all binaries for all platforms in the .zip is just weird. We should drop it for the New Year's release. Andrei
Nov 27 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:15, H. S. Teoh wrote:

 Didn't Nick already write a script that would generate separate tarballs
 / zipballs / whatnot for each supported platform?

Yes, https://github.com/D-Programming-Language/installer/pull/24 -- /Jacob Carlborg
Nov 27 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:00, Andrei Alexandrescu wrote:

 I agree the business with distributing all binaries for all platforms in
 the .zip is just weird. We should drop it for the New Year's release.

I would hate to see the cross-platform zip gone. It would break DVM and probably other tools/scripts. I don't mind additional packages but please, keep the current zip as well. -- /Jacob Carlborg
Nov 27 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 10:57 AM, Jacob Carlborg wrote:
 On 2013-11-27 19:00, Andrei Alexandrescu wrote:

 I agree the business with distributing all binaries for all platforms in
 the .zip is just weird. We should drop it for the New Year's release.

I would hate to see the cross-platform zip gone. It would break DVM and probably other tools/scripts. I don't mind additional packages but please, keep the current zip as well.

Wait, you mean with binaries and all? Andrei
Nov 27 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 19:58, Andrei Alexandrescu wrote:

 Wait, you mean with binaries and all?

I mean that I like the current zip to stay. -- /Jacob Carlborg
Nov 27 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-28 19:25, Marco Leise wrote:

 Don't be so ignorant. The zip is broken for all Linux systems
 bug Debian. I thought that with D Version Manager you tried to
 support more than one Linux distribution...

Yes, I would like that. But if the the current zip is removed it will break _every_ platform. I can't remember hearing anyone complain about the zip isn't working. The only complains I've heard of is with curl and perhaps some ancient libc version. Removing the current zip doesn't fix the problem anyway, it only creates new ones. What's the harm in leaving it? -- /Jacob Carlborg
Nov 28 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/13 12:21 PM, Jacob Carlborg wrote:
 On 2013-11-28 19:25, Marco Leise wrote:

 Don't be so ignorant. The zip is broken for all Linux systems
 bug Debian. I thought that with D Version Manager you tried to
 support more than one Linux distribution...

Yes, I would like that. But if the the current zip is removed it will break _every_ platform. I can't remember hearing anyone complain about the zip isn't working.

There have been plenty of complaints that it's illogical to distribute all binaries in one archive.
 The only complains I've heard of is with curl and perhaps some ancient
 libc version.

 Removing the current zip doesn't fix the problem anyway, it only creates
 new ones. What's the harm in leaving it?

We'll probably keep it but not advertise it. But you're making a terrible argument - you want us to keep a bankrupt artifact just because you don't want to update your code. Andrei
Nov 28 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-29 03:39, Andrei Alexandrescu wrote:

 We'll probably keep it but not advertise it. But you're making a
 terrible argument - you want us to keep a bankrupt artifact just because
 you don't want to update your code.

That's the standard argument, "can't remove/change this, will break backwards compatibility". -- /Jacob Carlborg
Nov 29 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/29/13 5:47 AM, Jacob Carlborg wrote:
 On 2013-11-29 03:39, Andrei Alexandrescu wrote:

 We'll probably keep it but not advertise it. But you're making a
 terrible argument - you want us to keep a bankrupt artifact just because
 you don't want to update your code.

That's the standard argument, "can't remove/change this, will break backwards compatibility".

I'd say alternative installers are in a different category than user code. Anyhow, you are making a good point that if your tool is out there, others may be too. Andrei
Nov 29 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 20:13, Dicebot wrote:

 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually support

So we don't support the zip we release every time? -- /Jacob Carlborg
Nov 27 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 20:13, Dicebot wrote:

 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually support

It works perfectly fine on Mac OS X and Windows. No need to break it on all platforms just because one platform is broken. I haven't had any problems on Linux either. -- /Jacob Carlborg
Nov 27 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-27 21:46, Dicebot wrote:

 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.

If we're removing the zip as it currently is, it's another prof that D isn't stable as it claims to be. And I repeat, why should we break for every platform when it's only one platform that is broken. -- /Jacob Carlborg
Nov 27 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/28/13 11:17 AM, Jonathan M Davis wrote:
 On Thursday, November 28, 2013 08:36:08 Jacob Carlborg wrote:
 On 2013-11-27 21:46, Dicebot wrote:
 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.

If we're removing the zip as it currently is, it's another prof that D isn't stable as it claims to be.

The zip has nothing to do with the stability of D itself. It just has to do with the stalibity of how D is distributed, which is a completely different issue. I can see why you care about the zip, given that you wrote and maintain a tool which relies on it, but calling D unstable because we might change how we release it would be like calling KDE or gnome unstable, because they changed from distributing their code in tar.gz files to tar.bz2 files. How a program or library is distributed has nothing to do with the stability of the code itself.

Besides the fundamental point remains. People don't need to download all platform files on one platform. It just doesn't follow. And the larger the distribution will become the more annoying the issue will be. Andrei
Nov 28 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-28 20:44, H. S. Teoh wrote:

 I don't see what's the big deal with providing one zip per supported
 platform (and each Linux distro is to be considered a separate platform
 here), *and* a dmd_everything.zip for distributors who *want* to have
 everything in one (they are installing D onto a network of heterogenous
 machines, they're making a distribution CD, etc.).

 The default download *should* be the platform-specific zip / tarball /
 package, but there's no need to kill the all-inclusive zip that contains
 literally *everything*, for the people who want it for whatever reason.

Thank you. I agree. -- /Jacob Carlborg
Nov 28 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-28 15:51, Dicebot wrote:

 We can leave old .zip as-is just make it explicit that it is legacy
 layout and only 100% expected to work on Debian-compatible distros. I
 really don't care how it is distributed as I am using git tag anyway -
 what I do care about are users having wrong expectations when they come
 to download page.

That's only what I wanted. -- /Jacob Carlborg
Nov 28 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-28 20:17, Jonathan M Davis wrote:

 The zip has nothing to do with the stability of D itself. It just has to do
 with the stalibity of how D is distributed, which is a completely different
 issue. I can see why you care about the zip, given that you wrote and maintain
 a tool which relies on it, but calling D unstable because we might change how
 we release it would be like calling KDE or gnome unstable, because they
 changed from distributing their code in tar.gz files to tar.bz2 files. How a
 program or library is distributed has nothing to do with the stability of the
 code itself.

So where do we draw the line, what's considered to be part of D, if the releases aren't. -- /Jacob Carlborg
Nov 28 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 27, 2013 at 10:00:38AM -0800, Andrei Alexandrescu wrote:
 On 11/27/13 4:56 AM, Dicebot wrote:

No. We provide Linux binaries in .zip and pretend they are expected
to work for all distros which simply can't happen in general. Fedora
bug report would not have been valid if not implication that there is
a supported binary distribution for Fedora.

I suggest to simply stop pretending any kind of such support exist
and drop Linux binaries from .zip at all. Motivating people to create
dmd packages for their native distros scales much better (and solves
most/all curl issues for Linux)

I agree the business with distributing all binaries for all platforms in the .zip is just weird. We should drop it for the New Year's release.

Didn't Nick already write a script that would generate separate tarballs / zipballs / whatnot for each supported platform? I agree that putting *all* platforms in a *single* zip is very strange. It makes people wonder, "why did they make me download the binaries for every single platform out there when I only need support for one platform?" T -- Doubt is a self-fulfilling prophecy.
Nov 27 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 18:16:34 UTC, H. S. Teoh wrote:
 Didn't Nick already write a script that would generate separate 
 tarballs
 / zipballs / whatnot for each supported platform?

It still implies that "Linux xx-bit" is a single platform which is not true.
Nov 27 2013
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Monday, 25 November 2013 at 10:08:12 UTC, Sönke Ludwig wrote:
 Or, even better (because it doesn't get out of date), an 
 automatically
 generated list like this one:
 http://code.dlang.org/?sort=name&category=library.binding

Yeah, absolutely :)
Nov 25 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:

 The continuing stream of issues associated with curl is 
 puzzling me. We chose curl _exactly_ because (a) it was 
 available or trivial to install everywhere, (b) it had accreted 
 through experience a bunch of tricks that made little sense to 
 rediscover the hard way.

 Now it seems curl is failing at least the first premise, so we 
 should consider various alternatives. First I'd like to gather 
 an understanding on why we seem to have this problem (far as I 
 understand, the likes of php and python are doing fine with 
 curl, but maybe I'm wrong). So is there a short and simple list 
 of issues that anyone can put together?

 If we do decide to do away with libcurl, one possible solution 
 would be to embed its source code within our build. That way we 
 wouldn't break code that already uses it.


 Andrei

A solution could be to ship curl with phobos, if license allow to do so.
Nov 25 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:
 First I'd like to gather an understanding on why we seem to 
 have this problem (far as I understand, the likes of php and 
 python are doing fine with curl, but maybe I'm wrong).

A major difference is there's only one php/python binary, usually build on the same system that uses it. Phobos, on the other hand, is generally built on the packager's computer, which isn't always binary compatible with the deployment box. On Windows, the problem is simply that curl isn't packaged with dmd, for some weird reason, meaning people have to get curl.lib separately. That's idiotic. But then again, so are a lot of the dmd Windows deficiencies.
 If we do decide to do away with libcurl, one possible solution 
 would be to embed its source code within our build. That way we 
 wouldn't break code that already uses it.

Yes, that would be ideal, we should just statically link curl right into the phobos build so it just works everywhere.
Nov 25 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, November 25, 2013 16:13:57 Andrei Alexandrescu wrote:
 The continuing stream of issues associated with curl is puzzling me. We
 chose curl _exactly_ because (a) it was available or trivial to install
 everywhere, (b) it had accreted through experience a bunch of tricks
 that made little sense to rediscover the hard way.
 
 Now it seems curl is failing at least the first premise, so we should
 consider various alternatives. First I'd like to gather an understanding
 on why we seem to have this problem (far as I understand, the likes of
 php and python are doing fine with curl, but maybe I'm wrong). So is
 there a short and simple list of issues that anyone can put together?
 
 If we do decide to do away with libcurl, one possible solution would be
 to embed its source code within our build. That way we wouldn't break
 code that already uses it.

The number one problem on Windows is the fact that libcurl does not come with Windows and that you have to get a version of it build which is compatible with dmd has proven to be a huge hurdle for Windows developers. There may be other problems, but that's the biggest. Distributing libcurl with dmd could fix that, but IIRC, there was some reason why we couldn't do that (or maybe it was just that Walter didn't want to - I can't remember). On Linux, I believe that the biggest problem is that dmd then has to be built on your specific distro for it to work with your distro, so the libraries in the zip file were only working on debian-based systems. Regardless of why libcurl is not distributed with dmd on Windows, we couldn't distribute for Linux to fix this problem, because that would conflict with the system's libcurl. There may be other problems, but I think that those are the two bigs ones, and they've come up several times (particularly the issues with libcurl and Windows). There's also Don's recent rant on why curl was ever supported in the first place, but I guess that he just wasn't paying enough attention to it given that he only complained about it recently rather than during the review: http://forum.dlang.org/thread/CADpwU1cu=0r+NJ+4GN9P9Fd_XisK1dsduYrrkjpGAvM0YJ_tUA mail.gmail.com At this point, I'm inclined to argue that Phobos should not depend on any libraries other than the system libraries for each OS so that we can avoid further dependencies that may not be available - especially when it comes to Windows, since Windows comes with no 3rd party development libraries at all. So, I would very much be against std.net.curl's inclusion at this point and would argue that it should be in a library separate from Phobos and that if we want similar functionality in Phobos, we need to implement it without relying on anything other than system libraries. And from previous discussions on the topic, I believe that Walter and several other dmd/Phobos devs agree with that. So, I'm all for removing std.net.curl, but if we do that, we'd obviously need to deprecate it first rather than simply removing it, and we'd need to have a separate library with std.net.curl in it (preferably one which could be pulled in via dub) which people could switch to using instead. Whether that library would be maintained by us, by Jonas (since he created it), or someone else, I don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I think that we should avoid putting any more 3rd party dependencies in Phobos in the future. - Jonathan M Davis
Nov 25 2013
prev sibling next sibling parent reply Martin Krejcirik <mk-junk i-line.cz> writes:
On 25.11.2013 8:38, Jordi Sayol wrote:
 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux

I'm for removing as I always compile Phobos without curl anyway (I don't want to install all dependencies). -- mk
Nov 25 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 5:51 PM, Martin Krejcirik wrote:
 On 25.11.2013 8:38, Jordi Sayol wrote:
 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux

I'm for removing as I always compile Phobos without curl anyway (I don't want to install all dependencies).

Does curl in turn depend on many other libraries? Andrei
Nov 25 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 6:22 PM, Jordi Sayol wrote:
 El 26/11/13 03:11, Andrei Alexandrescu ha escrit:
 On 11/25/13 5:51 PM, Martin Krejcirik wrote:
 On 25.11.2013 8:38, Jordi Sayol wrote:
 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux

I'm for removing as I always compile Phobos without curl anyway (I don't want to install all dependencies).

Does curl in turn depend on many other libraries?

On Debian testing (64-bit): $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0 linux-vdso.so.1 (0x00007fff36519000) libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000) librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000) libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000) libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f28cc724000) libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f28cc332000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f28cc0f2000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f28cbee3000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f28cbc92000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000) libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f28cb1f1000) libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f28ca870000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f28ca462000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f28ca25e000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000) /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000) libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f28c99fa000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f28c97f6000)

But those are already installed libraries, right? Andrei
Nov 26 2013
parent reply Martin Krejcirik <mk-junk i-line.cz> writes:
On 26.11.2013 20:02, Andrei Alexandrescu wrote:
 (0x00007f28ca047000)
     libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
     libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
     libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
     libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)

But those are already installed libraries, right?

The problem (in my case) is, that you also need to install the -dev packages of these libraries just to compile Phobos (even if you don't use curl in your programs at all). Maybe we just need something like --without-curl for Phobos makefile. -- mk
Nov 27 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 11:26, Martin Krejcirik wrote:

 The problem (in my case) is, that you also need to install the -dev
 packages of these libraries just to compile Phobos (even if you don't
 use curl in your programs at all).

 Maybe we just need something like --without-curl for Phobos makefile.

That's only needed when compiling as a shared library. -- /Jacob Carlborg
Nov 27 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/27/13 2:26 AM, Martin Krejcirik wrote:
 On 26.11.2013 20:02, Andrei Alexandrescu wrote:
 (0x00007f28ca047000)
      libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
      libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
      libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
      libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)

But those are already installed libraries, right?

The problem (in my case) is, that you also need to install the -dev packages of these libraries just to compile Phobos (even if you don't use curl in your programs at all). Maybe we just need something like --without-curl for Phobos makefile.

We could add that. I'll note that building Phobos is something for the experts only. Andrei
Nov 27 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 6:46 AM, Wyatt wrote:
 On Tuesday, 26 November 2013 at 02:22:50 UTC, Jordi Sayol wrote:
 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
     linux-vdso.so.1 (0x00007fff36519000)

Evidently, Debian builds curl with --enable-everything-under-the-sun. Here's mine, with just ssl (via OpenSSL) support: wyatt Yue ~ $ ldd /usr/lib/libcurl.so linux-vdso.so.1 (0x00007fff7f7ff000) libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007ff835865000) libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007ff83547c000) libz.so.1 => /lib64/libz.so.1 (0x00007ff835265000) librt.so.1 => /lib64/librt.so.1 (0x00007ff83505c000) libc.so.6 => /lib64/libc.so.6 (0x00007ff834cb2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007ff834aad000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff834890000) /lib64/ld-linux-x86-64.so.2 (0x00007ff835d73000) I'll be generous and allow that zlib might somehow be a sticking point, but there's really nothing unexpected here, IMO.

Great. All this is fine - whatever libcurl is installed will be used. Right? Andrei
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 5:51 PM, Martin Krejcirik wrote:
 On 25.11.2013 8:38, Jordi Sayol wrote:
 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux

I'm for removing as I always compile Phobos without curl anyway (I don't want to install all dependencies).

So then why are you bothered? Are you willing to break a lot of people's code because you don't need this feature? Andrei
Nov 26 2013
parent Martin Krejcirik <mk-junk i-line.cz> writes:
On 26.11.2013 20:01, Andrei Alexandrescu wrote:
 I'm for removing as I always compile Phobos without curl anyway (I don't
 want to install all dependencies).

So then why are you bothered? Are you willing to break a lot of people's code because you don't need this feature?

I'm not that bothered really, not a major point for me. Just seems more flexible having the curl wrapper as an independent library. Or some system for enabling/disabling components (assuming other wrappers will be added in the future). -- mk
Nov 27 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Jonathan M Davis:

 At minimum, I think that we should avoid putting any
 more 3rd party dependencies in Phobos in the future.

I am not expert/qualified enough to discuss about all this, but having some "batteries included" in D is nice. I am using curl in D. I'd even like in the standard D distribution some mean to produce a little graphics on the screen, shared on all systems D compiles on (like the simplegraphics.d module). Bye, bearophile
Nov 25 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 01:36, Jonathan M Davis ha escrit:
[...]
 
 At this point, I'm inclined to argue that Phobos should not depend on any 
 libraries other than the system libraries for each OS so that we can avoid 
 further dependencies that may not be available - especially when it comes to 
 Windows, since Windows comes with no 3rd party development libraries at all. 
 So, I would very much be against std.net.curl's inclusion at this point and 
 would argue that it should be in a library separate from Phobos and that if we 
 want similar functionality in Phobos, we need to implement it without relying 
 on anything other than system libraries. And from previous discussions on the 
 topic, I believe that Walter and several other dmd/Phobos devs agree with 
 that.
 
 So, I'm all for removing std.net.curl, but if we do that, we'd obviously need 
 to deprecate it first rather than simply removing it, and we'd need to have a 
 separate library with std.net.curl in it (preferably one which could be pulled 
 in via dub) which people could switch to using instead. Whether that library 
 would be maintained by us, by Jonas (since he created it), or someone else, I 
 don't know, but ideally, it wouldn't be in Phobos any longer. At minimum, I 
 think that we should avoid putting any more 3rd party dependencies in Phobos 
 in the future.
 
 - Jonathan M Davis
 

+1 -- Jordi Sayol
Nov 25 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu 
wrote:
 (far as I understand, the likes of php and python are doing 
 fine with curl, but maybe I'm wrong).

If you have a look at Python download page (http://www.python.org/download/) they don't provide binaries for Linux, only source tarball. And Windows installer includes curl AFAIK.
Nov 25 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 03:11, Andrei Alexandrescu ha escrit:
 On 11/25/13 5:51 PM, Martin Krejcirik wrote:
 On 25.11.2013 8:38, Jordi Sayol wrote:
 As Jonathan M Davis said: --- Several of the main devs (including
 Walter) have stated that having std.net.curl on Phobos was a mistake
 given all of the problems that we've had with it on Windows and Linux

I'm for removing as I always compile Phobos without curl anyway (I don't want to install all dependencies).

Does curl in turn depend on many other libraries?

On Debian testing (64-bit): $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0 linux-vdso.so.1 (0x00007fff36519000) libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000) librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000) libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000) libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f28cc724000) libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f28cc332000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f28cc0f2000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f28cbee3000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f28cbc92000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000) libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007f28cb1f1000) libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f28ca870000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f28ca462000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f28ca25e000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000) /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000) libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f28c99fa000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f28c97f6000) -- Jordi Sayol
Nov 25 2013
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
I don't use curl (at this point), I use D on Debian and Windows. 
I have had no issues with curl dependencies.

Reading the issues at hand it looks they all come from people 
actually trying to use curl. It sounds like this suggest isn't a 
solution but a means to clean ones hands of the problem.

I don't think I understand having to build dmd just because you 
have a different distribution, Phobos maybe but dmd... Even for 
Phobos I'm skeptical but am not going to install a different 
distro to experiment myself.

If it is moved to a third party library, the problems will still 
remain. If the Phobos team is expected to support it, then the 
burden hasn't shifted either.
Nov 25 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 07:53, H. S. Teoh wrote:
 Having a hello world program be uncompilable just because libcurl isn't
available, is a
 very bad first impression ("why does this stupid language need libcurl
 just to compile a hello world program?!").

Isn't curl only needed if std.net.curl is actually used? I have never used std.net.curl and I never had any problems with it tough. Although I use Mac OS X where curl is pre-installed. -- /Jacob Carlborg
Nov 25 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/25/13 10:53 PM, H. S. Teoh wrote:
 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]
 I don't think I understand having to build dmd just because you have
 a different distribution, Phobos maybe but dmd... Even for Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.

If you have used enough Linux distros (or even different versions of the same distro!), you will realize that all it takes is for a single shared library on the system to be a different version, missing, or just the same version built with different compiler flags, and the executable will not work. Not to mention that filesystem layout can be different across distros (and versions of the same distro), which will break things. Basically, to guarantee a program runs on distro X, the only way is it has to be built from source. If you're lucky, somebody else has already done that, and you can just download the binary. (Or better yet, somebody packaged it for your distro, then you can just install it via your system's package system.) But if not, you'll just have to do it yourself.

Do we with our std.net.curl make this matter any worse?
 If it is moved to a third party library, the problems will still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.

This I agree with. However, it misses the point. The point is that the core D toolchain (dmd/phobos/druntime) would have one less dependency, which is generally a good thing, because you want to make it as easy as possible for people to get a working compiler up and running.

But my understanding is that the dependency is conditional, i.e. you have no dependency on libcurl unless you attempt to use std.net.curl. Which strikes me as entirely reasonable. No?
 Requiring
 this library and that library and that other tool, just to get a bare
 bones D toolchain working, is a good way to turn people off D.

Bare bones == no std.net.curl. No?
 If a bare
 bones D compiler is actually working, then they'd have more incentive to
 actually put in more effort to make the frills work too. Having a hello
 world program be uncompilable just because libcurl isn't available, is a
 very bad first impression ("why does this stupid language need libcurl
 just to compile a hello world program?!").

My understanding is that a missing or broken libcurl does not break hello world. Is that correct? As a note regarding my many messages in this thread: I decided to get to the bottom of this given that I've asked "what is exactly the problem with libcurl" and all I got was unclear answers. Thanks, Andrei
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:14 AM, Dicebot wrote:
 On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu wrote:
 But my understanding is that the dependency is conditional, i.e. you
 have no dependency on libcurl unless you attempt to use std.net.curl.
 Which strikes me as entirely reasonable. No?

There is no such thing as _The_ curl library. Different distros may have versions with incompatible symbol versioning installed - this is the very issue with Fedora which was actively discussed in bugzilla.

But can't we say "to use std.net.curl with dmd 2.xxx you must have libcurl 7.10 or higher"? Andrei
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:57 AM, Dicebot wrote:
 On Tuesday, 26 November 2013 at 19:29:02 UTC, Andrei Alexandrescu wrote:
 There is no such thing as _The_ curl library. Different distros may have
 versions with incompatible symbol versioning installed - this is the
 very issue with Fedora which was actively discussed in bugzilla.

But can't we say "to use std.net.curl with dmd 2.xxx you must have libcurl 7.10 or higher"? Andrei

It is not that simple. Well, I remember you taking part in https://d.puremagic.com/issues/show_bug.cgi?id=10710 so you should be aware of it. It is not about version number of curl but about transitive dependency - openssl vs gnutls. We can say "you must have this version of libcurl built with this flags" but it is not much different from saying "use Debian-likes or get out".

What flags are we requiring for use with std.net.curl? Are those depending on what specific calls are being used (e.g. using https etc)? We could document that stuff. All in all it doesn't seem unreasonable to me to require certain specifics for using std.net.curl. Andrei
Nov 26 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 21:04, Andrei Alexandrescu wrote:

 What flags are we requiring for use with std.net.curl? Are those
 depending on what specific calls are being used (e.g. using https etc)?
 We could document that stuff. All in all it doesn't seem unreasonable to
 me to require certain specifics for using std.net.curl.

Preferably std.net.curl should work with any variant of libcurl. Regardless if it's using OpenSSL or GNU TLS. -- /Jacob Carlborg
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 3:03 PM, deadalnix wrote:
 In some cases, it can prevent the whole stuff to link.

This is very vague. Are there cases in which something NOT using std.net.curl fails to link?
 Otherwise, hello world should simply work.

I guess that's where we want it. My conclusion after this whole discussion: in the short term we need to have both Fedora and Debian builds. Then we'll move from there. Andrei
Nov 26 2013
parent Martin Nowak <code dawg.eu> writes:
On 11/27/2013 12:07 AM, Andrei Alexandrescu wrote:
 My conclusion after this whole discussion: in the short term we need to
 have both Fedora and Debian builds. Then we'll move from there.

Right, different distributions => different shared libraries.
Nov 29 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
[...]
 I don't think I understand having to build dmd just because you have
 a different distribution, Phobos maybe but dmd... Even for Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.

If you have used enough Linux distros (or even different versions of the same distro!), you will realize that all it takes is for a single shared library on the system to be a different version, missing, or just the same version built with different compiler flags, and the executable will not work. Not to mention that filesystem layout can be different across distros (and versions of the same distro), which will break things. Basically, to guarantee a program runs on distro X, the only way is it has to be built from source. If you're lucky, somebody else has already done that, and you can just download the binary. (Or better yet, somebody packaged it for your distro, then you can just install it via your system's package system.) But if not, you'll just have to do it yourself. Copying an executable from one Linux box to another is generally a bad idea. If it runs at all, there's a chance that it is actually subtly malfunctioning. I've seen heisenbugs that have no representation in source code, but show up when you run an executable in an environment that's close to, but not quite, the environment it was built on. Rebuilding from source (often with exactly the same commands, same compile flags, etc.) makes the problems go away.
 If it is moved to a third party library, the problems will still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.

This I agree with. However, it misses the point. The point is that the core D toolchain (dmd/phobos/druntime) would have one less dependency, which is generally a good thing, because you want to make it as easy as possible for people to get a working compiler up and running. Requiring this library and that library and that other tool, just to get a bare bones D toolchain working, is a good way to turn people off D. If a bare bones D compiler is actually working, then they'd have more incentive to actually put in more effort to make the frills work too. Having a hello world program be uncompilable just because libcurl isn't available, is a very bad first impression ("why does this stupid language need libcurl just to compile a hello world program?!"). T -- EMACS = Extremely Massive And Cumbersome System
Nov 25 2013
prev sibling next sibling parent "Tavi Cacina" <octavian.cacina outlook.com> writes:
On Tuesday, 26 November 2013 at 00:36:22 UTC, Adam D. Ruppe wrote:
 On Windows, the problem is simply that curl isn't packaged with 
 dmd

I see the required DLLs for win32 are in dmd2\windows\bin (libcurl.dll, libeay32.dll, libssl32.dll) and in dmd2\Windows the needed stuff for SSL (certificate bundle). At least, the docs should be improved so that the user knows he has to deliver these files with his EXE and has to enable the certificate bundle.
Nov 25 2013
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 26 November 2013 at 02:22:50 UTC, Jordi Sayol wrote:
 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
 	linux-vdso.so.1 (0x00007fff36519000)

Evidently, Debian builds curl with --enable-everything-under-the-sun. Here's mine, with just ssl (via OpenSSL) support: wyatt Yue ~ $ ldd /usr/lib/libcurl.so linux-vdso.so.1 (0x00007fff7f7ff000) libssl.so.1.0.0 => /usr/lib64/libssl.so.1.0.0 (0x00007ff835865000) libcrypto.so.1.0.0 => /usr/lib64/libcrypto.so.1.0.0 (0x00007ff83547c000) libz.so.1 => /lib64/libz.so.1 (0x00007ff835265000) librt.so.1 => /lib64/librt.so.1 (0x00007ff83505c000) libc.so.6 => /lib64/libc.so.6 (0x00007ff834cb2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007ff834aad000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff834890000) /lib64/ld-linux-x86-64.so.2 (0x00007ff835d73000) I'll be generous and allow that zlib might somehow be a sticking point, but there's really nothing unexpected here, IMO. -Wyatt
Nov 26 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 06:54:51 UTC, H. S. Teoh wrote:
 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]
 I don't think I understand having to build dmd just because 
 you have
 a different distribution, Phobos maybe but dmd... Even for 
 Phobos
 I'm skeptical but am not going to install a different distro to
 experiment myself.

If you have used enough Linux distros (or even different versions of the same distro!), you will realize that all it takes is for a single shared library on the system to be a different version, missing, or just the same version built with different compiler flags, and the executable will not work.

Yes, the most common being libc. Usually coming from compiling against a newer version and trying to use it with an older version (but not all projects are as disciplined as libc).
 Not to mention that filesystem layout can be different
 across distros (and versions of the same distro), which will 
 break
 things.

It can, but that isn't relevant to the discussion of linking we have here.
 Basically, to guarantee a program runs on distro X, the only 
 way is it
 has to be built from source. If you're lucky, somebody else has 
 already
 done that, and you can just download the binary. (Or better yet,
 somebody packaged it for your distro, then you can just install 
 it via
 your system's package system.) But if not, you'll just have to 
 do it
 yourself.

All you've said is that we are distributing DMD wrong and should be building it on every platform, I still don't know what building DMD has to do with libcurl though (DMD doesn't use it).
 This I agree with. However, it misses the point. The point is 
 that the
 core D toolchain (dmd/phobos/druntime) would have one less 
 dependency,
 which is generally a good thing, because you want to make it as 
 easy as
 possible for people to get a working compiler up and running.

I didn't miss this point, I started with explaining that I've never thought about curl before (other than wondering if I had a problem which I needed to solve with it). Though, it seems libcurl is installed due to other dependencies. So I'm less likely to see this issue. I guess I should mention I don't use the Windows installer, just extract the zip. So doubt libcurl is on any of the Windows machines I use.
Nov 26 2013
prev sibling next sibling parent reply "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.

+1 It should have been removed from Phobos long ago.
Nov 26 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 7:48 AM, Dejan Lekic wrote:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

+1 It should have been removed from Phobos long ago.

The problem is that "long ago" adding libcurl seemed like a perfectly reasonable option, particularly considering that we didn't (and we don't) have anything to replace it. Andrei
Nov 26 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-11-26 18:58, Iain Buclaw wrote:

 Especially when you look at the "competitors"

 http://curl.haxx.se/libcurl/competitors.html


 The checkboxes that would need to be ticked for me:

 1. Does it have any third party dependencies?  Preferably no.
 2. Can it be built into libphobos easily - see: zlib in phobos, or
 sqllite3.amalgamation.c which I thought *was* in phobos at some
 point... maybe not?

What's nice about libcurl, if we're not building or own solution, is that is available on basically all platforms and comes pre-installed on many. -- /Jacob Carlborg
Nov 26 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 1:33 PM, Jonas Drewsen wrote:
 On Tuesday, 26 November 2013 at 21:23:05 UTC, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei Alexandrescu wrote:
 The problem is that "long ago" adding libcurl seemed like a perfectly
 reasonable option, particularly considering that we didn't (and we
 don't) have anything to replace it.

Meh, doing the http and smtp stuff isn't rocket science. I did a little http library in ~300 lines on my github that some D programmers who don't want curl use.

Yes - basic http, smtp is easy. But have a look at the libcurl API docs and think about how many lines it would take to implement that in D.

It seems it's one of those cases in which from proof of concept to bells & whistles there's a long way. Andrei
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 2:04 PM, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 21:33:50 UTC, Jonas Drewsen wrote:
 But have a look at the libcurl API docs and think about how many lines
 it would take to implement that in D.

Meh, I don't think I've ever needed any of it. Set header and request data is enough to do anything on http.

That's quite a claim to make. I know very little on the topic, but from what I see at http://curl.haxx.se/changes.html it seems to me like a hard working professional is responding to real bug reports and feature requests from the field (as opposed to just toying with some sophisticated useless artifact). Andrei
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 3:40 PM, Adam D. Ruppe wrote:
 Again though, my position is that curl is ok and I use it myself, but
 let's not assume we need the entire kitchen sink that curl offers,
 especially given the fact that std.net.curl barely scratches it anyway.
 If someone does need one of curl's less common features, it is still
 there for them, but for the 99% we can do with the std.net.curl api, I'm
 sure we could make that happen in ~1000 lines of curl replacement.

Care to make it interesting? I may have some funds for this. But we're looking for no less than a glorious 100% replacement. Andrei
Nov 26 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 5:05 PM, Adam D. Ruppe wrote:
 On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei Alexandrescu wrote:
 Care to make it interesting? I may have some funds for this. But we're
 looking for no less than a glorious 100% replacement.

Meh, my first choice is still to just bundle curl with phobos, like we do with zlib. It seems silly to me to want a glorious 100% replacement of an *open source* library. Regardless, I probably wouldn't be the ideal choice for changing the implementation because I barely use the interface: I wrote my own http modules, including one that uses curl and one that doesn't, before std.net.curl came around. The phobos thing had no compelling benefit to me, so I never switched most my code. Thus, I'm not very familiar with std.net.curl's strengths and weaknesses and would likely break it somehow while changing the implementation. ...unless doing a new interface is on the table too. Then, we can leave std.net.curl exactly how it is, so people who use it don't have broken code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are phased in for people who want them. I could get behind that.

A new interface is also on the table, but that brings the additional burden of defining your own design. std.net.curl has already been approved. Andrei
Nov 26 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-27 05:09, Adam D. Ruppe wrote:

 2) Some kind of SSL sockets, including adding ssl to an already
 open socket (needed for STARTTLS on smtp at least). I did a
 partial implementation as a subclass of Phobos' Socket with
 openssl. Should also do a Windows impl and probably a gnutls as
 well.

Doesn't GnuTLS provide an OpenSSL API? -- /Jacob Carlborg
Nov 26 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 7:48 AM, Dejan Lekic wrote:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

+1 It should have been removed from Phobos long ago.

Why? Andrei
Nov 26 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/26/13 11:13 AM, Iain Buclaw wrote:
 Remembering to include '-lcurl' when importing std.net.curl is perhaps
 one urk.  But this will be a no longer a problem if libphobos is built
 shared.

Interesting. Can we supply a pragma that automates the flag thingie? Andrei
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.

As others said, Deimos isn't really the correct place for it since it is a wrapper. I'm fine with it being removed but only after dub has become the official package manager and there is a replacement for it on dub that is easy for users to switch to.
Nov 26 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 16:39, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 On 11/26/13 7:48 AM, Dejan Lekic wrote:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

+1 It should have been removed from Phobos long ago.

The problem is that "long ago" adding libcurl seemed like a perfectly reasonable option, particularly considering that we didn't (and we don't) have anything to replace it. Andrei

Especially when you look at the "competitors" http://curl.haxx.se/libcurl/competitors.html The checkboxes that would need to be ticked for me: 1. Does it have any third party dependencies? Preferably no. 2. Can it be built into libphobos easily - see: zlib in phobos, or sqllite3.amalgamation.c which I thought *was* in phobos at some point... maybe not?
Nov 26 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 19:07, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 On 11/26/13 7:48 AM, Dejan Lekic wrote:
 On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that having
 std.net.curl on Phobos was a mistake given all of the problems that
 we've had with it on Windows and Linux as well, and at least some of
 them have expressed a desire for it to be removed. I expect that
 there's a good chance that it can and will be removed from Phobos if
 brought up for discussion.

 Certainly, I think that it's clear that we will not add any further
 external dependencies like curl, because curl has proven to be a big
 problem. It's very desirable to have bindings and wrappers for C
 libraries - but putting them in the standard library when it's not
 guaranteed that the appropriate library is on the system by default
 has proven to be too problematic, so they should be left to external
 projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make building
 script on Deimos in order to compile all projects included on it,
 generating "libdeimos.a" and "libdeimos.so", documentation, etc.

+1 It should have been removed from Phobos long ago.

Why? Andrei

The problem I think is a distribution one. Not a compiler/library problem. Remembering to include '-lcurl' when importing std.net.curl is perhaps one urk. But this will be a no longer a problem if libphobos is built shared.
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:00:35 UTC, Andrei Alexandrescu 
wrote:
 On 11/25/13 4:36 PM, Jonathan M Davis wrote:
 The number one problem on Windows is the fact that libcurl 
 does not come with
 Windows and that you have to get a version of it build which 
 is compatible
 with dmd has proven to be a huge hurdle for Windows developers.

Why is "compatible with dmd" an issue? Do we make it any harder for libcurl than other clients? Could you please explain?

To use 32-bit curl you need to generate an OMF import library from the libcurl DLL. It's not terribly hard. I did it originally for when I added downloading of libcurl to the Windows installer. It's not at all obvious to users that they need to distribute libcurl.dll et al when they distribute their application but that's a documentation issue. If I remember correctly Walter didn't want to distribute in the dmd zip anything that wasn't boost licensed which is why curl on Windows comes as a separate download. zlib is a notable exception but I seem to recall Walter regretted including zlib. I can't remember off the top of my head if libcurl's license allows it to be statically linked with a boost licensed library but that would certainly simplify things for end users (but complicate the building of phobos a bit).
Nov 26 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu 
wrote:
 But my understanding is that the dependency is conditional, 
 i.e. you have no dependency on libcurl unless you attempt to 
 use std.net.curl. Which strikes me as entirely reasonable. No?

There is no such thing as _The_ curl library. Different distros may have versions with incompatible symbol versioning installed - this is the very issue with Fedora which was actively discussed in bugzilla. So yes, dependency is conditional but on some distros it doesn't work _at all_, even if libcurl is installed. There are only 2 possible ways out of it - distributing only source tarballs or creating distro-specific distribution archives / packages. Nothing else is guaranteed to work. And latter is rarely a good idea.
Nov 26 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On 26 November 2013 19:21, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:
 On 11/26/13 11:13 AM, Iain Buclaw wrote:
 Remembering to include '-lcurl' when importing std.net.curl is perhaps
 one urk.  But this will be a no longer a problem if libphobos is built
 shared.

Interesting. Can we supply a pragma that automates the flag thingie?

If pragma(lib) worked for me. (not talking dmd here :-)
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:23:54 UTC, Dmitry Olshansky 
wrote:
 26-Nov-2013 23:00, Andrei Alexandrescu пишет:
 There may be other problems, but I think that those are the 
 two bigs
 ones, and
 they've come up several times (particularly the issues with 
 libcurl and
 Windows).

I've asked the question "what exactly is the problem" several times and you came closest to answering it, but a bunch of stuff is still unclear to me.

I thought I'd chime in on this. I find that if we actually worked on getting curl integrated better it may just as well be an all right network client lib. For the moment it looks as if half people are pissed off because of the work they would need to do on it (workaround, shipping, maintenance, etc.), and the other half (using it) are having usability problems on all fronts. Now to my limited experience with it. When I work with it on Linux I have to always jump through hoops because of: https://d.puremagic.com/issues/show_bug.cgi?id=7044

This is a constant annoyance of mine using std.net.curl on Linux.
 Now if that doesn't set off the alarms. Win32/64 curl libraries 
 are NOT shipped with dmd zip or whatever. The better thing is 
 that last time I checked there is no proper which means that 
 for Win32 one had to get the rare OMF object file format (a 
 custom compilation no less).

Nah, it was as simple as generating an import library from the DLL. There's no need to build libcurl yourself to do that. The installer automatically downloads one I prepared a long time ago already. brocolis recently prepared an updated libcurl zip with the OMF import library as well as 64-bit support. It also uses WinSSL instead of OpenSSL so using it is even easier than it was before. The installer has already been updated for it: https://github.com/D-Programming-Language/installer/pull/27 With that libcurl should just work on Windows for both 32 and 64 bits. Linux problems are basically all that remains of the libcurl issues.
 You may or may not find them at the VARY BOTTOM of downloads 
 page, and figure it out how to fit them, what flags to use etc. 
 Last but not least it's Win32 only. No links for Win64.

The installer handles downloading it. End users don't need to find it themselves unless they download the dmd zip file (I still don't understand why anyone does that). brocolis wrote up a guide for how he made prepared the updated curl zip and put it up on the wiki. End users shouldn't need to do that though. It's just documented for the phobos developers. http://wiki.dlang.org/Curl_on_Windows
Nov 26 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 17:58:46 UTC, Iain Buclaw wrote:
 sqllite3.amalgamation.c which I thought *was* in phobos at some
 point... maybe not?

I had done work in that regard, but it was decided that it shouldn't be added so Phobos didn't ship with old versions that had known security bugs.
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei Alexandrescu 
wrote:
 On 11/26/13 11:13 AM, Brad Anderson wrote:
 To use 32-bit curl you need to generate an OMF import library 
 from the
 libcurl DLL. It's not terribly hard. I did it originally for 
 when I
 added downloading of libcurl to the Windows installer.

When is the OMF library needed? 1. While building dmd 2. While building druntime 3. While building phobos 4. While building client code that does NOT use std.net.curl 5. While building client code that DOES use std.net curl

5. std.net.curl actually existed for several releases of dmd before I went to use it one day and got linker errors and decided to fix it with the installer change.
 It's not at all
 obvious to users that they need to distribute libcurl.dll et 
 al when
 they distribute their application but that's a documentation 
 issue.

Or they could kick the can down the road by requiring their customers to have it.

The MSVCRT.dll hell of yore has shown this is a terrible option. Windows applications almost universally ship with everything needed to run. There is no package manager on Windows to handle dependencies like there is on Linux. You always include the DLLs or installers in your application (or instruct the user to download themselves if licensing forbids redistribution but this is avoided as much as possible).
 If I remember correctly Walter didn't want to distribute in 
 the dmd zip
 anything that wasn't boost licensed which is why curl on 
 Windows comes
 as a separate download. zlib is a notable exception but I seem 
 to recall
 Walter regretted including zlib.

On the other hand we've hardly heard one peep from anyone in literally years that their zip code has trouble building. We can ask the libcurl author if it comes down to redistributing a libcurl binary. But we want to make sure this is the solution of choice.

Walter actually already did that and the author of libcurl said redistributing it was fine. Here's an old thread discussing the licensing: http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org
Nov 26 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 19:29:02 UTC, Andrei Alexandrescu 
wrote:
 There is no such thing as _The_ curl library. Different 
 distros may have
 versions with incompatible symbol versioning installed - this 
 is the
 very issue with Fedora which was actively discussed in 
 bugzilla.

But can't we say "to use std.net.curl with dmd 2.xxx you must have libcurl 7.10 or higher"? Andrei

It is not that simple. Well, I remember you taking part in https://d.puremagic.com/issues/show_bug.cgi?id=10710 so you should be aware of it. It is not about version number of curl but about transitive dependency - openssl vs gnutls. We can say "you must have this version of libcurl built with this flags" but it is not much different from saying "use Debian-likes or get out".
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 19:43:02 UTC, Andrei Alexandrescu 
wrote:
 This is pretty awesome. Is  brocolis around on this forum?

 Andrei

I'm not sure. He goes by dnewbie/dnwbie on IRC.
Nov 26 2013
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 26 November 2013 at 20:02:29 UTC, Andrei Alexandrescu 
wrote:
 On 11/26/13 11:56 AM, Brad Anderson wrote:
 On Tuesday, 26 November 2013 at 19:27:07 UTC, Andrei 
 Alexandrescu wrote:
 On 11/26/13 11:13 AM, Brad Anderson wrote:
 To use 32-bit curl you need to generate an OMF import 
 library from the
 libcurl DLL. It's not terribly hard. I did it originally for 
 when I
 added downloading of libcurl to the Windows installer.

When is the OMF library needed? 1. While building dmd 2. While building druntime 3. While building phobos 4. While building client code that does NOT use std.net.curl 5. While building client code that DOES use std.net curl

5. std.net.curl actually existed for several releases of dmd before I went to use it one day and got linker errors and decided to fix it with the installer change.

Awesome. So can we assume https://github.com/D-Programming-Language/installer/pull/27 takes care of Windows for good?

It still needs one of the guys that can upload to the downloads page to upload the new binary but yeah, it's more or less ready for all scenarios now.
 Thanks for your work!!! I'll make sure I'll get a word of 
 thanks to  broccolis as well.

Will do.
 We can ask the libcurl author if it comes down to 
 redistributing a
 libcurl binary. But we want to make sure this is the solution 
 of choice.

Walter actually already did that and the author of libcurl said redistributing it was fine. Here's an old thread discussing the licensing: http://forum.dlang.org/post/wgnsetqkzbzgulccphvu forum.dlang.org

Terrific, thanks. Andrei

Nov 26 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu 
wrote:
 As many have mentioned before. Everything (in this case 
 Phobos) should
 be built on the same platform as it is shipping. So we needed 
 specific
 releases for each Linux distribution we want to support.

OK, thanks. That seems like something approachable from our end. Is this a common approach among other language distributions, e.g. python, ruby, go, rust etc. etc? What is a list of platforms we need to support?

I have already linked Python example - they provide only source tarballs for Linux, no binary distribution. It is assumed that Linux way of getting stuff is either using native package manager or compiling manually. One can try to prepare archives for every distro separately but it hardly scales. First, you need to decide how many you would like to support (few? dozens? hundreds?) Second, those will always be inferior to packages supplied from native repository (assuming active maintainer) - it is a pretty time-consuming job to do this right and no build czar can do it for dozens of platforms at once. I really think providing just source + single additional .deb package as an example is the best way to go.
Nov 26 2013
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 26 November 2013 at 19:13:40 UTC, Iain Buclaw wrote:
 Remembering to include '-lcurl' when importing std.net.curl is 
 perhaps
 one urk.  But this will be a no longer a problem if libphobos 
 is built
 shared.

Isn't that like -lm for C programs? Doesn't seem big to me.
Nov 26 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 26 November 2013 at 19:40:10 UTC, Andrei Alexandrescu 
wrote:

 [...] and presumably quite a few people who do use std.net.curl 
 without troubles and don't know we plan to pull the plug on it. 
 [...]

That's me! It works for me and my company.
Nov 26 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu 
wrote:
 On 11/26/13 12:21 PM, Dicebot wrote:
 I really think providing just source + single additional .deb 
 package as
 an example is the best way to go.

Well we kind of do that already. No?

The issue is (mainly) with people expecting the binaries from the official .zip file to work on all Linux platforms. David
Nov 26 2013
prev sibling next sibling parent "Parke" <parke.nexus gmail.com> writes:
 On 11/26/13 12:06 PM, Jacob Carlborg wrote:
 As many have mentioned before. Everything (in this case 
 Phobos) should
 be built on the same platform as it is shipping. So we needed 
 specific
 releases for each Linux distribution we want to support.


On Tuesday, 26 November 2013 at 20:10:19 UTC, Andrei Alexandrescu wrote:
 OK, thanks. That seems like something approachable from our 
 end. Is this a common approach among other language 
 distributions, e.g. python, ruby, go, rust etc. etc?

(Greetings all. I just started using D yesterday.) Languages that are entirely open source and popular just publish their source code, and the Linux distributions compile, package and distribute those languages. For example, if you look at Debian's Python3 package, you can see who the package maintainers are (right column, part way down): http://packages.debian.org/wheezy/python3 The Debian Python3 package maintainers may (or may not) work on Python itself. I believe that parts of DMD are closed source, which may create some level of barrier (both technological and ideological) to distributing DMD (and its libraries) in the standard Linux fashion.
Nov 26 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Tuesday, 26 November 2013 at 21:12:23 UTC, Andrea Fontana 
wrote:
 On Tuesday, 26 November 2013 at 19:40:10 UTC, Andrei 
 Alexandrescu wrote:

 [...] and presumably quite a few people who do use 
 std.net.curl without troubles and don't know we plan to pull 
 the plug on it. [...]

That's me! It works for me and my company.

It's worked fine for me as well. I also don't quite see why the same backwards compatibility requirements as for the rest of Phobos shouldn't apply here. If libcurl is deemed to be an unacceptable dependency, then it should be possible to provide an API-compatible reimplementation for at least the parts not directly exposing the libcurl interface. David
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei Alexandrescu 
wrote:
 The problem is that "long ago" adding libcurl seemed like a 
 perfectly reasonable option, particularly considering that we 
 didn't (and we don't) have anything to replace it.

Meh, doing the http and smtp stuff isn't rocket science. I did a little http library in ~300 lines on my github that some D programmers who don't want curl use. Uses openssl for https. curl isn't exactly bug free either - I found one a few weeks ago on a random stack overflow question about php! That said, I think curl is ok, we should just bundle it with phobos better.
Nov 26 2013
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
Personally I have always thought of std.net.curl as a temporary 
solution until a native implementation in D could be created. But 
I was also fully aware that people were not standing in line to 
make a native one and that we needed something else in the 
meantime. That is why I implemented it in the first place.

So until there is a proper replacement in phobos I really think 
it should stay, since a standard library really should include 
networking support out of the box IMHO.

/Jonas
Nov 26 2013
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Tuesday, 26 November 2013 at 21:23:05 UTC, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 16:39:50 UTC, Andrei 
 Alexandrescu wrote:
 The problem is that "long ago" adding libcurl seemed like a 
 perfectly reasonable option, particularly considering that we 
 didn't (and we don't) have anything to replace it.

Meh, doing the http and smtp stuff isn't rocket science. I did a little http library in ~300 lines on my github that some D programmers who don't want curl use.

Yes - basic http, smtp is easy. But have a look at the libcurl API docs and think about how many lines it would take to implement that in D.
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 21:33:50 UTC, Jonas Drewsen wrote:
 But have a look at the libcurl API docs and think about how 
 many lines it would take to implement that in D.

Meh, I don't think I've ever needed any of it. Set header and request data is enough to do anything on http.
Nov 26 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 26 November 2013 at 19:06:26 UTC, Andrei Alexandrescu 
wrote:
 On 11/25/13 10:53 PM, H. S. Teoh wrote:
 On Tue, Nov 26, 2013 at 06:44:03AM +0100, Jesse Phillips wrote:
 [...]
 I don't think I understand having to build dmd just because 
 you have
 a different distribution, Phobos maybe but dmd... Even for 
 Phobos
 I'm skeptical but am not going to install a different distro 
 to
 experiment myself.

If you have used enough Linux distros (or even different versions of the same distro!), you will realize that all it takes is for a single shared library on the system to be a different version, missing, or just the same version built with different compiler flags, and the executable will not work. Not to mention that filesystem layout can be different across distros (and versions of the same distro), which will break things. Basically, to guarantee a program runs on distro X, the only way is it has to be built from source. If you're lucky, somebody else has already done that, and you can just download the binary. (Or better yet, somebody packaged it for your distro, then you can just install it via your system's package system.) But if not, you'll just have to do it yourself.

Do we with our std.net.curl make this matter any worse?
 If it is moved to a third party library, the problems will 
 still
 remain. If the Phobos team is expected to support it, then the
 burden hasn't shifted either.

This I agree with. However, it misses the point. The point is that the core D toolchain (dmd/phobos/druntime) would have one less dependency, which is generally a good thing, because you want to make it as easy as possible for people to get a working compiler up and running.

But my understanding is that the dependency is conditional, i.e. you have no dependency on libcurl unless you attempt to use std.net.curl. Which strikes me as entirely reasonable. No?
 Requiring
 this library and that library and that other tool, just to get 
 a bare
 bones D toolchain working, is a good way to turn people off D.

Bare bones == no std.net.curl. No?
 If a bare
 bones D compiler is actually working, then they'd have more 
 incentive to
 actually put in more effort to make the frills work too. 
 Having a hello
 world program be uncompilable just because libcurl isn't 
 available, is a
 very bad first impression ("why does this stupid language need 
 libcurl
 just to compile a hello world program?!").

My understanding is that a missing or broken libcurl does not break hello world. Is that correct? As a note regarding my many messages in this thread: I decided to get to the bottom of this given that I've asked "what is exactly the problem with libcurl" and all I got was unclear answers. Thanks, Andrei

In some cases, it can prevent the whole stuff to link. Otherwise, hello world should simply work.
Nov 26 2013
prev sibling next sibling parent "nazriel" <spam dzfl.pl> writes:
On Monday, 25 November 2013 at 07:38:38 UTC, Jordi Sayol wrote:
 As Jonathan M Davis said:
 ---
 Several of the main devs (including Walter) have stated that 
 having std.net.curl on Phobos was a mistake given all of the 
 problems that we've had with it on Windows and Linux as well, 
 and at least some of them have expressed a desire for it to be 
 removed. I expect that there's a good chance that it can and 
 will be removed from Phobos if brought up for discussion.

 Certainly, I think that it's clear that we will not add any 
 further external dependencies like curl, because curl has 
 proven to be a big problem. It's very desirable to have 
 bindings and wrappers for C libraries - but putting them in the 
 standard library when it's not guaranteed that the appropriate 
 library is on the system by default has proven to be too 
 problematic, so they should be left to external projects.
 ---

 "std.net.curl" can also be moved from Phobos to Deimos.
 Deimos can be rethink, i.e. new build master can add a make 
 building script on Deimos in order to compile all projects 
 included on it, generating "libdeimos.a" and "libdeimos.so", 
 documentation, etc.

We need something to replace std.net.curl with in the first place. The problem is that if want to support for example HTTPS or FTPS we would end with dependencie (on openssl for example) anyways. Another thing is that making anything more than trivial requests, requires a bit of work - compressions, encodings, authentication, RFC (*Grins) etc - it all makes it complex task. Removing std.net.curl at this point will probably break a bit of code. About cases where people have problems with linking when using std.net.curl for http - advise them to use Vibe.d http client or Vladimir's ae.net.* stuff or Adam's Ruppe custom framework? Dunno. Merge any of those 3 projects work into Phobos? Bounty for D implementation of libCURL - like for phobos inclusion?
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 26 November 2013 at 23:05:08 UTC, Andrei Alexandrescu 
wrote:
 That's quite a claim to make.

If you've used HTTP, you'll know how true it is. Indeed, since a request consists solely of a request and a body, being able to change them by definition gives you full access! The most complex things in http are proxying and caching. curl does proxying, wich my 300 line http.d doesn't do. As far as I know, curl doesn't do caching at all. (My curl.d offers a cache functionality on top of curl, but it ignores the http headers, instead opting simply for a max age provided by the programmer, e.g. "refresh cache if the file is more than four hours old or doesn't exist, otherwise just return the cache file's contents".)
 but from what I see at http://curl.haxx.se/changes.html it 
 seems to me like a hard working professional is responding to 
 real bug reports and feature requests from the field

Yet, very little of that stuff is actually exposed through std.net.curl anyway, except perhaps through the download() and upload() functions. Phobos doesn't directly expose tftp, imap, pop, globbing, ntlm, nor most of the other things mentioned on that changelog. SSL is legitimately a bit of a pain, but curl doesn't actually implement ssl, it just uses it, with a few different backends - openssl, gnutls, and so forth. Indeed, this is one of the difficulties in using curl with D, it is compiled with openssl but the client machine uses gnutls or whatever. We'd have that same problem. Async requests are similar to ssl: it should implemented in a generic way for all I/O, then the http layer can just use that. It's probably just a matter of time before there's demand for an async file and socket facility in there anyway. Again though, my position is that curl is ok and I use it myself, but let's not assume we need the entire kitchen sink that curl offers, especially given the fact that std.net.curl barely scratches it anyway. If someone does need one of curl's less common features, it is still there for them, but for the 99% we can do with the std.net.curl api, I'm sure we could make that happen in ~1000 lines of curl replacement.
Nov 26 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Nov 26, 2013 at 04:19:31PM -0800, Andrei Alexandrescu wrote:
 On 11/26/13 3:40 PM, Adam D. Ruppe wrote:
Again though, my position is that curl is ok and I use it myself, but
let's not assume we need the entire kitchen sink that curl offers,
especially given the fact that std.net.curl barely scratches it
anyway.  If someone does need one of curl's less common features, it
is still there for them, but for the 99% we can do with the
std.net.curl api, I'm sure we could make that happen in ~1000 lines
of curl replacement.

Care to make it interesting? I may have some funds for this. But we're looking for no less than a glorious 100% replacement.

This piqued my interest. Please excuse another of my TL;DR stories / rants. Some time earlier this year, I wanted to write a simple utility to (1) login to a website with a preconfigured username/password; (2) navigate to a specific page on the site; (3) extract a specific HTML table from that page, and (4) parse and format the table into an ASCII-graphics text format. Sounds like the perfect job for a quick-n-dirty D program, right? In the old days, I'd hack up a Perl script to do this in a couple o' hours, but this time, I thought, since I have D at my disposal, let's do this the Right Way(tm). Where in the past I'd just run wget or curl (the CLI utility) to fetch the web pages, I thought it'd be much cleaner to do the web interaction directly so that I didn't have to mess around with parsing temporary cookie files and what-not just to keep track of my login session. So I scrolled through the Phobos docs, and lo and behold, there was std.net.curl sitting right there, looking all pretty and beckoning to me. I glanced over its API, and found that it was just the right fit for what I needed to do. Or so I thought... About 30 minutes into the coding, I ran into a blocker: std.net.curl didn't let me specify Content-Type of POST requests!!! The server simply refused to accept anything that wasn't explicitly stated to be x/www-urlencoded, but std.net.curl only allows text/plain or application/octet-stream! I simply couldn't login. Thinking that I must've missed something obvious, I spent the rest of the day (and probably several more hours the next day) combing through the docs to find out how to set the Content-Type to something else. Finally I resorted to reading the std.net.curl source code, only to discover, to my dismay, that there was NO way to do this unless I bypassed all the nice high-level syntactic sugar, and got my hands down and dirty with low-level libcurl API calls (or their thin wrappers thereof). At that point, I threw up my hands. The original task was far too simple for something this heavy-handed, so I backed off and said, it was nice knowing you, std.net.curl, but I think we're better off going our separate ways. TL;DR: I would *love* to see a replacement for std.net.curl that isn't so straitjacketed that I can't even do something so basic as logging on to a website. I'm sure there are very good reasons why the current API is the way it is, but the fact of the matter is that, from an end-user's POV, its limitations were a deal-breaker. So, I'm all for a D-only replacement of std.net.curl with a superior API. Getting rid of the external dependence on libcurl would be an added bonus. T -- "I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you already said that." -- User-Friendly
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 00:19:31 UTC, Andrei 
Alexandrescu wrote:
 Care to make it interesting? I may have some funds for this. 
 But we're looking for no less than a glorious 100% replacement.

Meh, my first choice is still to just bundle curl with phobos, like we do with zlib. It seems silly to me to want a glorious 100% replacement of an *open source* library. Regardless, I probably wouldn't be the ideal choice for changing the implementation because I barely use the interface: I wrote my own http modules, including one that uses curl and one that doesn't, before std.net.curl came around. The phobos thing had no compelling benefit to me, so I never switched most my code. Thus, I'm not very familiar with std.net.curl's strengths and weaknesses and would likely break it somehow while changing the implementation. ...unless doing a new interface is on the table too. Then, we can leave std.net.curl exactly how it is, so people who use it don't have broken code, while a new std.net.http, std.net.smtp, std.net.ftp, and so on are phased in for people who want them. I could get behind that.
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 00:49:57 UTC, H. S. Teoh wrote:
 Sounds like the perfect job for a quick-n-dirty D program, 
 right?

Oh yeah, my http.d and dom.d can make short work of that. I had a similar problem last year and ended up getting it to work very easily (well, that one used my curl.d, but http.d does cookies too now).
 coding, I ran into a blocker: std.net.curl didn't let me specify
 Content-Type of POST requests!!!

.....holy crap, you're not kidding. I thought addRequestHeader can do it, but that duplicates the header. Unbelievable. My curl.d is an ugly piece of garbage. Seriously, I hate it, just look at this signature: string curlAuth(string url, string data = null, string username = null, string password = null, string contentType = "application/x-www-form-urlencoded", string methodOverride = null, string[] customHeaders = null, string cookieJar = null) { but at least it gets the job done - I use it on my work projects to access all kinds of things from arbitrary website downloads (one of the work sites offers a link posting functionality similar to Facebook, it downloads the site and reads description, images, etc. to make a richer link preview) to a variety of APIs like bing search, Facebook, twitter, linkedin, youtube, Authorize.net, I've done a lot with it, and especially my helper oauth.d file for many of them, all built on top of this hideous do-it-all function. If I was doing a whole new interface, I think I'd prefer to do an object that keeps this state internally. So it'd be like: auto client = new HttpClient("host", port); client.userAgent = "whatever"; // authorization would ideally be a UFCS function or at least do some template method check, so it can be extended by things like oauth client.authorization = BasicAuth("username", "password"); client.host = "host override if you want.com"; auto response = client.post("/login", ["username":"lol", "password":"rofl"]); // or client.post("/login", "text/plain", "text"); // or client.post("/login", "text/xml", someXmlInputRange); or something like that. Url and method could be set via properties too; get, post, head, put, delete, and so on could perhaps be implemented in terms of opDispatch that sets the properties then calls .execute(); Anyway, the important thing would be if you do future changes to this object, it maintains a bit of state. Cookies and such. That could work with async too, since the response returned would be more like a handle, so you then do response.wait(); or do callbacks or whatever. But the point is it'd be nice and simple for most series of requests.
Nov 26 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 10:44:07 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 11/25/13 4:36 PM, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:
 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).

A major difference is there's only one php/python binary, usually build on the same system that uses it. Phobos, on the other hand, is generally built on the packager's computer, which isn't always binary compatible with the deployment box. On Windows, the problem is simply that curl isn't packaged with dmd, for some weird reason, meaning people have to get curl.lib separately. That's idiotic. But then again, so are a lot of the dmd Windows deficiencies.

Please let me know whether my understanding of the situation is correct: 1. If people have a working installation of libcurl on their machine, we work with it. 2. Otherwise, phobos works fine but attempting to use std.net.curl will fail. Is this correct? If not, please explain exactly why. If yes, this setup seems entirely appropriate to me.
 If we do decide to do away with libcurl, one possible solution would
 be to embed its source code within our build. That way we wouldn't
 break code that already uses it.

Yes, that would be ideal, we should just statically link curl right into the phobos build so it just works everywhere.

There are several questions associated with this. 1. Does the author of libcurl agree with such? My understanding is he would. 2. Would we need to actually build libcurl from source, or just distribute it? In the former case, it would be quite odd that we'd need to do that (and presumably we'd need to take some maintenance burden, too). 3. What if people already have some working libcurl version and would want to use that? Andrei

Point 3 is a bigger issue than it may seem at first. A reason to use shared libraries is that the modularity makes it trivial to get security updates, whereas a monolithic Phobos is not necessarily updated when a security fix for cURL becomes available. This page shows that sometimes security holes are found on a daily basis: http://curl.haxx.se/docs/security.html -- Marco
Nov 26 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, November 27, 2013 02:05:39 Adam D. Ruppe wrote:
 ...unless doing a new interface is on the table too. Then, we can
 leave std.net.curl exactly how it is, so people who use it don't
 have broken code, while a new std.net.http, std.net.smtp,
 std.net.ftp, and so on are phased in for people who want them. I
 could get behind that.

If we were going to replace std.net.curl, I would fully expect us to end up with a new API, because the current API is what is beacuse of what curl can and can't do rather than because it's the best API for what you're conceptually trying to do. The new API might end up being fairly close to the current one, but I would not consider it desirable to try and maintain a 100% compatible API if we were replacing the current implementation except in cases where the current API is the best choice even without curl underneath it. Maybe that would be most of the API, but I'd hate to see us put ourselves in a straightjacket based on how curl works when writing an API which didn't depend on curl - particularly as writing our own would give us the opportunity to make sure that we had something that made the most conceptual sense rather than necessarily having what curl has. - Jonathan M Davis
Nov 26 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 27 November 2013 at 01:40:15 UTC, Mike Parker wrote:
 This problem would be solved by going with a dynamic binding 
 for libcurl rather than a static binding. Then std.net.curl 
 could load libcurl on demand and it doesn't matter how it was 
 compiled (unless, of course, the libcurl API changes with 
 compiler flags).

In addition, even with Phobos as a shared library libcurl wouldn't necessarily be loaded if it's not actually used then. David
Nov 26 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 11:02:06 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 11/25/13 6:22 PM, Jordi Sayol wrote:
 On Debian testing (64-bit):

 $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4.3.0
 	linux-vdso.so.1 (0x00007fff36519000)
 	libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f28ccdc5000)
 	librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007f28ccbab000)
 	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f28cc982000)
 	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
(0x00007f28cc724000)
 	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
(0x00007f28cc332000)
 	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
(0x00007f28cc0f2000)
 	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
(0x00007f28cbee3000)
 	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
(0x00007f28cbc92000)
 	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f28cba79000)
 	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f28cb85d000)
 	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f28cb4b1000)
 	libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26
(0x00007f28cb1f1000)
 	libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007f28caf72000)
 	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f28cad6e000)
 	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f28caa99000)
 	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007f28ca870000)
 	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f28ca66c000)
 	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0
(0x00007f28ca462000)
 	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007f28ca25e000)
 	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f28ca047000)
 	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f28c9e2b000)
 	/lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
 	libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3 (0x00007f28c9c1a000)
 	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
(0x00007f28c99fa000)
 	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0
(0x00007f28c97f6000)

But those are already installed libraries, right? Andrei

Yes, the curl package on this Linux distribution depends on all the listed libraries and when curl is installed, they are installed with it. Often these dependencies are added on a per case basis. E.g. when the distribution adds a program using Kerberos network authentication through curl, it is enabled in the curl package by the maintainer. A notable exception is with source only distributions like Gentoo, where packages can depend on curl WITH certain options enabled. In this case curl is recompiled whenever an additional requirement occurs and ideally stays as big as necessary and as small as possible ;) -- Marco
Nov 26 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 26 Nov 2013 12:04:59 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 What flags are we requiring for use with std.net.curl? Are those 
 depending on what specific calls are being used (e.g. using https etc)? 
 We could document that stuff. All in all it doesn't seem unreasonable to 
 me to require certain specifics for using std.net.curl.
 
 Andrei

Yes, listing requirements is always good to do. But let's take a look at some hard facts first. `ldd libphobos2.so' on the linux amd64 version from the zip (for 2.064.2) reveals: linux-vdso.so.1 (0x00007fff2fd96000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f1294e66000) libcurl-gnutls.so.4 => not found libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1294c48000) libm.so.6 => /lib64/libm.so.6 (0x00007f1294951000) librt.so.1 => /lib64/librt.so.1 (0x00007f1294748000) libc.so.6 => /lib64/libc.so.6 (0x00007f12943a0000) /lib64/ld-linux-x86-64.so.2 (0x0000003989a00000) libcurl-gnutls is not found on *my* installation of Linux. You correctly made out https (SSL) as a culprit. It is not a single flag though, but up to 6(!) different SSL "backends" you can chose from. Typically it comes down to OpenSSL and GnuTLS. The Debian system that this phobos.so was compiled on obviously used a GnuTLS version that was suffixed with "-gnutls" to allow a parallel installation of a curl version with e.g. OpenSSL. On my Linux on the other hand, I can only install one version of curl at a time and no suffix is used. So: 1. curl library names differ on distributions Next, when you issue: `readelf --version-info libphobos2.so | less' you will get a dump of the .gnu.version section that contains entries like "CURL_GNUTLS_3". These correlate to the respective "versioned symbols" from libcurl.so. Versioned symbols are the regular C export names with an -suffix. The following command should reveal them: `readelf -s /usr/lib64/libcurl-gnutls.so.4 | grep GNUTLS_3' What this means, is that not only the library name phobos.so uses is linked to the gnutls version, but also each of the SSL specific functions! This makes for: 2. phobos.so uses GNUTLS version of curl functions You can rely on the fact that curl is installed on Linux and that it exposes SSL functionality, but the way phobos.so is created it is Debian and GnuTLS specific at the moment. -- Marco
Nov 26 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
Alexandrescu wrote:
 A new interface is also on the table, but that brings the 
 additional burden of defining your own design. std.net.curl has 
 already been approved.

Aye, but that'd be worth it. There's a few pieces we should get together: 1) a url module. My cgi.d has one that may be usable with a little testing: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022 about 150 lines long, including some unittest. Also has some functions that build on std.uri to make encoding multiple values easier - working with associative arrays instead of just strings. This is useful since we can then pass type-safe urls to various functions and get consistent parsing and creation. The basedOn() method in my code there is meant to do relative links in a browser fetch situation. Regardless, all this is fairly useful independent of network functions too. 2) Some kind of SSL sockets, including adding ssl to an already open socket (needed for STARTTLS on smtp at least). I did a partial implementation as a subclass of Phobos' Socket with openssl. Should also do a Windows impl and probably a gnutls as well. I think this would be ideally picked at runtime, with dlopen(). If gnutls is available, make that work. Otherwise, use openssl, and if none are there, throw upon ssl requests. On Windows, it can just use the built in ssl facilities the operating system provides. I'd probably make this a new module that extends std.socket rather than editing std/socket.d. 3) async (overlapped) I/O on those sockets. This might need std.socket to be edited to add the functionality. Then, we go into the curl replacement: 4) The http module. Offers simple functions (get(), post() that just work), a reusable client object, and advanced functions, all in sync and async versions. My http.d can be a starting point, though I have a new api in mind. I'll perhaps slap together a first draft over the next week, if my other work leaves enough time. I have to write a book chapter on Phobos soon too! lol maybe time to scratch the std.net.curl portion. 5) The smtp module. Offers the low level send functions, and a higher level MIME message builder class. I've written the latter already. https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/email.d#L27 though it uses my html to text module, which uses my dom module, which uses my characterencodings module. That could be easily pulled out though, it would just require the user to set the plain text version of an html email themselves. As you can see there too, it currently uses std.net.curl to actually send the email, since my smtp function (written but it isn't great) doesn't support the starttls command that Amazon SES uses and i needed that for a work project... My module can also read email, but I don't think that's needed in the std.net.email. 6) The ftp module. I haven't written ftp code for a long time, it would perhaps be better if someone else wrote this, though if nobody else does, I'm sure I can refresh myself well enough with a lil time. But the beauty of modules is we can divide and conquer! 7) Any other protocols we care about. Personally, I'd kinda love some ssh and scp code... but I'm afraid actually writing that is something I don't want to do, perhaps we can do a wrapper for libssh. Perhaps not appropriate for phobos due to dependency and licensing but I'd still love someone to write it and post a link here! 8) If we want to, there could be a combined module that reads a runtime url and dispatches to the appropriate protocol, like the std.net.curl.download and upload can do. But tbh I don't even care much about this. It is trivial once the other modules are written anyway. 9) Maybe my OAuth module's client code can be cleaned up and added too, as an add-on to the http functionality. Each piece here is pretty independent, except the first three, and I have enough written already for them to get started at least.
Nov 26 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Wednesday, 27 November 2013 at 04:09:53 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 27 November 2013 at 01:30:26 UTC, Andrei
 Alexandrescu wrote:
 A new interface is also on the table, but that brings the 
 additional burden of defining your own design. std.net.curl 
 has already been approved.

Aye, but that'd be worth it. There's a few pieces we should get together: 1) a url module. My cgi.d has one that may be usable with a little testing: https://github.com/adamdruppe/misc-stuff-including-D-programming-language-web-stuff/blob/master/cgi.d#L2022

Should we consider interoperability with std.path too?
 4) The http module. Offers simple functions (get(), post() that
 just work), a reusable client object, and advanced functions, 
 all
 in sync and async versions.

I worked on http protocol, and I don't think that's will be a big issue. Http client side is not so complex if you need a working subset. I think we don't need gzip compression in first release, for example.
 9) Maybe my OAuth module's client code can be cleaned up and 
 added too, as an add-on to the http functionality.

+1 for an oauth module. It would be very useful to work with third party api (facebook, google, etc ...) and to write api too! However I think it depends on http(s) module.
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu 
wrote:
 I really think providing just source + single additional .deb 
 package as
 an example is the best way to go.

Well we kind of do that already. No? Andrei

No. We provide Linux binaries in .zip and pretend they are expected to work for all distros which simply can't happen in general. Fedora bug report would not have been valid if not implication that there is a supported binary distribution for Fedora. I suggest to simply stop pretending any kind of such support exist and drop Linux binaries from .zip at all. Motivating people to create dmd packages for their native distros scales much better (and solves most/all curl issues for Linux)
Nov 27 2013
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Wednesday, 27 November 2013 at 10:26:50 UTC, Martin Krejcirik 
wrote:
 The problem (in my case) is, that you also need to install the 
 -dev
 packages of these libraries just to compile Phobos (even if you 
 don't
 use curl in your programs at all).

no real benefit. :/
 Maybe we just need something like --without-curl for Phobos 
 makefile.

If it's _really_ important that a curl binary is distributed with D, it should be built statically. But I doubt it's actually THAT important. Make it a proper dep and let the distro packagers do what they do best. The --without-curl switch is a good idea, btw. -Wyatt
Nov 27 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 07:39:17 UTC, Jacob Carlborg 
wrote:
 Doesn't GnuTLS provide an OpenSSL API?

idk. If so, that's easy. I've just seen curl compiled separately for gnutls and openssl so I assumed they must be different. Nevertheless, I really think dynamic linking is the way to go here since that can gracefully degrade when no ssl library is available at all too.
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 14:07:02 UTC, Wyatt wrote:
 Make it a proper dep and let the distro packagers do what they 
 do best.  The --without-curl switch is a good idea, btw.

Distro packagers don't have any issues even now (assuming they build stuff locally). It is all about "official" .zip
Nov 27 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 08:45:37 UTC, Andrea Fontana 
wrote:
 Should we consider interoperability with std.path too?

Generally, I think no, though it would be easy enough to get uri.path as a decoded string and pass it to those functions. But std.path is a bit different on Windows and Linux (I think; I've never actually used it), whereas urls have a slightly different and stricter interpretation (which may be treated like win or lin depending on the server).
 I worked on http protocol, and I don't think that's will be a 
 big issue. Http client side is not so complex if you need a 
 working subset. I think we don't need gzip compression in first 
 release, for example.

Yeah, http is pretty simple. Even gzip isn't really hard.
 +1 for an oauth module. It would be very useful to work with 
 third party api (facebook, google, etc ...) and to write api 
 too! However I think it depends on http(s) module.

Yup. I wasn't thinking of putting the server code in the std.net.http, though I have some in my oauth.d (isSignatureValid, isOauthRequest, etc., the functions that take a Cgi parameter) I've barely actually used it, so it is probably buggy, but it is there and wasn't horribly hard to write after getting the client working. (The biggest pain with clients btw? Buggy servers! It has gotten a little better, but when I started with this, Twitter, LinkedIn, and AWeber, the three I had to use.... all interpreted the spec slightly differently, so what worked on one wouldn't work on the other two, and, of course, the error messages are never helpful and the documentation sucks. Massive pain.)
Nov 27 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 27 November 2013 at 17:58:48 UTC, Andrei 
Alexandrescu wrote:
 On 11/27/13 2:26 AM, Martin Krejcirik wrote:
 On 26.11.2013 20:02, Andrei Alexandrescu wrote:
 (0x00007f28ca047000)
     libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2
 (0x00007f28c9e2b000)
     /lib64/ld-linux-x86-64.so.2 (0x00007f28cd28e000)
     libtasn1.so.3 => /usr/lib/x86_64-linux-gnu/libtasn1.so.3
 (0x00007f28c9c1a000)
     libp11-kit.so.0 => 
 /usr/lib/x86_64-linux-gnu/libp11-kit.so.0
 (0x00007f28c99fa000)
     libgpg-error.so.0 => 
 /lib/x86_64-linux-gnu/libgpg-error.so.0
 (0x00007f28c97f6000)

But those are already installed libraries, right?

The problem (in my case) is, that you also need to install the -dev packages of these libraries just to compile Phobos (even if you don't use curl in your programs at all). Maybe we just need something like --without-curl for Phobos makefile.

We could add that. I'll note that building Phobos is something for the experts only. Andrei

I'm not sure I would consider myself an expert but I compile dmd/druntime/phobos once a day on 3 different machines. Hah, now I type that it seems a little excessive!
Nov 27 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Nov 27, 2013 at 07:45:07PM +0100, John Colvin wrote:
 On Wednesday, 27 November 2013 at 17:58:48 UTC, Andrei Alexandrescu
 wrote:

We could add that. I'll note that building Phobos is something for
the experts only.

Andrei

I'm not sure I would consider myself an expert but I compile dmd/druntime/phobos once a day on 3 different machines. Hah, now I type that it seems a little excessive!

I don't have 3 different machines, but I find myself compiling git HEAD so often, that I've written a script that automatically pulls upstream master, pushes to my github fork, runs make clean, and rebuilds everything from scratch. Sometimes, I run that script multiple times a day. :) But yeah, if you can build the D toolchain by hand, I'd consider you as an expert. Perhaps not as expert as the phobos devs, maybe, but definitely not among the lay people. ;-) To D's credit, though, building the toolchain really *is* as simple as just running the makefiles in the source code in the right order (dmd, druntime, then phobos), perhaps with an option or two (like MODEL=64). Try doing that with, say, the gcc toolchain, and you're in for a journey of pain that will last, if you're lucky, days, or if not, weeks and maybe even months. (Unless you're just one of those geniuses that seem to have no problem whatsoever in taming the gcc / glibc build process. I'm certainly not among them. The gcc build scripts hate me, and hate me a lot, for some reason, and they just refuse to work in my environment. I flinch every time the thought of rebuilding gcc comes to mind. The dmd toolchain is, in comparison, a pleasure to build.) T -- A mathematician is a device for turning coffee into theorems. -- P. Erdos
Nov 27 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 27 November 2013 at 18:58:42 UTC, Andrei 
Alexandrescu wrote:
 Wait, you mean with binaries and all?

Yeah, I like it the way it is too since I use the binaries too, building windows programs from my linux box calling wine dmd. My scripts currently assume the zip layout and it works pretty beautifully. I hate installers and package managers, the zip rox. Though, I've said this before, but I could actually live with a separate zip per platform if: 1) the folder layouts remains the same. So if I unzip dmd-windows.zip, unzip dmd-linux.zip in the same place, it would recreate the same directory layout we have now. 2) The Windows zip actually offers a compelling benefit: include up-to-date .lib files and more complete Win32 bindings out of the box!
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 18:57:23 UTC, Jacob Carlborg 
wrote:
 On 2013-11-27 19:00, Andrei Alexandrescu wrote:

 I agree the business with distributing all binaries for all 
 platforms in
 the .zip is just weird. We should drop it for the New Year's 
 release.

I would hate to see the cross-platform zip gone. It would break DVM and probably other tools/scripts. I don't mind additional packages but please, keep the current zip as well.

DVM is already broken if it relies on binaries present in .zip We can't claim that we support things that we don't actually support
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 19:10:26 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 27 November 2013 at 18:58:42 UTC, Andrei 
 Alexandrescu wrote:
 Wait, you mean with binaries and all?

Yeah, I like it the way it is too since I use the binaries too, building windows programs from my linux box calling wine dmd. My scripts currently assume the zip layout and it works pretty beautifully.

I don't think Windows binaries will go anywhere as they do work on all supported Windows versions. Problem is with considering "Linux" a single platform while in practice each distro is more like separate operating system (despite being so similar in many things). My vision is that we can keep current layout but to make it mandatory to run `make` on target Linux system to create actual binaries in the same places they are present right now. It should work quite reliably at only added cost of actual compilation time (which is relatively small for dmd + phobos)
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 20:05:29 UTC, Jacob Carlborg 
wrote:
 On 2013-11-27 20:13, Dicebot wrote:

 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually 
 support

So we don't support the zip we release every time?

Yes, this is exactly the very reason of this thread. Don't support and have never really supported. It does support Debian-based stuff and some other distros may happen to work so there were no real fuss about it. Does not make it less broken thugh.
Nov 27 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 27 November 2013 at 20:06:52 UTC, Jacob Carlborg 
wrote:
 On 2013-11-27 20:13, Dicebot wrote:

 DVM is already broken if it relies on binaries present in .zip
 We can't claim that we support things that we don't actually 
 support

It works perfectly fine on Mac OS X and Windows. No need to break it on all platforms just because one platform is broken. I haven't had any problems on Linux either.

Distributing binaries for Mac and Windows is fine (though it is much better to keep those separate archives and patch DVM accordingly), this thread is about Linux ones. You did not have any problems with it because Debian derivatives take major percentage of users and you are likely to have one.
Nov 27 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Wednesday, 27 November 2013 at 14:16:33 UTC, Adam D. Ruppe 
wrote:
 (The biggest pain with clients btw? Buggy servers! It has 
 gotten a little better, but when I started with this, Twitter, 
 LinkedIn, and AWeber, the three I had to use.... all 
 interpreted the spec slightly differently, so what worked on 
 one wouldn't work on the other two, and, of course, the error 
 messages are  never helpful and the documentation sucks. 
 Massive pain.)

Because they started using oauth 2.0 when rfc draft changes every day. I remember the pain. Now it seems that there's a final rfc. I remember that google implemented draft #xx twitter draft #yy and so on...
Nov 27 2013
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Wednesday, 27 November 2013 at 01:32:18 UTC, Jonathan M Davis 
wrote:
 On Wednesday, November 27, 2013 02:05:39 Adam D. Ruppe wrote:
 ...unless doing a new interface is on the table too. Then, we 
 can
 leave std.net.curl exactly how it is, so people who use it 
 don't
 have broken code, while a new std.net.http, std.net.smtp,
 std.net.ftp, and so on are phased in for people who want them. 
 I
 could get behind that.

If we were going to replace std.net.curl, I would fully expect us to end up with a new API, because the current API is what is beacuse of what curl can and can't do rather than because it's the best API for what you're conceptually trying to do. The new API might end up being fairly close to the current one, but I would not consider it desirable to try and maintain a 100% compatible API if we were replacing the current implementation except in cases where the current API is the best choice even without curl underneath it. Maybe that would be most of the API, but I'd hate to see us put ourselves in a straightjacket based on how curl works when writing an API which didn't depend on curl - particularly as writing our own would give us the opportunity to make sure that we had something that made the most conceptual sense rather than necessarily having what curl has.

I definitely agree. In addition D has obtained some nice features since the definition of the std.net.curl API that I would liked have used back then. I do believe that people underestimate the work needed to make a proper http client implementation with the bells and whristles that people expect like cookie store, chuncked transfers, ssl, authentication... and handle wierd per server type bugs as well. I definitely think that a native solution is the end goal here. But if that is the case then it should be based on a proper event based system (std.event?) to support async handling (peek at how vibe has done it very nicely). But that in turn depends somewhat on allocators being well defined and implemented as I see it. -Jonas
Nov 28 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 28 November 2013 at 07:36:09 UTC, Jacob Carlborg 
wrote:
 On 2013-11-27 21:46, Dicebot wrote:

 Distributing binaries for Mac and Windows is fine (though it 
 is much
 better to keep those separate archives and patch DVM 
 accordingly), this
 thread is about Linux ones. You did not have any problems with 
 it
 because Debian derivatives take major percentage of users and 
 you are
 likely to have one.

If we're removing the zip as it currently is, it's another prof that D isn't stable as it claims to be. And I repeat, why should we break for every platform when it's only one platform that is broken.

We can leave old .zip as-is just make it explicit that it is legacy layout and only 100% expected to work on Debian-compatible distros. I really don't care how it is distributed as I am using git tag anyway - what I do care about are users having wrong expectations when they come to download page.
Nov 28 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 26/11/13 19:44, Andrei Alexandrescu ha escrit:
 On 11/25/13 4:36 PM, Adam D. Ruppe wrote:
 On Tuesday, 26 November 2013 at 00:13:57 UTC, Andrei Alexandrescu wrote:
 First I'd like to gather an understanding on why we seem to have this
 problem (far as I understand, the likes of php and python are doing
 fine with curl, but maybe I'm wrong).

A major difference is there's only one php/python binary, usually build on the same system that uses it. Phobos, on the other hand, is generally built on the packager's computer, which isn't always binary compatible with the deployment box. On Windows, the problem is simply that curl isn't packaged with dmd, for some weird reason, meaning people have to get curl.lib separately. That's idiotic. But then again, so are a lot of the dmd Windows deficiencies.

Please let me know whether my understanding of the situation is correct: 1. If people have a working installation of libcurl on their machine, we work with it. 2. Otherwise, phobos works fine but attempting to use std.net.curl will fail. Is this correct? If not, please explain exactly why. If yes, this setup seems entirely appropriate to me.

Point 2. is correct only with phobos static library but not with shared one. A simple "Hello World" program cannot be linked against shared phobos library if libcurl is not available on system, at least on Linux. It cannot run either. -- Jordi Sayol
Nov 28 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 27/11/13 13:56, Dicebot ha escrit:
 On Tuesday, 26 November 2013 at 21:04:31 UTC, Andrei Alexandrescu wrote:
 I really think providing just source + single additional .deb package as
 an example is the best way to go.

Well we kind of do that already. No? Andrei

No. We provide Linux binaries in .zip and pretend they are expected to work for all distros which simply can't happen in general. Fedora bug report would not have been valid if not implication that there is a supported binary distribution for Fedora. I suggest to simply stop pretending any kind of such support exist and drop Linux binaries from .zip at all. Motivating people to create dmd packages for their native distros scales much better (and solves most/all curl issues for Linux)

We should then generate dmd packages for every distro release because they can fail between releases too. The only thing that breaks the current dmd zip system is libcurl dependency. All other thinks properly works on all Linux distros/releases. Should we change all this well working system due to a single problem with an external library dependency? -- Jordi Sayol
Nov 28 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 28 November 2013 at 15:59:12 UTC, Jordi Sayol wrote:
 We should then generate dmd packages for every distro release 
 because they can fail between releases too.

Or just let specific distro package maintainers do their job.
 The only thing that breaks the current dmd zip system is 
 libcurl dependency. All other thinks properly works on all 
 Linux distros/releases.

Are you actually sure? _all_ stuff? _all_ distros? I doubt it.
 Should we change all this well working system due to a single 
 problem with an external library dependency?

It is not working, it just happens to not work for a very tiny user base which is more likely to workaround the issue by building from source other than reporting it. Really, I don't care what the solution is. You can leave current archive as-is and close that Fedora issue as invalid, it is not that important. Just stop pretending it is supported for all platforms and that it can _possibly_ be supported for all platforms.
Nov 28 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 28 November 2013 at 11:37:45 UTC, Jonas Drewsen 
wrote:
 I definitely agree. In addition D has obtained some nice 
 features since the definition of the std.net.curl API that I 
 would liked have used back then.

What are you thinking of there? I think plain old classes and delegates are a good fit for http stuff.
Nov 28 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 28/11/13 17:12, Dicebot ha escrit:
 On Thursday, 28 November 2013 at 15:59:12 UTC, Jordi Sayol wrote:
 We should then generate dmd packages for every distro release because they can
fail between releases too.

Or just let specific distro package maintainers do their job.
 The only thing that breaks the current dmd zip system is libcurl dependency.
All other thinks properly works on all Linux distros/releases.

Are you actually sure? _all_ stuff? _all_ distros? I doubt it.

Please can you show something on dmd/phobos that fails on any actively maintained Linux distro that properly works on another one? Just libcurl dependency.
 
 Should we change all this well working system due to a single problem with an
external library dependency?

It is not working, it just happens to not work for a very tiny user base which is more likely to workaround the issue by building from source other than reporting it. Really, I don't care what the solution is. You can leave current archive as-is and close that Fedora issue as invalid, it is not that important. Just stop pretending it is supported for all platforms and that it can _possibly_ be supported for all platforms.

Things, if simpler, much better. I think is better to keep a single dmd/phobos binary/libray for Linux if they are capable to work on all active distros, and this is the case without libcurl dependency. If a new libcurl D wrapper project is created, I'll be the first to generate deb packages for it, but out of phobos. -- Jordi Sayol
Nov 28 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 28 November 2013 at 17:12:25 UTC, Jordi Sayol wrote:
 Please can you show something on dmd/phobos that fails on any 
 actively maintained Linux distro that properly works on another 
 one?

The dmd zip doesn't work on CentOS 5 because of a libc version mismatch, so it has to be recompiled from source, and then sometimes dmd.conf has to be edited too because it passes an option that older ld's didn't understand. CentOS 5 may be old, but it is still in use in a lot of places. BTW I'm in favor of keeping the zip, it is perfect for me on my slackware+wine setup. But adding other options is cool too.
Nov 28 2013
prev sibling next sibling parent Jordi Sayol <g.sayol yahoo.es> writes:
El 28/11/13 18:17, Adam D. Ruppe ha escrit:
 On Thursday, 28 November 2013 at 17:12:25 UTC, Jordi Sayol wrote:
 Please can you show something on dmd/phobos that fails on any actively
maintained Linux distro that properly works on another one?

The dmd zip doesn't work on CentOS 5 because of a libc version mismatch, so it has to be recompiled from source, and then sometimes dmd.conf has to be edited too because it passes an option that older ld's didn't understand. CentOS 5 may be old, but it is still in use in a lot of places.

It properly runs on Centos 6 (6.4). Which Centos version? 5.0? 5.10?
 
 
 BTW I'm in favor of keeping the zip, it is perfect for me on my slackware+wine
setup. But adding other options is cool too.

Sure, I'm agree. -- Jordi Sayol
Nov 28 2013
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 27 Nov 2013 21:08:20 +0100
schrieb Jacob Carlborg <doob me.com>:

 I mean that I like the current zip to stay.

Don't be so ignorant. The zip is broken for all Linux systems bug Debian. I thought that with D Version Manager you tried to support more than one Linux distribution... -- Marco
Nov 28 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 28, 2013 08:36:08 Jacob Carlborg wrote:
 On 2013-11-27 21:46, Dicebot wrote:
 Distributing binaries for Mac and Windows is fine (though it is much
 better to keep those separate archives and patch DVM accordingly), this
 thread is about Linux ones. You did not have any problems with it
 because Debian derivatives take major percentage of users and you are
 likely to have one.

If we're removing the zip as it currently is, it's another prof that D isn't stable as it claims to be.

The zip has nothing to do with the stability of D itself. It just has to do with the stalibity of how D is distributed, which is a completely different issue. I can see why you care about the zip, given that you wrote and maintain a tool which relies on it, but calling D unstable because we might change how we release it would be like calling KDE or gnome unstable, because they changed from distributing their code in tar.gz files to tar.bz2 files. How a program or library is distributed has nothing to do with the stability of the code itself. - Jonathan M Davis
Nov 28 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Nov 28, 2013 at 11:29:04AM -0800, Andrei Alexandrescu wrote:
[...]
 Besides the fundamental point remains. People don't need to download
 all platform files on one platform. It just doesn't follow. And the
 larger the distribution will become the more annoying the issue will
 be.

I don't see what's the big deal with providing one zip per supported platform (and each Linux distro is to be considered a separate platform here), *and* a dmd_everything.zip for distributors who *want* to have everything in one (they are installing D onto a network of heterogenous machines, they're making a distribution CD, etc.). The default download *should* be the platform-specific zip / tarball / package, but there's no need to kill the all-inclusive zip that contains literally *everything*, for the people who want it for whatever reason. T -- First Rule of History: History doesn't repeat itself -- historians merely repeat each other.
Nov 28 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, November 28, 2013 21:23:33 Jacob Carlborg wrote:
 On 2013-11-28 20:17, Jonathan M Davis wrote:
 The zip has nothing to do with the stability of D itself. It just has to
 do
 with the stalibity of how D is distributed, which is a completely
 different
 issue. I can see why you care about the zip, given that you wrote and
 maintain a tool which relies on it, but calling D unstable because we
 might change how we release it would be like calling KDE or gnome
 unstable, because they changed from distributing their code in tar.gz
 files to tar.bz2 files. How a program or library is distributed has
 nothing to do with the stability of the code itself.

So where do we draw the line, what's considered to be part of D, if the releases aren't.

D is the language specification, dmd is the reference implementation of the language, druntime is its standard runtime library, and Phobos is its standard library. I don't see how the format that dmd and the standard runtime and library are distributed in is at all "part of D." It's just how it's currently distributed. You have a legitimate concern in that you have written a program which relies on how dmd is currently distributed, but I don't see how you can argue that that has anything to do with what D is. And while I don't think that we should just willy-nilly change how we distribute dmd, I would point out that we never actually promised that anything about how dmd is distributed would always stay that way. dmd has been distributed as a zip, because that happens to be how Walter decided to do it, and there's technically no guarantee that we will continue to do so. Now, the fact that a useful tool written by one of this community's well-known members and which is used by a number of folks in the community relies on the zip being put together the way that it's put together and named the way that it's name should be taken into consideration when we look at changing how dmd is distributed. But I think that you have to acknowledge that as useful as DVM is, it's relying on something that was never actually guaranteed. It just happens to be how we've done it up to now. And a number of people have been clamoring for some time that we change how do it and that the zip makes no sense, so I find it unlikely that we will continue to distribute the zip as-is. Maybe we leave it as-is temporarily, but I think that there's a good chance that at some point, you're just going to have to change how DVM works if you want it to continue to function - particularly since the zip really doesn't make sense and makes less and less sense as time goes on (e.g. the shared library situation doesn't work at all with the zip, and soon enough that will be more than just Linux). - Jonathan M Davis
Nov 28 2013
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Tuesday, 26 November 2013 at 19:21:31 UTC, Andrei Alexandrescu 
wrote:
 On 11/26/13 11:13 AM, Iain Buclaw wrote:
 Remembering to include '-lcurl' when importing std.net.curl is 
 perhaps
 one urk.  But this will be a no longer a problem if libphobos 
 is built
 shared.

Interesting. Can we supply a pragma that automates the flag thingie? Andrei

It doesn't work: http://forum.dlang.org/thread/yosxnqziavcnnxkfhtgm forum.dlang.org
Nov 29 2013
prev sibling next sibling parent "Jonas Drewsen" <nospam4321 hotmail.com > writes:
On Thursday, 28 November 2013 at 16:24:34 UTC, Adam D. Ruppe 
wrote:
 On Thursday, 28 November 2013 at 11:37:45 UTC, Jonas Drewsen 
 wrote:
 I definitely agree. In addition D has obtained some nice 
 features since the definition of the std.net.curl API that I 
 would liked have used back then.

What are you thinking of there? I think plain old classes and delegates are a good fit for http stuff.

Indeed, the basic design for wrapping curl is not that bad I think. One thing that was not available was UFCS which I think could have made the API look a bit different. Another thing that could have have affected the API is support for named parameters which is still not supported (and probably never will?). /Jonas
Dec 02 2013
prev sibling parent "Suliman" <evermind live.ru> writes:
Till we have such issues 
http://forum.dlang.org/thread/mailman.357.1384163617.9546.digital
ars-d puremagic.com 
I think we should have native replacement.
May 07 2014