www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Why doesn't Phobos have HTTP related functions?

reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Is it just me, or does Phobos lack URI parsing and HTTP related functions?

Yes, there are other solutions - like cURL - available, but they 
definitely don't take advantage of D's benefits over C/C++.  Ironically, 
I would think HTTP and similar would be an area D would shine in (since 
some many strings are involved.)

It's not difficult to write a conformant HTTP/1.1 client in D, 
especially with std.zlib and similar.  I have one (which only allows 
GET, but that's all I need it for) and would be willing to contribute 
it, or a more thorough implementation (POST, etc.) if it would be welcome.

I'm also aware of the htmlget.d sample... but, that's not exactly a good 
example of a conformant HTTP/1.1 client.  By the way: it is definitely 
an HTTP/1.0 client, and should not be advertising otherwise.

Anyway, as long winded as I can be, my question is: is it desired, and 
if so, how would the interface be preferred (e.g. more like cURL, more 
like .NET's HttpWebRequest, more like PHP's file_get_contents, or more 
unique)?

Thanks,
-[Unknown]
Dec 08 2005
parent reply Don Clugston <dac nospam.com.au> writes:
Unknown W. Brackets wrote:

 Is it just me, or does Phobos lack URI parsing and HTTP related functions?



 Yes, there are other solutions - like cURL - available, but they 
 definitely don't take advantage of D's benefits over C/C++.  Ironically, 
 I would think HTTP and similar would be an area D would shine in (since 
 some many strings are involved.)


Isn't all that stuff in Mango (at dsource)? From what I've seen, it 
looks *very* comprehensive. Mango has the worst one-line project 
description I've ever seen: "The Mango tree". Apparently, that means 
HTTP? But the code is fantastic.
Dec 08 2005
next sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
And yet:

1. It nor htmlget.d properly support multiple-address domains, failover, 
etc.
2. Nowhere in Mango do I see an implementation of chunked encoding, 
required as per the HTTP/1.1 spec.
3. It does not appear to (properly) follow redirections for you.
4. It does not seem to implement any sort of timeout.
5. It is not very easy to use in a D program, and involves too much 
logic that should be removed to the library when dealing with such a 
technology.

Of course, this is all just my opinion - and may not be part of the 
design goals of Mango (which looks concentrated on implementing a D 
servlet to me.)

I deal with a lot of web-related stuff, but it seems to me that - these 
days - most non-trivial non-console programs end up having some reason 
to access the web before long.  Even if just for version updates.

-[Unknown]



 Unknown W. Brackets wrote:
 

 Is it just me, or does Phobos lack URI parsing and HTTP related 
 functions?


 
 

 Yes, there are other solutions - like cURL - available, but they 
 definitely don't take advantage of D's benefits over C/C++.  
 Ironically, I would think HTTP and similar would be an area D would 
 shine in (since some many strings are involved.)


 
 
 Isn't all that stuff in Mango (at dsource)? From what I've seen, it 
 looks *very* comprehensive. Mango has the worst one-line project 
 description I've ever seen: "The Mango tree". Apparently, that means 
 HTTP? But the code is fantastic.
Dec 08 2005
parent reply "Kris" <fu bar.com> writes:
Yes :-)

You should definately have a look at Mango for HTTP needs!


"Unknown W. Brackets" <unknown simplemachines.org> wrote in

 And yet:

 1. It nor htmlget.d properly support multiple-address domains, failover, 
 etc.


Correct. The mango HTTP support is intended to be somewhat simplistic. But, 
an HTTP *client* doesn't require the features you note.



 2. Nowhere in Mango do I see an implementation of chunked encoding, 
 required as per the HTTP/1.1 spec.


There is currently no chunking, and the documentation notes that. However, 
you can easily add chunking yourself. Even better ... add suppport to Mango 
~ it is open source and welcomes input from everyone!



 3. It does not appear to (properly) follow redirections for you.


The redirection (MovedPermanently, MovedTemporarily) is returned to the 
client, which can decide what it wants to do.



 4. It does not seem to implement any sort of timeout.


Not true. The socket support in Mango, from Chris Miller, has timout 
control. By default, the Mango HTTP client has a 3s timeout ~ fully 
configurable.



 5. It is not very easy to use in a D program, and involves too much logic 
 that should be removed to the library when dealing with such a technology.



? Here's a sample use case from the documentation :

=======================

        // callback for client reader
        void sink (char[] content)
        {
                Stdout (content);
        }

        // create client for a GET request
        auto client = new HttpClient (HttpClient.Get, 
"http://www.digitalmars.com/d/intro.html");

        // setup a Host header
        client.getRequestHeaders.add (HttpHeader.Host, 
client.getUri.getHost());

        // make request
        client.open ();

        // check return status for validity
        if (client.isResponseOK)
           {
           // extract content length (be aware of -1 return, for no header)
           int length = client.getResponseHeaders.getInt 
(HttpHeader.ContentLength);
           if (length < 0)
               length = int.max;

           // display all returned headers
           Stdout (client.getResponseHeaders);

           // display remaining content
           client.read (&sink, length);
           }
        else
           Stderr (client.getResponse);

================



 Of course, this is all just my opinion - and may not be part of the design 
 goals of Mango (which looks concentrated on implementing a D servlet to 
 me.)



I do wish you'd take a closer look. The Mango project has all kinds of 
goodies:

* A high-performance and very flexible I/O system ~ puts Java I/O to shame 
(that's not hard though).
* Full ICU bindings
* It remains the /only/ D library that supports char, wchar, dchar 
conversion from any external representation; out of the box. Even the 
console is correctly encoded for the relevant O/S.
* Most of the I/O, formatting, parsing etc is fully enabled for 
char/wchar/dchar
* Has easy to use UnicodeFile and UnicodeBom modules (currently in SVN)
* Has a low-overhead clustered-caching system for those you use such things 
~ intended as part of server-failover support.
* A very high performance HTTP server that can serve requests using zero 
runtime heap-allocations. It's been called the "poster child" for D 
array-slicing :-)
* A D Servlet engine (also zero runtime heap-allocation)
* An implementation of Log4J (again, no runtime heap-allocation) with remote 
Chainsaw support, and dynamic web-based logger control over a running 
application
* A bunch of other goodies, including XML, RPC, and container classes not 
yet fully released/documented.
* Compiles with Ares or Phobos
* It's free. It's Open Source.

Mango is also reasonably streamlined ~ the "Hello World \u263a" is 77KB, 
with a unicode enabled console on Ares. I recently saw a post that noted the 
Phobos version was ~300KB? The Mango version would be notably smaller if it 
were possible to remove the redundant I/O support from the C library (it's 
all tied into snn.lib in a variety of ways).



 I deal with a lot of web-related stuff, but it seems to me that - these 
 days - most non-trivial non-console programs end up having some reason to 
 access the web before long.  Even if just for version updates.


I agree. Would you perhaps consider helping to strengthen the Mango HTTP 
facilities? Proxy support would be great, along with auto-rediection 
perhaps? Chunking maybe? It'd be great to have more people work on this 
stuff ... it'd be even better to get some expert assistance!

What do you say? Can you help out?




 -[Unknown]



 Unknown W. Brackets wrote:


 Is it just me, or does Phobos lack URI parsing and HTTP related 
 functions?




 Yes, there are other solutions - like cURL - available, but they 
 definitely don't take advantage of D's benefits over C/C++.  Ironically, 
 I would think HTTP and similar would be an area D would shine in (since 
 some many strings are involved.)



 Isn't all that stuff in Mango (at dsource)? From what I've seen, it looks 
 *very* comprehensive. Mango has the worst one-line project description 
 I've ever seen: "The Mango tree". Apparently, that means HTTP? But the 
 code is fantastic.
Dec 08 2005
parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
Kris,

That's not what I mean.  I'm not looking for an HTTP client.  I consider 
myself more than capable of writing one - that's not my concern at all.

Perhaps I should rephrase; I think Phobos, or at least Mango, should 
have a web client.  I despise that phrase for it (and "Web 2.0"), 
though, even while it is more directly what I mean.

If I didn't know much about HTTP, and I wanted to make a text editor 
check a text file on a server for the latest version number of my 
software, I don't want to have to deal with contrived, complicated, and 
knowledge-I-don't-have-requiring libraries.  What is a header?  The same 
goes for a program that is grabbing a list (xml) of themes, extensions, 
plugins, whatever.

Chunked encoding is definitely anything but difficult, but it's not 
completely trivial - and it generally makes things more efficient. 
Accepting compressed (gzip/deflate) content, supported by most servers, 
also decreases bandwidth - if, for example, my program also downloads 
said extensions via HTTP.

Of course, this could all be done with XML-RPC, which is far from 
trivial.  I find it completely possible and reasonable that someone or 
some group of people without the experience and knowledge necessary to 
deal with XML-RPC might want to, and be able to, develop a non-trivial 
application.

Again, Mango seems more-or-less intended for people with experience with 
HTTP, which is not what I'm speaking on.  I expect such people to be 
more than qualified to deal with the issues involved, or know that they 
don't need to.  These people will think to deal with redirections before 
it is necessary to do so.

I see no immediately apparent way to create an HttpClient with a timeout 
of 30 seconds.

That code segment... 12 lines or so, to what?  Inefficiently read in a 
file, without sending a user agent (some servers/scripts erroneously 
send a 500 in this case) and without redirection handling in the case 
that Walter changes his links.  From my perspective (knowing HTTP), the 
class is nothing but a socket wrapper, and is nearly entirely useless.

For what Mango looks to be designed for, it is well and good, and looks 
to have many interesting and good features.  Server failover, XML 
parsing, RPC, and clustered caching are all things I've implemented in 
the past (not in D) and I'm quite glad to see them in Mango.

Still, that's simply not what I was talking about.

I may have time to implement, or provide code for, some of those things. 
  Both are quite simple really; for redirects, the idea is simply to 
recurse to a set (configurable, I should think, at least globally at 
compile time) level, following the Location header for 301/302/303/307 
responses.  The hardest part of chunking is parsing the valid hex 
characters on a line and skipping any others.

But I still feel that Phobos could use convenience functions.  Probably 
some even matching the WHAT-WG WebApps interfaces (e.g., the 
recently-popularized XMLHttpRequest.)

-[Unknown]



 Yes :-)
 
 You should definately have a look at Mango for HTTP needs!
 
 
 "Unknown W. Brackets" <unknown simplemachines.org> wrote in
 

And yet:

1. It nor htmlget.d properly support multiple-address domains, failover, 
etc.


 
 
 Correct. The mango HTTP support is intended to be somewhat simplistic. But, 
 an HTTP *client* doesn't require the features you note.
 
 
 

2. Nowhere in Mango do I see an implementation of chunked encoding, 
required as per the HTTP/1.1 spec.


 
 
 There is currently no chunking, and the documentation notes that. However, 
 you can easily add chunking yourself. Even better ... add suppport to Mango 
 ~ it is open source and welcomes input from everyone!
 
 
 

3. It does not appear to (properly) follow redirections for you.


 
 
 The redirection (MovedPermanently, MovedTemporarily) is returned to the 
 client, which can decide what it wants to do.
 
 
 

4. It does not seem to implement any sort of timeout.


 
 
 Not true. The socket support in Mango, from Chris Miller, has timout 
 control. By default, the Mango HTTP client has a 3s timeout ~ fully 
 configurable.
 
 
 

5. It is not very easy to use in a D program, and involves too much logic 
that should be removed to the library when dealing with such a technology.


 
 
 
 ? Here's a sample use case from the documentation :
 
 =======================
 
         // callback for client reader
         void sink (char[] content)
         {
                 Stdout (content);
         }
 
         // create client for a GET request
         auto client = new HttpClient (HttpClient.Get, 
 "http://www.digitalmars.com/d/intro.html");
 
         // setup a Host header
         client.getRequestHeaders.add (HttpHeader.Host, 
 client.getUri.getHost());
 
         // make request
         client.open ();
 
         // check return status for validity
         if (client.isResponseOK)
            {
            // extract content length (be aware of -1 return, for no header)
            int length = client.getResponseHeaders.getInt 
 (HttpHeader.ContentLength);
            if (length < 0)
                length = int.max;
 
            // display all returned headers
            Stdout (client.getResponseHeaders);
 
            // display remaining content
            client.read (&sink, length);
            }
         else
            Stderr (client.getResponse);
 
 ================
 
 
 

Of course, this is all just my opinion - and may not be part of the design 
goals of Mango (which looks concentrated on implementing a D servlet to 
me.)


 
 
 
 I do wish you'd take a closer look. The Mango project has all kinds of 
 goodies:
 
 * A high-performance and very flexible I/O system ~ puts Java I/O to shame 
 (that's not hard though).
 * Full ICU bindings
 * It remains the /only/ D library that supports char, wchar, dchar 
 conversion from any external representation; out of the box. Even the 
 console is correctly encoded for the relevant O/S.
 * Most of the I/O, formatting, parsing etc is fully enabled for 
 char/wchar/dchar
 * Has easy to use UnicodeFile and UnicodeBom modules (currently in SVN)
 * Has a low-overhead clustered-caching system for those you use such things 
 ~ intended as part of server-failover support.
 * A very high performance HTTP server that can serve requests using zero 
 runtime heap-allocations. It's been called the "poster child" for D 
 array-slicing :-)
 * A D Servlet engine (also zero runtime heap-allocation)
 * An implementation of Log4J (again, no runtime heap-allocation) with remote 
 Chainsaw support, and dynamic web-based logger control over a running 
 application
 * A bunch of other goodies, including XML, RPC, and container classes not 
 yet fully released/documented.
 * Compiles with Ares or Phobos
 * It's free. It's Open Source.
 
 Mango is also reasonably streamlined ~ the "Hello World \u263a" is 77KB, 
 with a unicode enabled console on Ares. I recently saw a post that noted the 
 Phobos version was ~300KB? The Mango version would be notably smaller if it 
 were possible to remove the redundant I/O support from the C library (it's 
 all tied into snn.lib in a variety of ways).
 
 
 

I deal with a lot of web-related stuff, but it seems to me that - these 
days - most non-trivial non-console programs end up having some reason to 
access the web before long.  Even if just for version updates.


 
 
 I agree. Would you perhaps consider helping to strengthen the Mango HTTP 
 facilities? Proxy support would be great, along with auto-rediection 
 perhaps? Chunking maybe? It'd be great to have more people work on this 
 stuff ... it'd be even better to get some expert assistance!
 
 What do you say? Can you help out?
Dec 10 2005
parent "Kris" <fu bar.com> writes:
Yes; you are quite right, and I thank you for such a considered post. FWIW, 
the timeout for HttpClient is in the method open():

~~~~~~~~~~

        /*************************************************

                Make a request for the resource specified via the ctor,
                using the specified timeout period (in milli-seconds).The
                return value represents the input buffer, from which all
                returned headers and content may be accessed.

        *************************************************/

        IBuffer open (uint timeout = DefaultReadTimeout)
        {
                return open (timeout, null);
        }

~~~~~~~~~

I will add chunking to both the server and to the client. I'll also add an 
HttpClient subclass that handles redirect. Probably during the week. If 
you'd be willing to identify any further special cases for each of the 
redirects, that would be much appreciated. As would any other assistance be, 
if and as you find time. Perhaps the best place to post such things would be 
over at dsource.org, in the Mango forums?

Per XML-RPC, you may be interested in what Teqdruid has been doing in that 
arena.

Regards;

- Kris




"Unknown W. Brackets" <unknown simplemachines.org> wrote in message 
news:dngdga$hse$1 digitaldaemon.com...

 Kris,

 That's not what I mean.  I'm not looking for an HTTP client.  I consider 
 myself more than capable of writing one - that's not my concern at all.

 Perhaps I should rephrase; I think Phobos, or at least Mango, should have 
 a web client.  I despise that phrase for it (and "Web 2.0"), though, even 
 while it is more directly what I mean.

 If I didn't know much about HTTP, and I wanted to make a text editor check 
 a text file on a server for the latest version number of my software, I 
 don't want to have to deal with contrived, complicated, and 
 knowledge-I-don't-have-requiring libraries.  What is a header?  The same 
 goes for a program that is grabbing a list (xml) of themes, extensions, 
 plugins, whatever.

 Chunked encoding is definitely anything but difficult, but it's not 
 completely trivial - and it generally makes things more efficient. 
 Accepting compressed (gzip/deflate) content, supported by most servers, 
 also decreases bandwidth - if, for example, my program also downloads said 
 extensions via HTTP.

 Of course, this could all be done with XML-RPC, which is far from trivial. 
 I find it completely possible and reasonable that someone or some group of 
 people without the experience and knowledge necessary to deal with XML-RPC 
 might want to, and be able to, develop a non-trivial application.

 Again, Mango seems more-or-less intended for people with experience with 
 HTTP, which is not what I'm speaking on.  I expect such people to be more 
 than qualified to deal with the issues involved, or know that they don't 
 need to.  These people will think to deal with redirections before it is 
 necessary to do so.

 I see no immediately apparent way to create an HttpClient with a timeout 
 of 30 seconds.

 That code segment... 12 lines or so, to what?  Inefficiently read in a 
 file, without sending a user agent (some servers/scripts erroneously send 
 a 500 in this case) and without redirection handling in the case that 
 Walter changes his links.  From my perspective (knowing HTTP), the class 
 is nothing but a socket wrapper, and is nearly entirely useless.

 For what Mango looks to be designed for, it is well and good, and looks to 
 have many interesting and good features.  Server failover, XML parsing, 
 RPC, and clustered caching are all things I've implemented in the past 
 (not in D) and I'm quite glad to see them in Mango.

 Still, that's simply not what I was talking about.

 I may have time to implement, or provide code for, some of those things. 
 Both are quite simple really; for redirects, the idea is simply to recurse 
 to a set (configurable, I should think, at least globally at compile time) 
 level, following the Location header for 301/302/303/307 responses.  The 
 hardest part of chunking is parsing the valid hex characters on a line and 
 skipping any others.

 But I still feel that Phobos could use convenience functions.  Probably 
 some even matching the WHAT-WG WebApps interfaces (e.g., the 
 recently-popularized XMLHttpRequest.)

 -[Unknown]



 Yes :-)

 You should definately have a look at Mango for HTTP needs!


 "Unknown W. Brackets" <unknown simplemachines.org> wrote in


And yet:

1. It nor htmlget.d properly support multiple-address domains, failover, 
etc.



 Correct. The mango HTTP support is intended to be somewhat simplistic. 
 But, an HTTP *client* doesn't require the features you note.




2. Nowhere in Mango do I see an implementation of chunked encoding, 
required as per the HTTP/1.1 spec.



 There is currently no chunking, and the documentation notes that. 
 However, you can easily add chunking yourself. Even better ... add 
 suppport to Mango ~ it is open source and welcomes input from everyone!




3. It does not appear to (properly) follow redirections for you.



 The redirection (MovedPermanently, MovedTemporarily) is returned to the 
 client, which can decide what it wants to do.




4. It does not seem to implement any sort of timeout.



 Not true. The socket support in Mango, from Chris Miller, has timout 
 control. By default, the Mango HTTP client has a 3s timeout ~ fully 
 configurable.




5. It is not very easy to use in a D program, and involves too much logic 
that should be removed to the library when dealing with such a 
technology.




 ? Here's a sample use case from the documentation :

 =======================

         // callback for client reader
         void sink (char[] content)
         {
                 Stdout (content);
         }

         // create client for a GET request
         auto client = new HttpClient (HttpClient.Get, 
 "http://www.digitalmars.com/d/intro.html");

         // setup a Host header
         client.getRequestHeaders.add (HttpHeader.Host, 
 client.getUri.getHost());

         // make request
         client.open ();

         // check return status for validity
         if (client.isResponseOK)
            {
            // extract content length (be aware of -1 return, for no 
 header)
            int length = client.getResponseHeaders.getInt 
 (HttpHeader.ContentLength);
            if (length < 0)
                length = int.max;

            // display all returned headers
            Stdout (client.getResponseHeaders);

            // display remaining content
            client.read (&sink, length);
            }
         else
            Stderr (client.getResponse);

 ================




Of course, this is all just my opinion - and may not be part of the 
design goals of Mango (which looks concentrated on implementing a D 
servlet to me.)




 I do wish you'd take a closer look. The Mango project has all kinds of 
 goodies:

 * A high-performance and very flexible I/O system ~ puts Java I/O to 
 shame (that's not hard though).
 * Full ICU bindings
 * It remains the /only/ D library that supports char, wchar, dchar 
 conversion from any external representation; out of the box. Even the 
 console is correctly encoded for the relevant O/S.
 * Most of the I/O, formatting, parsing etc is fully enabled for 
 char/wchar/dchar
 * Has easy to use UnicodeFile and UnicodeBom modules (currently in SVN)
 * Has a low-overhead clustered-caching system for those you use such 
 things ~ intended as part of server-failover support.
 * A very high performance HTTP server that can serve requests using zero 
 runtime heap-allocations. It's been called the "poster child" for D 
 array-slicing :-)
 * A D Servlet engine (also zero runtime heap-allocation)
 * An implementation of Log4J (again, no runtime heap-allocation) with 
 remote Chainsaw support, and dynamic web-based logger control over a 
 running application
 * A bunch of other goodies, including XML, RPC, and container classes not 
 yet fully released/documented.
 * Compiles with Ares or Phobos
 * It's free. It's Open Source.

 Mango is also reasonably streamlined ~ the "Hello World \u263a" is 77KB, 
 with a unicode enabled console on Ares. I recently saw a post that noted 
 the Phobos version was ~300KB? The Mango version would be notably smaller 
 if it were possible to remove the redundant I/O support from the C 
 library (it's all tied into snn.lib in a variety of ways).




I deal with a lot of web-related stuff, but it seems to me that - these 
days - most non-trivial non-console programs end up having some reason to 
access the web before long.  Even if just for version updates.



 I agree. Would you perhaps consider helping to strengthen the Mango HTTP 
 facilities? Proxy support would be great, along with auto-rediection 
 perhaps? Chunking maybe? It'd be great to have more people work on this 
 stuff ... it'd be even better to get some expert assistance!

 What do you say? Can you help out?
Dec 10 2005
prev sibling parent reply Brad Anderson <brad dsource.dot.org> writes:
Don Clugston wrote:

 Mango has the worst one-line project 
 description I've ever seen: "The Mango tree".


http://trac.dsource.org/projects/mango/

There's a little better description on the wiki.

BA
Dec 08 2005
parent reply "Kris" <fu bar.com> writes:
Thanks, Brad.

Everyone should check out what Brad has provisioned using Trac, at dsource. 
He's doing the community an enormous favour ... we all owe him a very big 
"thank you" :)

- Kris

P.S. for some nice Trac goodies, check out the Roadmap and Tickets at 
http://trac.dsource.org/projects/ddl


"Brad Anderson" <brad dsource.dot.org> wrote

 Don Clugston wrote:

 Mango has the worst one-line project description I've ever seen: "The 
 Mango tree".


 http://trac.dsource.org/projects/mango/

 There's a little better description on the wiki.

 BA
Dec 08 2005
parent pragma <pragma_member pathlink.com> writes:
In article <dna8nh$29tq$1 digitaldaemon.com>, Kris says...

Thanks, Brad.

Everyone should check out what Brad has provisioned using Trac, at dsource. 
He's doing the community an enormous favour ... we all owe him a very big 
"thank you" :)

- Kris

P.S. for some nice Trac goodies, check out the Roadmap and Tickets at 
http://trac.dsource.org/projects/ddl


Thanks for the plug Kris. ;)

And seriously, this tool alone has changed how I work nearly as much as D has:
I never used to be this organized.

-- Dsource: becuase we're better looking than sourceforge. --


- EricAnderton at yahoo
Dec 08 2005