www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - And yet another cool project: fetching multiple URLs at once

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Right now we can nicely stream an URL as an input range. A great 
extension would be to fetch several URLs at once. When accessing r.front 
for that range, the user gets a pair of URL and data chunk.

Of course the point is to make all these fetches work simultaneously. 
Inside, the range would e.g. use asynchronous curl.


Andrei
Dec 16 2015
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 16 December 2015 at 15:54:09 UTC, Andrei 
Alexandrescu wrote:

 Right now we can nicely stream an URL as an input range. A 
 great extension would be to fetch several URLs at once. When 
 accessing r.front for that range, the user gets a pair of URL 
 and data chunk.

 Of course the point is to make all these fetches work 
 simultaneously. Inside, the range would e.g. use asynchronous 
 curl.


 Andrei


Whenever I want to do that I use vibe.d and my observe library 
(based on RX observables) and the code would look a bit like this:

urls
	.observe()
	.zip(localPaths)
	.fork(numParallel)
	.concatMap!((part)
	{
		return downloadFile(part[0],part[1]).retry(3)
	})
	.subscribe(
		(p){writeln("File %s downloaded to %s",p[0],p[1]);},
		(e){writeln("Error ",e);},
		(){writeln("Completed!");}
	);

Here observe() makes an observable out of any range; fork() uses 
vibe.d's Fibers to achieve concurrency; downloadFile() returns 
another observable that gets concatMap'ped; and retry does what 
you think it does. Also, errors get propagated without throws and 
catches.

I think what I am trying to say is that a lot of stuff is already 
available on code.dlang.org, just not in phobos. Which begs the 
question, should it be? And if it does, shouldn't you rather 
consider merging vibe.d with phobos? Or parts of it?
Dec 16 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/16/2015 02:12 PM, Sebastiaan Koppe wrote:

 I think what I am trying to say is that a lot of stuff is already
 available on code.dlang.org, just not in phobos. Which begs the
 question, should it be? And if it does, shouldn't you rather consider
 merging vibe.d with phobos? Or parts of it?


Generally phobos should be compelling in and of itself and grow in 
breadth and depth over time.

I think a facility for asynchronous HTTP fetching would be a great 
addition. Your code looks pretty rad and well integrated with the rest 
of Phobos. I encourage you to pursue merging it to Phobos.


Andrei
Dec 16 2015
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 16 December 2015 at 19:32:36 UTC, Andrei 
Alexandrescu wrote:

 On 12/16/2015 02:12 PM, Sebastiaan Koppe wrote:

 I think what I am trying to say is that a lot of stuff is 
 already
 available on code.dlang.org, just not in phobos. Which begs the
 question, should it be? And if it does, shouldn't you rather 
 consider
 merging vibe.d with phobos? Or parts of it?


 Generally phobos should be compelling in and of itself and grow 
 in breadth and depth over time.


Well it can never encompass all the library code people write, it 
having a way lower threshold. In that regard I suppose Phobos 
does well when it tries to support libraries by developing good 
low/mid-level code. Stuff like std.algorithms, collections and 
ndslice. But do you really want to put a ODBC client in there, as 
an example? Or a grayscale filter?


 I think a facility for asynchronous HTTP fetching would be a 
 great addition. Your code looks pretty rad and well integrated 
 with the rest of Phobos. I encourage you to pursue merging it 
 to Phobos.


Thanks. It is more an async/observable library than anything 
else. And it depends on vibe.d. When I get rid of the vibe.d 
dependency, or make it optional, I might consider prepping it up 
for inclusion.
Dec 16 2015
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Wednesday, 16 December 2015 at 20:09:23 UTC, Sebastiaan Koppe 
wrote:

 But do you really want to put a ODBC client in there, as an 
 example?


Not sure if you're being serious making a joke, because that 
exists and Andrei was the one who made it 
http://dlang.org/phobos/etc_c_odbc_sql.html
Dec 16 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/16/2015 03:09 PM, Sebastiaan Koppe wrote:

 But do you really want to put a ODBC client in there, as an example? Or
 a grayscale filter?


ODBC maybe, grayscale filter probably not.

I should add I've argued for including some of the core vibe.d stuff in 
Phobos. Sadly nobody is championing such a project for the time being.


Andrei
Dec 16 2015
next sibling parent BLM768 <blm768 gmail.com> writes:
On Wednesday, 16 December 2015 at 21:00:55 UTC, Andrei 
Alexandrescu wrote:

 I should add I've argued for including some of the core vibe.d 
 stuff in Phobos. Sadly nobody is championing such a project for 
 the time being.


 Andrei


Would that include its stream stuff? We've been needing a 
std.stream replacement for a long time.
Dec 16 2015
prev sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Wednesday, 16 December 2015 at 21:00:55 UTC, Andrei 
Alexandrescu wrote:

 On 12/16/2015 03:09 PM, Sebastiaan Koppe wrote:

 But do you really want to put a ODBC client in there, as an 
 example? Or
 a grayscale filter?


 ODBC maybe, grayscale filter probably not.

 I should add I've argued for including some of the core vibe.d 
 stuff in Phobos. Sadly nobody is championing such a project for 
 the time being.


 Andrei


That is a good idea. What would a good approach for such a 
project be? Many small pr's that move core functions to phobos 
piece by piece.
Dec 16 2015