www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - (Phobos - SocketStream) Am I doing something wrong or is this a bug?

reply Zane <zane.sims gmail.com> writes:
While trying to learn sockets in Phobos, I ran into a problem.  I was trying to
download the google logo via an HTTP GET.  when I do a socketStream.read, the
returned "bytes read" value indicates that less bytes were read in than were
available (4097 bytes to be exact).  The google logo is 8558 bytes in size as
indicated by the HTTP headers received.  Here is my code:

module HttpManager;

import std.stream;
import std.stdio;
import std.socket;
import std.socketstream;

import std.string;

int main()
{
	char[] line;
	ubyte[] data = new ubyte[8558];
	uint num = 0;

	TcpSocket socket = new TcpSocket(new InternetAddress("www.google.com", 80));

	socket.send("GET /intl/en_ALL/images/logo.gif HTTP/1.0\r\n\r\n");

	SocketStream socketStream = new SocketStream(socket);
	
	while(!socketStream.eof)
	{
		line = socketStream.readLine();

		if (line=="")
			break;

		writef("%s\n", line);
	}

	num = socketStream.read(data);

	writef("\n\nNum: %d\n\n", num);
	
	socketStream.close;
	socket.close;

	File file = new File("logo.gif", FileMode.Out);
	file.write(data);
	file.close;

	return 0;
}

The console output is:

	HTTP/1.1 200 OK
	Content-Type: image/gif
	Last-Modified: Wed, 07 Jun 2006 19:38:24 GMT
	Date: Tue, 03 Nov 2009 13:04:57 GMT
	Expires: Wed, 03 Nov 2010 13:04:57 GMT
	X-Content-Type-Options: nosniff
	Server: gws
	Content-Length: 8558
	Cache-Control: public, max-age=31536000
	Age: 28322
	X-XSS-Protection: 0


	Num: 4097

I am using dmd v1.050 with dsss v0.78

Sorry if this is a stupid question.  Any help would be very appreciated :)

Thanks,
Zane
Nov 03 2009
parent reply BCS <none anon.com> writes:
Hello Zane,


 While trying to learn sockets in Phobos, I ran into a problem.  I was
 trying to download the google logo via an HTTP GET.  when I do a
 socketStream.read, the returned "bytes read" value indicates that less
 bytes were read in than were available (4097 bytes to be exact).  The
 google logo is 8558 bytes in size as indicated by the HTTP headers
 received.  Here is my code:
 


The network connection might not have all the data yet. Try calling it in 
a loop kind of like your header reading code.
Nov 03 2009
parent reply Zane <zane.sims gmail.com> writes:
BCS Wrote:


 Hello Zane,
 

 While trying to learn sockets in Phobos, I ran into a problem.  I was
 trying to download the google logo via an HTTP GET.  when I do a
 socketStream.read, the returned "bytes read" value indicates that less
 bytes were read in than were available (4097 bytes to be exact).  The
 google logo is 8558 bytes in size as indicated by the HTTP headers
 received.  Here is my code:
 


 
 The network connection might not have all the data yet. Try calling it in 
 a loop kind of like your header reading code.
 
 


I do not see how this can be the case.  Per the Phobos spec for InputStream,
"Read a block of data big enough to fill the given array buffer. "  This sounds
to me like it should do just that, fill my array....and not return until done. 
readLine needs to be looped because it only reads up to a carriage return /
newline combination (and there are multiple lines).  Can anyone confirm that my
speculations are correct?  Otherwise, the documentation seems misleading and I
will need help with a correct implementation.

Zane
Nov 03 2009
parent reply BCS <none anon.com> writes:
Hello Zane,


 BCS Wrote:
 

 Hello Zane,
 

 While trying to learn sockets in Phobos, I ran into a problem.  I
 was trying to download the google logo via an HTTP GET.  when I do a
 socketStream.read, the returned "bytes read" value indicates that
 less bytes were read in than were available (4097 bytes to be
 exact).  The google logo is 8558 bytes in size as indicated by the
 HTTP headers received.  Here is my code:
 


 The network connection might not have all the data yet. Try calling
 it in a loop kind of like your header reading code.
 


 I do not see how this can be the case.  Per the Phobos spec for
 InputStream, "Read a block of data big enough to fill the given array
 buffer. "  This sounds to me like it should do just that, fill my
 array....and not return until done.  readLine needs to be looped
 because it only reads up to a carriage return / newline combination
 (and there are multiple lines).  Can anyone confirm that my
 speculations are correct?  Otherwise, the documentation seems
 misleading and I will need help with a correct implementation.


If I were you, I'd just try the looped version because there might be a bug. 
I seem to recall that the stream stuff in some places gets defined to return 
on out of data conditions. (BTW this would be consistent with the liked named 
"read" function from posix.)

Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't 
block if there isn't enough data. 

If this is a code bug or a documentation deficiency is another question.
Nov 03 2009
parent reply Zane <zane.sims gmail.com> writes:
BCS Wrote:


 Hello Zane,
 

 BCS Wrote:
 

 Hello Zane,
 

 While trying to learn sockets in Phobos, I ran into a problem.  I
 was trying to download the google logo via an HTTP GET.  when I do a
 socketStream.read, the returned "bytes read" value indicates that
 less bytes were read in than were available (4097 bytes to be
 exact).  The google logo is 8558 bytes in size as indicated by the
 HTTP headers received.  Here is my code:
 


 The network connection might not have all the data yet. Try calling
 it in a loop kind of like your header reading code.
 


 I do not see how this can be the case.  Per the Phobos spec for
 InputStream, "Read a block of data big enough to fill the given array
 buffer. "  This sounds to me like it should do just that, fill my
 array....and not return until done.  readLine needs to be looped
 because it only reads up to a carriage return / newline combination
 (and there are multiple lines).  Can anyone confirm that my
 speculations are correct?  Otherwise, the documentation seems
 misleading and I will need help with a correct implementation.


 
 If I were you, I'd just try the looped version because there might be a bug. 
 I seem to recall that the stream stuff in some places gets defined to return 
 on out of data conditions. (BTW this would be consistent with the liked named 
 "read" function from posix.)
 
 Looking into the code (Stream.read() and SocketStream.readBlock()) it doesn't 
 block if there isn't enough data. 
 
 If this is a code bug or a documentation deficiency is another question.
 
 


I assumed since the underlying TcpSocket was blocking, then calling read
through the stream would also block....dunno.  Anyway, so if I were to use a
loop, how could I do this with read?  The size of the read buffer must be
initialized before the reading takes place, however, I do not know how much
will be read for the next "chunk".  If I am to receive these in arbitrarily
sized chunks for concatenation, I don't see a sensible way of constructing a
loop.  Example?

Zane
Nov 03 2009
next sibling parent Jesse Phillips <jessekphillips gmail.com> writes:
On Tue, 03 Nov 2009 20:05:17 -0500, Zane wrote:


 If I am to receive
 these in arbitrarily sized chunks for concatenation, I don't see a
 sensible way of constructing a loop.  Example?
 
 Zane


You can use the number of bytes read to determine and use string slicing 
to concatenation into the final array.
Nov 03 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 03 Nov 2009 20:05:17 -0500, Zane <zane.sims gmail.com> wrote:


 I assumed since the underlying TcpSocket was blocking, then calling read  
 through the stream would also block....dunno.  Anyway, so if I were to  
 use a loop, how could I do this with read?  The size of the read buffer  
 must be initialized before the reading takes place, however, I do not  
 know how much will be read for the next "chunk".  If I am to receive  
 these in arbitrarily sized chunks for concatenation, I don't see a  
 sensible way of constructing a loop.  Example?


Your interpretation of blocking sockets is incorrect. A socket blocks on a  
read only if *no* data is available.  If any data is available, it reads  
as much as it can and returns.

A non blocking socket returns immediately, even if no data is available  
(usually returning an error like EAGAIN).

A simple loop (keep in mind, I don't know phobos, and I didn't look up the  
exact usage):

// ubyte[] buf is the array we want to fill, predetermined length

ubyte[] tmp = buf[];

while(tmp.length > 0)
{
   auto bytesread = socket.read(tmp);
   if(bytesread == 0) break; // EOF (also should check for error here if  
necessary)
   tmp = tmp[bytesread..$];
}

This kind of thing can *easily* be put into a wrapper function.

-Steve
Nov 04 2009
prev sibling parent BCS <none anon.com> writes:
Hello Zane,


 BCS Wrote:
 

 Hello Zane,
 

 BCS Wrote:
 

 Hello Zane,
 

 While trying to learn sockets in Phobos, I ran into a problem.  I
 was trying to download the google logo via an HTTP GET.  when I do
 a socketStream.read, the returned "bytes read" value indicates
 that less bytes were read in than were available (4097 bytes to be
 exact).  The google logo is 8558 bytes in size as indicated by the
 HTTP headers received.  Here is my code:
 


 The network connection might not have all the data yet. Try calling
 it in a loop kind of like your header reading code.
 


 I do not see how this can be the case.  Per the Phobos spec for
 InputStream, "Read a block of data big enough to fill the given
 array buffer. "  This sounds to me like it should do just that, fill
 my array....and not return until done.  readLine needs to be looped
 because it only reads up to a carriage return / newline combination
 (and there are multiple lines).  Can anyone confirm that my
 speculations are correct?  Otherwise, the documentation seems
 misleading and I will need help with a correct implementation.
 


 If I were you, I'd just try the looped version because there might be
 a bug. I seem to recall that the stream stuff in some places gets
 defined to return on out of data conditions. (BTW this would be
 consistent with the liked named "read" function from posix.)
 
 Looking into the code (Stream.read() and SocketStream.readBlock()) it
 doesn't block if there isn't enough data.
 
 If this is a code bug or a documentation deficiency is another
 question.
 


 I assumed since the underlying TcpSocket was blocking, then calling
 read through the stream would also block....dunno.  Anyway, so if I
 were to use a loop, how could I do this with read?  The size of the
 read buffer must be initialized before the reading takes place,
 however, I do not know how much will be read for the next "chunk".  If
 I am to receive these in arbitrarily sized chunks for concatenation, I
 don't see a sensible way of constructing a loop.  Example?


ubyte[] buffer = new ubyte[size];
auto window = buffer;

while(!ss.eof() && window.length > 0)
{
   int i = ss.read(windows);
   window = window[i..$];
}


that, or take a look at readExect.
Nov 04 2009