www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Network in phobos

reply Adam Ruppe <destructionator gmail.com> writes:
Something that I think would be nice is being able to treat a network
connection the same way we can work with files and stdin.

Is there a plan already in place for phobos? If no, I might have
something usable to get started.

I peered into std.stdio's source and hacked something up. My plan was
simple: open the network, then assign the FILE* to it to the File
struct.

This is just for Linux; I haven't done Windows code like this for a
long time and I don't remember how. But this might be a useful
starting point anyway, so I figure I'll share it.


What I'd ultimately like is something like this for all D platforms,
and an add-on facility for handling multiple connections, including
incoming connections. I actually have something for this too, but it
is incompatible with the File struct (and Linux only again).


=========
import std.stdio;
import std.string;

import std.conv;

version(linux):

import std.c.linux.linux;
import std.c.linux.socket;

alias std.c.linux.socket sock;
alias std.c.linux.linux linux;

enum int PF_INET = 2;
enum int AF_INET = PF_INET;

extern(C) FILE* fdopen(int, const(char)*);

File openNetwork(string host, ushort port) {
	hostent* h;
	sockaddr_in addr;

	h = gethostbyname(std.string.toStringz(host));
	if(h is null)
		throw new StdioException("gethostbyname");

	int s = socket(PF_INET, SOCK_STREAM, 0);
	if(s == -1)
		throw new StdioException("socket");

	scope(failure)
		close(s);

	addr.sin_family = AF_INET;
	addr.sin_port = htons(port);
	std.c.string.memcpy(&addr.sin_addr.s_addr, h.h_addr, h.h_length);

	if(sock.connect(s, cast(sockaddr*) &addr, addr.sizeof) == -1)
		throw new StdioException("Connect failed");

	FILE* fp = fdopen(s, "w+".ptr);

	File f;

	auto imp = new File.Impl(fp, 1, host ~ ":" ~ to!string(port));

	f.p = imp;

	return f;
}
========
Mar 27 2010
next sibling parent Bane <branimir.milosavljevic gmail.com> writes:
Adam Ruppe Wrote:

 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.
 

I think I saw exactly that in Tango lib, maybe you can save time by looking it up. I don't know for sure, newer used it.
Mar 28 2010
prev sibling next sibling parent "Igor Lesik" <curoles yahoo.com> writes:
"Adam Ruppe"
 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.

Yes, it would be so nice to have a good network library, but many things have to be considered in order to cook decent library. One important aspect is support for asynchronous calls; for example TCP/UDP stuff in Boost is placed in to ASIO package.
 This is just for Linux; I haven't done Windows code like this for a
 long time and I don't remember how. But this might be a useful
 starting point anyway, so I figure I'll share it.

code for Linux and Windows. Windows socket API even has select() function. Igor
Mar 28 2010
prev sibling next sibling parent reply BCS <none anon.com> writes:
Hello Adam,

 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.

how about this: http://www.digitalmars.com/d/1.0/phobos/std_socketstream.html or http://www.digitalmars.com/d/2.0/phobos/std_socketstream.html -- ... <IXOYE><
Mar 28 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 03/28/2010 05:07 PM, BCS wrote:
 Hello Adam,

 Something that I think would be nice is being able to treat a network
 connection the same way we can work with files and stdin.

how about this: http://www.digitalmars.com/d/1.0/phobos/std_socketstream.html or http://www.digitalmars.com/d/2.0/phobos/std_socketstream.html

I think Adam's implementation is more interesting because it integrates with File and consequently with the ranges that will come on top of it (e.g. byCharacter, byLine, byChunk). Thanks Adam. If you're willing to make your implementation available to Phobos, you may want to put it in bugzilla. Andrei
Mar 28 2010
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sun, Mar 28, 2010 at 05:43:22PM -0500, Andrei Alexandrescu wrote:
 I think Adam's implementation is more interesting because it integrates 
 with File and consequently with the ranges that will come on top of it 
 (e.g. byCharacter, byLine, byChunk).

Yes, that's what I really wanted (and write/readln). stdin.byLine is possibly the greatest thing ever, and extending it to network connections, and other processes too, is a nice thing to have available. One size doesn't fit all
 Thanks Adam. If you're willing to make your implementation available to 
 Phobos, you may want to put it in bugzilla.

Yes, certainly: http://d.puremagic.com/issues/show_bug.cgi?id=4025 I've written a HTTP client on top of it too. http://arsdnet.net/dcode/http.d (the import network refers to the module now in bugzilla) I'm not completely happy with it yet, but it works pretty well anyway. The File based code is much simpler than the raw socket code I used to use, though the downside is File's calls block. OK for me, but probably not for everyone. -- Adam D. Ruppe http://arsdnet.net
Mar 28 2010
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sun, Mar 28, 2010 at 12:37:14PM -0700, Igor Lesik wrote:
 IMHO, such oversimplified approach is not very useful.

Depends on your needs. A lot of time, I just want to quickly whip something together that fetches something, and I don't care how long it takes*. It's the kind of thing I'd use the nc program on the linux command line to do, but right there in the D. * If I do care how long it takes, I'll just fork() the whole program and let the child to it. I'm thinking about making a generic background!(delegate) template that can do this automatically, but D2 should have threads that work just as well soon anyway. One example is I needed something very simple that speaks SMTP. I wrote it in my fancier network library, and it was a fair amount of code that was a bit overcomplicated. With this kind of thing, it is a fairly brief list of writefln and readln lines.
 Yes, it would be so nice to have a good network library, but many things
 have to be considered in order to cook decent library. One important
 aspect is support for asynchronous calls; for example TCP/UDP stuff
 in Boost is placed in to ASIO package.

In my fancier library <http://arsdnet.net/dcode/netman.d>, which needs some work still to finish, but is usable, all write and read calls go to a buffer. By the way, everything in my dcode directory there is free for the taking if anyone wants it. Anyway, you do something like this: class MyConnection : Connection { override void onDataReceived() { auto a = read(); // gets the buffer of new data write("some data"); // puts that in the outgoing buffer. not // actually written until the function returns changeReadPosition(pos); // discards [0..pos] of the internal // buffer, so the next time you get // called, you deal with leftover info. // good for handling incomplete requests } } The incoming and outgoing buffers aren't actually put on the network until the next call to NetworkManager.proceed(). So your main has to look something like this: void main() { auto netman = new NetworkManager; netman.connect("site.com", 80); while(netman.proceed()) { } } The nice thing about the network manager class is it does incoming connections too: void main() { auto netman = new NetworkManager; netman.listen(80); while(netman.proceed() { } } And each incoming connection is a particular specialization of the Connection class (which you create in a delegate, not shown here), which handles all the work. netman.proceed() calls select() on all its sockets, then calls the appropriate handler based on what happens: onDataReceived, onDisconnect, etc. The downside is my read() and write() functions are very simple, and kinda suck, and every call is async, so even if you don't care about speed, you have to code it with all your functions returning instantly. The various functions and ranges in the stdio File are much easier to work with for lots of things. Ideally, what I think would be nice is having File available for when you want it, and then being able to attach it to a NetworkManager kind of thing which adds async calls to the interface. Or something like that. Maybe the one size can't really fit all, but I'd like to have both options available somehow.
 As long as you use only socket API there is no problem to write generic
 code for Linux and Windows. Windows socket API even has select() function.

Cool. Most my code is just simple socket stuff, so it should be easy to port then. -- Adam D. Ruppe http://arsdnet.net
Mar 29 2010