www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Reading data from the network without knowing the size of the buffer

reply Roman Sztergbaum <rmscastle gmail.com> writes:
Hello !

in my function :

```D
   private config_load_answer load_config(string[] args)
     in(args !is null, "args cannot be null")
     in(args.length == 2, "need 1 arguments")
     out(r; r.state == "SUCCESS", "load_config should succeed")
     out(r; !r.config_name.empty, "config_name should not be 
empty")
     out(r; r.config_id > 0, "config_id should be different than 
0")
     {
         string config_key;
         string readonly_config_key;
         getopt(args, "key", &config_key, "readonly_key", 
&readonly_config_key);
         auto cfg = 
deserialize_to!config_load("../API_doc/json_recipes/config_load.json");
         cfg.config_key = config_key;
         cfg.readonly_config_key = readonly_config_key;
         writeln(cfg);
         writeln(cfg.serializeToJsonPretty);
         client_.socket.send(cfg.serializeToJson);
         auto answer = new ubyte[256]; //
         client_.socket.receive(answer);
         (cast(string) answer).writeln;
         return (cast(string) 
answer).deserialize!config_load_answer;
     }
```

I would like to get rid of the "ubytes[256]" because I do not 
know the size of the data that is comming, I would like to read 
the entire buffer that I send at once. Can someone point me?

(i'm pretty new to D langage, but i'm familiar with C++, and for 
example in boost asio  we have `byte_transfered` so we can create 
a buffer with this size and read the incoming data)

Thank's a lot for your help !
Mar 21
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum 
wrote:
 Hello !

 in my function :

 ```D
   private config_load_answer load_config(string[] args)
     in(args !is null, "args cannot be null")
     in(args.length == 2, "need 1 arguments")
     out(r; r.state == "SUCCESS", "load_config should succeed")
     out(r; !r.config_name.empty, "config_name should not be 
 empty")
     out(r; r.config_id > 0, "config_id should be different than 
 0")
     {
         string config_key;
         string readonly_config_key;
         getopt(args, "key", &config_key, "readonly_key", 
 &readonly_config_key);
         auto cfg = 
 deserialize_to!config_load("../API_doc/json_recipes/config_load.json");
         cfg.config_key = config_key;
         cfg.readonly_config_key = readonly_config_key;
         writeln(cfg);
         writeln(cfg.serializeToJsonPretty);
         client_.socket.send(cfg.serializeToJson);
         auto answer = new ubyte[256]; //
         client_.socket.receive(answer);
         (cast(string) answer).writeln;
         return (cast(string) 
 answer).deserialize!config_load_answer;
     }
 ```

 I would like to get rid of the "ubytes[256]" because I do not 
 know the size of the data that is comming, I would like to read 
 the entire buffer that I send at once. Can someone point me?

 (i'm pretty new to D langage, but i'm familiar with C++, and 
 for example in boost asio  we have `byte_transfered` so we can 
 create a buffer with this size and read the incoming data)

 Thank's a lot for your help !
I am also not a expert in this area. If the message received from the server exceeds 256 bytes, you have to call the receive message several times until you know you received all bytes. If the data received is smaller than 256 bytes, the array will contain NULL (character 0 value in the ascii table). This you might could use as indicator you received all values. Is the server really without protocol? If the server code is under your control, you could switch to http, which has a message length attribute. Ps. In case there is a network error, your program will terminate (out contract throws Error) without any chance to avoid this. You should throw an Exception at the end of method body instead for resource issues. Kind regards Andre
Mar 21
next sibling parent Roman Sztergbaum <rmscastle gmail.com> writes:
On Thursday, 21 March 2019 at 18:27:56 UTC, Andre Pany wrote:
 On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum 
 wrote:
 [...]
I am also not a expert in this area. If the message received from the server exceeds 256 bytes, you have to call the receive message several times until you know you received all bytes. If the data received is smaller than 256 bytes, the array will contain NULL (character 0 value in the ascii table). This you might could use as indicator you received all values. Is the server really without protocol? If the server code is under your control, you could switch to http, which has a message length attribute. Ps. In case there is a network error, your program will terminate (out contract throws Error) without any chance to avoid this. You should throw an Exception at the end of method body instead for resource issues. Kind regards Andre
Hello, I didn't precise it, but i use unix local socket communication, the layer is the KERNEL, and not TCP. I think i cannot use module http then. But, i'm surprise that is not an other way that's is more simple, i will wait for some other answers i think. Thank's for your help
Mar 21
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 21 March 2019 at 18:27:56 UTC, Andre Pany wrote:
 If the data received is smaller than 256 bytes, the array will 
 contain NULL (character 0 value in the ascii table). This you 
 might could use as indicator you received all values.
That's wrong, the correct thing to check is the return value of receive(). It gives the length it actually filled in (or 0 if the connection is finished, or negative on error conditions)
Mar 21
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Thursday, 21 March 2019 at 16:54:01 UTC, Roman Sztergbaum 
wrote:
 I would like to get rid of the "ubytes[256]" because I do not 
 know the size of the data that is comming, I would like to read 
 the entire buffer that I send at once. Can someone point me?
If you do not know the size of the response, then how can you know when you should stop waiting for more bytes? Some connections are stream-oriented, and some are datagram-oriented. With a datagram-oriented connection, each packet has a fixed size (though there is a maximum size). You mentioned this is a UNIX socket connection, and I believe UNIX sockets can be either stream-oriented or datagram-oriented. If the socket is stream-oriented, there should generally be a way for the other end to signal when it's done sending bytes. This can be in the form of a fixed-size "header" containing the body length, or maybe the other end just closes the connection when it's done writing. Other approaches, like reading until there are no more bytes in the buffer (though more may yet to arrive one millisecond later) are generally not reliable. Perhaps you should include more information like what program you are trying to talk to, or the protocol being used.
Mar 22