www.digitalmars.com         C & C++   DMDScript  

D - performance in std.stream

reply "Ben Hinkle" <bhinkle4 juno.com> writes:
Right now std.stream.readLine takes no inputs and returns a
char[] of the line read. Each time it gets called it builds a string
using ~=. Can there be an API that lets you pass in an
"inout char[]" argument that only gets resized if needed. That
would take a burden off the GC when reading files line by line.

That way the existing readLine could be reimplemented as:

char[] readLine()
{
   char[] result;
   readLine(result);
   return result;
}
void readLine(out char[] result)
{
   [fill and only reallocate result if needed]
}

The same would go for readLineW.
-Ben
Mar 03 2004
next sibling parent reply "Walter" <walter digitalmars.com> writes:
You're right, the performance of std.stream isn't good. Want to design a new
one? <g>
Mar 08 2004
parent reply larry cowan <larry_member pathlink.com> writes:
In article <c2j7a9$2kc5$1 digitaldaemon.com>, Walter says...
You're right, the performance of std.stream isn't good. Want to design a new
one? <g>

Giving it a preallocated buffer and indexing them in does not significantly improve the speed. OTOH the stream.d that Andy Friesen wrote a while back as a template "bolt-ins" example (#22877) is much quicker - I think because it does read's (instead of getc's trying to manage ungetc also). It is a neat start, but is incomplete.
Mar 09 2004
next sibling parent reply Andy Friesen <andy ikagames.com> writes:
larry cowan wrote:
 In article <c2j7a9$2kc5$1 digitaldaemon.com>, Walter says...
 
You're right, the performance of std.stream isn't good. Want to design a new
one? <g>

Giving it a preallocated buffer and indexing them in does not significantly improve the speed. OTOH the stream.d that Andy Friesen wrote a while back as a template "bolt-ins" example (#22877) is much quicker - I think because it does read's (instead of getc's trying to manage ungetc also). It is a neat start, but is incomplete.

The biggest disadvantage to the bolt-in approach is that it's inherently dependant on templates. In C++ land this would translate to horrendous compile times, here in D land it means occasional linker weirdness unless you link stream.obj explicitly. Also, what's missing? I tend to suffer from blindness to hypothetical needs other than my own. :) -- andy
Mar 09 2004
parent reply larry cowan <larry_member pathlink.com> writes:
In article <c2m5kb$21oj$1 digitaldaemon.com>, Andy Friesen says...
larry cowan wrote:
 In article <c2j7a9$2kc5$1 digitaldaemon.com>, Walter says...
 
You're right, the performance of std.stream isn't good. Want to design a new
one? <g>

Giving it a preallocated buffer and indexing them in does not significantly improve the speed. OTOH the stream.d that Andy Friesen wrote a while back as a template "bolt-ins". example (#22877) is much quicker - I think because it does read's (instead of getc's trying to manage ungetc also). It is a neat start, but is incomplete.

The biggest disadvantage to the bolt-in approach is that it's inherently dependant on templates. In C++ land this would translate to horrendous compile times, here in D land it means occasional linker weirdness unless you link stream.obj explicitly. Also, what's missing? I tend to suffer from blindness to hypothetical needs other than my own. :) -- andy

The writeWChar() and writeDChar() methods won't compile. It really needs overloaded this() in read and write to specify options for file open. An open() would be nice for reopening after closing, but it's not really needed when you can just instantiate it again easily - why allow explicit close?. Could a read/write be built for this? Or is this what the R and W inheritance (this(File*...)) is to support? Haven't tried to do anything with this, but there are seek requirements for switching back and forth if a read/write open is used (could be automated, but only for the common cases).
Mar 09 2004
parent reply Andy Friesen <andy ikagames.com> writes:
larry cowan wrote:

 The writeWChar() and writeDChar() methods won't compile.

Crazy. I'll take a look.
 It really needs overloaded this() in read and write to specify options for file
 open.

What sort of options? Text/binary, append, and the like?
 An open() would be nice for reopening after closing, but it's not really needed
 when you can just instantiate it again easily - why allow explicit close?.

close() is necessary because D's garbage collector isn't deterministic. The destructor can clean up if you forget, but it's not a good idea to have an open file descriptor laying around like that.
 Could a read/write be built for this? Or is this what the R and W inheritance
 (this(File*...)) is to support?  Haven't tried to do anything with this, but
 there are seek requirements for switching back and forth if a read/write open
is
 used (could be automated, but only for the common cases).

I hadn't considered simultaneous read/write; I doubt it will work. -- andy
Mar 09 2004
parent reply larry cowan <larry_member pathlink.com> writes:
In article <c2mcd0$2ebl$1 digitaldaemon.com>, Andy Friesen says...
larry cowan wrote:

 The writeWChar() and writeDChar() methods won't compile.

Crazy. I'll take a look.

streams.d(529): function toString overloads char[](char c) and char[](creal r) both match argument list for toString
 It really needs overloaded this() in read and write to specify options for file
 open.

What sort of options? Text/binary, append, and the like?

Yeah, the normal filemodes. And rw+ as indicated below... And rb and wb for windoughs.
 An open() would be nice for reopening after closing, but it's not really needed
 when you can just instantiate it again easily - why allow explicit close?.

close() is necessary because D's garbage collector isn't deterministic. The destructor can clean up if you forget, but it's not a good idea to have an open file descriptor laying around like that.

Ok, reasonable, but then you should be able to reopen it (not necessarily in the same mode.
 Could a read/write be built for this? Or is this what the R and W inheritance
 (this(File*...)) is to support?  Haven't tried to do anything with this, but
 there are seek requirements for switching back and forth if a read/write open
is
 used (could be automated, but only for the common cases).

I hadn't considered simultaneous read/write; I doubt it will work.

Not simultaneous - just on the same open(). You switch back and forth using a seek ( maybe to current point + 0L ) to clear the pointer mode and then can issue an opposite mode command, e.g., read, seek, overwrite, seek, read, read, seek-back, overwrite, overwrite, ... . This is used to maintain a work file of current events, or a fixed-size-rec database file. You can do the same with two open file pointers (read & write to same file), but bsd unix has supported this for over 20 years - and I think it came from even earlier at AT&T. You could support smart mode-switching, but usually you want to move the pointer back to the start of the record you just read, or to eof, or to front of file (to update a status rec of some kind), so that's not really useful.
  -- andy

Mar 10 2004
parent Charles Hixson <charleshixsn earthlink.net> writes:
larry cowan wrote:
 In article <c2mcd0$2ebl$1 digitaldaemon.com>, Andy Friesen says...
 
larry cowan wrote:
...
Could a read/write be built for this? Or is this what the R and W inheritance
(this(File*...)) is to support?  Haven't tried to do anything with this, but
there are seek requirements for switching back and forth if a read/write open is
used (could be automated, but only for the common cases).

I hadn't considered simultaneous read/write; I doubt it will work.

Not simultaneous - just on the same open(). You switch back and forth using a seek ( maybe to current point + 0L ) to clear the pointer mode and then can issue an opposite mode command, e.g., read, seek, overwrite, seek, read, read, seek-back, overwrite, overwrite, ... . This is used to maintain a work file of current events, or a fixed-size-rec database file. You can do the same with two open file pointers (read & write to same file), but bsd unix has supported this for over 20 years - and I think it came from even earlier at AT&T. You could support smart mode-switching, but usually you want to move the pointer back to the start of the record you just read, or to eof, or to front of file (to update a status rec of some kind), so that's not really useful.
 -- andy


Yes. Being able to do block oriented read-write to the same file is very important. I'm not sure whether it should be a part of std.stream, or whether it should be in a separate package, but it's really quite important. Without that capability one can, to take an extreme example, end up copying an entire database for a simple one character change.
Mar 10 2004
prev sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
On Wed, 10 Mar 2004 04:12:42 +0000 (UTC), larry cowan
<larry_member pathlink.com> wrote:

In article <c2j7a9$2kc5$1 digitaldaemon.com>, Walter says...
You're right, the performance of std.stream isn't good. Want to design a new
one? <g>

Giving it a preallocated buffer and indexing them in does not significantly improve the speed.

Yeah - there are probably bigger performance issues and I haven't actually tested any changes. But in general creating APIs that generate lots of garbage to collect is a performance problem waiting to happen.
OTOH the stream.d that Andy Friesen wrote a while back as a template "bolt-ins"
example (#22877) is much quicker - I think because it does read's (instead of
getc's trying to manage ungetc also).  It is a neat start, but is incomplete.

I wasn't aware of Andy's stream.d. I'm downloading "apropos" from http://ikagames.com/andy/d/ now since it looks like that is where I can find it. -Ben
Mar 10 2004
prev sibling parent reply "Kris" <someidiot earthlink.net> writes:
I've been building a multi-layer IO package for DSC (D Servlet Container)
that's fully buffered and generates zero garbage.

The layers include:

- raw byte-array style I/O

- informal text-oriented Tokenizer I/O with text/binary conversion

- structured binary I/O (directly to/from variables) with optional endian
flipping

- structured text-oriented input with data conversion

- structured class serialization/deserialization framework

It's quite flexible: for example, one can 'read' a line into a Token (using
a LineTokenizer) and then map said Token into any of the other layers for
further slicing. One can mix and match the different layers in pretty much
any way that makes sense. There is no copying of data unless requested by
the app, and no memory allocation other than app-allocated I/O buffers. It
does random-access where the underlying system supports it (not on a
socket), and is intended to support memory-mapped buffers (for *huge*
files). It uses lookahead instead of unget... for those cases that need such
things.

I'm writing this stuff for a high-performance server (in D), so one of the
primary goals is very low runtime overhead (i.e. zero memory allocation). If
anyone's interested, I'll try to get the first major I/O cut out the door
by, say, the end of the month.

- Kris


"Ben Hinkle" <bhinkle4 juno.com> wrote in message
news:c257jd$1ph8$1 digitaldaemon.com...
 Right now std.stream.readLine takes no inputs and returns a
 char[] of the line read. Each time it gets called it builds a string
 using ~=. Can there be an API that lets you pass in an
 "inout char[]" argument that only gets resized if needed. That
 would take a burden off the GC when reading files line by line.

 That way the existing readLine could be reimplemented as:

 char[] readLine()
 {
    char[] result;
    readLine(result);
    return result;
 }
 void readLine(out char[] result)
 {
    [fill and only reallocate result if needed]
 }

 The same would go for readLineW.
 -Ben

Mar 10 2004
next sibling parent reply Charles Hixson <charleshixsn earthlink.net> writes:
Kris wrote:
 I've been building a multi-layer IO package for DSC (D Servlet Container)
 that's fully buffered and generates zero garbage.
 
 The layers include:
 
 - raw byte-array style I/O
 
 - informal text-oriented Tokenizer I/O with text/binary conversion
 
 - structured binary I/O (directly to/from variables) with optional endian
 flipping
 
 - structured text-oriented input with data conversion
 
 - structured class serialization/deserialization framework
 
 It's quite flexible: for example, one can 'read' a line into a Token (using
 a LineTokenizer) and then map said Token into any of the other layers for
 further slicing. One can mix and match the different layers in pretty much
 any way that makes sense. There is no copying of data unless requested by
 the app, and no memory allocation other than app-allocated I/O buffers. It
 does random-access where the underlying system supports it (not on a
 socket), and is intended to support memory-mapped buffers (for *huge*
 files). It uses lookahead instead of unget... for those cases that need such
 things.
 
 I'm writing this stuff for a high-performance server (in D), so one of the
 primary goals is very low runtime overhead (i.e. zero memory allocation). If
 anyone's interested, I'll try to get the first major I/O cut out the door
 by, say, the end of the month.
 
 - Kris



That sounds extremely interesting. Would it be portable, or platform specific?
Mar 10 2004
parent "Kris" <someidiot earthlink.net> writes:
 That sounds extremely interesting.  Would it be portable, or platform
 specific?

Portable.
Mar 10 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle4 juno.com> writes:
Sounds nice! I'm definitely interested. Some comments/questions below

"Kris" <someidiot earthlink.net> wrote in message
news:c2nl32$1mm2$1 digitaldaemon.com...
 I've been building a multi-layer IO package for DSC (D Servlet Container)
 that's fully buffered and generates zero garbage.

 The layers include:

 - raw byte-array style I/O

 - informal text-oriented Tokenizer I/O with text/binary conversion

I'm not sure what informal means here or what text/binary conversion means. Does it include scanf/printf? Any replacement for std.stream needs to have that.
 - structured binary I/O (directly to/from variables) with optional endian
 flipping

 - structured text-oriented input with data conversion

not sure what structured means but I'm guessing the data being read has a format like "rows of numbers separated by tabs with the first column being a string" or something. Is this what you mean?
 - structured class serialization/deserialization framework

I don't know what you have in mind here but it sounds like a big chunk of work. Could it go into another module?
 It's quite flexible: for example, one can 'read' a line into a Token

 a LineTokenizer) and then map said Token into any of the other layers for
 further slicing. One can mix and match the different layers in pretty much
 any way that makes sense. There is no copying of data unless requested by
 the app, and no memory allocation other than app-allocated I/O buffers. It
 does random-access where the underlying system supports it (not on a
 socket), and is intended to support memory-mapped buffers (for *huge*
 files). It uses lookahead instead of unget... for those cases that need

 things.

 I'm writing this stuff for a high-performance server (in D), so one of the
 primary goals is very low runtime overhead (i.e. zero memory allocation).

 anyone's interested, I'll try to get the first major I/O cut out the door
 by, say, the end of the month.

Mar 10 2004
parent "Kris" <someidiot earthlink.net> writes:
Below:

 I'm not sure what informal means here or what text/binary conversion
 means. Does it include scanf/printf? Any replacement for std.stream needs
 to have that.

formatted", like a command-line or word-count input, or line-input. Formal, or structured, stipulates the input data must conform exactly. Formal inputs throw an exception when the input fails to match an expectation, whereas informal ones just return eof (or an empty Token). Text/binary conversion is converting numbers to text and vice versa (what printf and scanf do). Here's a trivial example: { TextWriter w; int j; // convert to text w << "number is : " << j << w.Newline; } Printf is supported via a thin adaptor: w << Printf.format("%d %d", 10, 20) << ....
 not sure what structured means but I'm guessing the data being read
 has a format like "rows of numbers separated by tabs with the
 first column being a string" or something. Is this what you mean?

an agreed-upon layout that is reflected in the source code. That layout might be implemented as raw binary, or as delimited text (as you suggest). If it's raw binary, then there should also be an agreed Endian layout.
 I don't know what you have in mind here but it sounds like a big
 chunk of work. Could it go into another module?

case it's just a couple of trivial interfaces. Each serializable class must implement a read & write method. If one were deserializing a (previously serialized) class, one would probably follow the structured/formal approach. Here's an example: class MyClass : IReadable, IWritable { private int foo; private float bar; // the IReadable interface void read (IReader r) { r >> foo >> bar; } // the IWritable interface void write (IWriter w) { w << foo << bar; } } { MyClass mc = new MyClass(...); .... BinaryWriter w = new BinaryWriter(...); w << mc; ... } Note that there are a variety of Readers and Writers (binary, text, token, etc) and all of them can be used directly against any class implementing IReadable and/or IWritable. In other words, the implementing class doesn't know, or care, whether it's serialized in binary, text, or something else. As an aside, there's also an ISerializable interface that adds methods getGuid() and create(). This would be used when shipping class instances around a network, sans reflection (something that DSC will support for clustering).
Mar 10 2004
prev sibling parent reply "Carlos Santander B." <carlos8294 msn.com> writes:
"Kris" <someidiot earthlink.net> wrote in message
news:c2nl32$1mm2$1 digitaldaemon.com
| I've been building a multi-layer IO package for DSC (D
| Servlet Container) that's fully buffered and generates
| zero garbage.
| ...

Do you know when you could make a release for DSC?

-----------------------
Carlos Santander Bernal
Mar 10 2004
parent reply "Kris" <someidiot earthlink.net> writes:
Carlos,

I would hope to release an Alpha sometime in April. Initial release will not
support https, because I don't have that expertise. Any volunteers? <g>

- Kris


"Carlos Santander B." <carlos8294 msn.com> wrote in message
news:c2ok8o$efd$1 digitaldaemon.com...
 "Kris" <someidiot earthlink.net> wrote in message
 news:c2nl32$1mm2$1 digitaldaemon.com
 | I've been building a multi-layer IO package for DSC (D
 | Servlet Container) that's fully buffered and generates
 | zero garbage.
 | ...

 Do you know when you could make a release for DSC?

 -----------------------
 Carlos Santander Bernal

Mar 10 2004
parent "Carlos Santander B." <carlos8294 msn.com> writes:
"Kris" <someidiot earthlink.net> wrote in message
news:c2olf2$gn9$1 digitaldaemon.com
| Carlos,
|
| I would hope to release an Alpha sometime in April.
| Initial release will not support https, because I don't
| have that expertise. Any volunteers? <g>
|
| - Kris
|

Not here, sorry. I was just asking because right now I'm in the middle of
building an e-business site but using JSP (including https, certificates,
signed mail...), so I got curious to know if/how that could be done using
DSC. Not that I would use it: deadline is this monday and I still have to
figure out how to attach a file to a signed mail, and make Apache find the
jar for creating the graphic that will be attached. So, I still have a lot
of research to do.

-----------------------
Carlos Santander Bernal
Mar 10 2004