www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.stream very slow

reply "Bruno A. Costa" <bruno codata.com.br> writes:
Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.
Jun 22 2004
next sibling parent Martin <Martin_member pathlink.com> writes:
I think the problem is source.eof(). You can open the stream.d (in
source/phobos/std) and see for yourself. This eof() uses size() and size() seeks
to the end of stream to find out the length of it. It is quite slow. 
THERE IS A SIMPLE FIX: edit the stream.d a little and change the class Stream.
Make an extra variable called size_at_least. Streams can't shrink they only
grow. (Am I right? Well atleast it should be so with files.) So the function
eof() does not have to call size() every time. It can compare the position first
to the size_at_least and if position is bigger (or equal) then it updates the
size_at_least and compares again, and if position is still bigger (or equal)
then it decides that it really is the end of stream. So 99% of cases it only
uses the variable size_at_least not the function size()

There can be a better fix, but it is the simpliest, I think. 
So maybe WALTER BRIGHT can make this little change in Phobos?
But since phobos is open source, you can do it yourself too. (I mean, if you
need the Stream class to work faster.)

Thank you for pointing this problem out! This is very helpful for developers, I
think.

Greetengs, Martin.

In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...
Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.

Martin
Jun 22 2004
prev sibling next sibling parent Arcane Jill <Arcane_member pathlink.com> writes:
std.stream is slow because it's unbuffered. I only discovered this recently, but
you can speed it up DRAMATICALLY by constructing a BufferedStream using a File
as its construction parameter.

This is not documented. Bah.

Arcane Jill
Jun 22 2004
prev sibling next sibling parent Sean Kelly <sean f4.ca> writes:
The standard D streams are non-buffered.  You'll want to use BufferedFileStream
to get better performance.  Beyond that, I'm in the process of working on
stream.d to add some features to it, and I'll look at performance in the
process.  The way buffering is currently done is something I'd like to discuss
once I have a better idea of whether there are alternatives I think may work
better.  None of my changes are guranteed to make it into Phobos, but I'm hoping
that if the community is happy with them then perhaps they will.


Sean
Jun 22 2004
prev sibling next sibling parent Charlie <Charlie_member pathlink.com> writes:
All i can do is laugh at this really, how long has this been a problem ??

You should also checkout Mango's IO system ( http://dsource.org/projects/mango/
) , very clean , and I hear very speedy ( im checking this out myself now ).

Charlie

In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...
Hi,

It seems to me that some parts of the std.stream module are working VERY
slow. I've done some tests with file manipulation and some comparissions
with C++ and the results are strange.

ccopy.cc: simple file copy application writen in C++
dcopy.d:  simple file copy application writen in D
text.txt: a text file with ~4MB

Here are the results:

$ time ./ccopy text.txt test1
real    0m0.985s
user    0m0.854s
sys     0m0.047s

Less than one second to copy 4MB.

$ time ./dcopy text.txt test2
real    1m22.886s
user    0m25.014s
sys     0m57.011s
$

More than one minute to copy 4MB.

Here are the sources (both do the same thing: read and write char by char)

/////////////////////////////////////////
// ccopy.cpp
/////////////////////////////////////////
#include <iostream>
#include <fstream>
                                                                               
                                             
void error (const char* p, const char* p2=" ")
{
    std::cerr << p << ' ' << p2 << '\n';
    std::exit(1);
}
                                                                               
                                             
int main ( int argc, char** argv )
{
    if (argc != 3)
        error ("usage: copy SOURCE DESTINATION");
                                                                               
                                             
    std::ifstream origem (argv[1]);
    if (!origem) error ("error openning file ", argv[1]);
                                                                               
                                             
    std::ofstream destino (argv[2]);
    if (!destino) error ("error copying file ", argv[2]);
                                                                               
                                             
    char ch;
    while ( origem.get(ch) ) destino.put(ch);
                                                                               
                                             
    origem.close();
    destino.close();
}

//////////////////////////////////////////
// dcopy.d
//////////////////////////////////////////
import std.stream;
                                                                               
                                             
int main (char[][] args)
{
    if (args.length <= 2) {
        printf ("Usage: dcopy <source> <destiny>\n");
        goto EXIT;
    }
                                                                               
                                             
    char c;
    File source = new File (args[1]);
    File dest = new File ();
    dest.create (args[2]);
                                                                               
                                             
    while (!source.eof()) {
        dest.write (source.getc());
        //printf ("%c", source.getc());
    }
                                                                               
                                             
EXIT:
    return 0;
}


Thanks,

Bruno.

Jun 22 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
When I used BufferedFile the D code went twice as fast as the C++ code but
that probably is a function of the default buffer size. I don't know how big
the C++ buffer is, but in D it is 8K.
Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
class in stream.d. It needs a destructor that closes the stream:
 ~this(){ close(); }
Such a destructor is in File but not in Stream. I wonder if it should be
moved from File to Stream. Otherwise when I ran your code the final buffer
contents wouldn't be written to the output file.

-Ben

"Bruno A. Costa" <bruno codata.com.br> wrote in message
news:cb99vc$1uur$1 digitaldaemon.com...
 Hi,

 It seems to me that some parts of the std.stream module are working VERY
 slow. I've done some tests with file manipulation and some comparissions
 with C++ and the results are strange.

 ccopy.cc: simple file copy application writen in C++
 dcopy.d:  simple file copy application writen in D
 text.txt: a text file with ~4MB

 Here are the results:

 $ time ./ccopy text.txt test1
 real    0m0.985s
 user    0m0.854s
 sys     0m0.047s

 Less than one second to copy 4MB.

 $ time ./dcopy text.txt test2
 real    1m22.886s
 user    0m25.014s
 sys     0m57.011s
 $

 More than one minute to copy 4MB.

 Here are the sources (both do the same thing: read and write char by char)

 /////////////////////////////////////////
 // ccopy.cpp
 /////////////////////////////////////////
 #include <iostream>
 #include <fstream>

 void error (const char* p, const char* p2=" ")
 {
     std::cerr << p << ' ' << p2 << '\n';
     std::exit(1);
 }

 int main ( int argc, char** argv )
 {
     if (argc != 3)
         error ("usage: copy SOURCE DESTINATION");

     std::ifstream origem (argv[1]);
     if (!origem) error ("error openning file ", argv[1]);

     std::ofstream destino (argv[2]);
     if (!destino) error ("error copying file ", argv[2]);

     char ch;
     while ( origem.get(ch) ) destino.put(ch);

     origem.close();
     destino.close();
 }

 //////////////////////////////////////////
 // dcopy.d
 //////////////////////////////////////////
 import std.stream;

 int main (char[][] args)
 {
     if (args.length <= 2) {
         printf ("Usage: dcopy <source> <destiny>\n");
         goto EXIT;
     }

     char c;
     File source = new File (args[1]);
     File dest = new File ();
     dest.create (args[2]);

     while (!source.eof()) {
         dest.write (source.getc());
         //printf ("%c", source.getc());
     }

 EXIT:
     return 0;
 }


 Thanks,

 Bruno.

Jun 22 2004
parent "Bruno A. Costa" <bruno codata.com.br> writes:
Hi,

I switched from File to BufferedFile and it worked pretty well:
$ time ./dcopy text.txt teste9
real    0m0.848s
user    0m0.641s
sys     0m0.052s

Much Better. Thank you.

Bruno.

Ben Hinkle wrote:

 When I used BufferedFile the D code went twice as fast as the C++ code but
 that probably is a function of the default buffer size. I don't know how
 big the C++ buffer is, but in D it is 8K.
 Also I noticed a bug (well, a bummer if not a bug) in the BufferedStream
 class in stream.d. It needs a destructor that closes the stream:
  ~this(){ close(); }
 Such a destructor is in File but not in Stream. I wonder if it should be
 moved from File to Stream. Otherwise when I ran your code the final buffer
 contents wouldn't be written to the output file.
 
 -Ben
 
 "Bruno A. Costa" <bruno codata.com.br> wrote in message
 news:cb99vc$1uur$1 digitaldaemon.com...
 Hi,

 It seems to me that some parts of the std.stream module are working VERY
 slow. I've done some tests with file manipulation and some comparissions
 with C++ and the results are strange.


Jun 22 2004
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...
                                                                               
                                        
    char ch;
    while ( origem.get(ch) ) destino.put(ch);

By the way, this may be a tad faster than the above: destino << origem.rdbuf(); I'm planning on putting similar capability in stream.d, though it may not result in much of a speedup. Sean
Jun 22 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>




"Sean Kelly" <sean f4.ca> wrote in message
news:cb9vtv$2ma$1 digitaldaemon.com...
 In article <cb99vc$1uur$1 digitaldaemon.com>, Bruno A. Costa says...
    char ch;
    while ( origem.get(ch) ) destino.put(ch);

By the way, this may be a tad faster than the above: destino << origem.rdbuf(); I'm planning on putting similar capability in stream.d, though it may not

 in much of a speedup.


 Sean

Jun 22 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cba3cv$8l3$1 digitaldaemon.com>, Kris says...
Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>

Yes, I'm convinced, mango is brilliant. But much of my work involves writing libraries rather than applications, and there are arguments both for and against any one library being dependent upon any other. If some small portion of Deimos (for example) wanted to write to a file, would it be /fair/ to users of Deimos to require them also to download and link against Mango? Should libraries be independent of each other, or should at least the best of them become intertwined. Based on what I've seen so far, I reckon Mango ought to become /the/ streams standard for D, in which case, no problem - but at least, I need to ask the question. Arcane Jill
Jun 22 2004
parent reply "teqDruid (formerly DemmeGod)" <me teqdruid.com> writes:
Perhaps, in this case, it would be a better idea for part of mango to move
to deimos, and for mango to be dependent on deimos.  As I understand it,
mango is supposed to be a server library, however something like streams
should be in a standard library, like deimos.

John

On Tue, 22 Jun 2004 21:42:04 +0000, Arcane Jill wrote:

 In article <cba3cv$8l3$1 digitaldaemon.com>, Kris says...
Or use mango.io to do a

FileConduit in = new FileConduit (inpName, FileStyle.ReadExisting);
FileConduit out = new FileConduit (outName, FileStyle.WriteTruncate);

in.copyTo (out);

<g>

Yes, I'm convinced, mango is brilliant. But much of my work involves writing libraries rather than applications, and there are arguments both for and against any one library being dependent upon any other. If some small portion of Deimos (for example) wanted to write to a file, would it be /fair/ to users of Deimos to require them also to download and link against Mango? Should libraries be independent of each other, or should at least the best of them become intertwined. Based on what I've seen so far, I reckon Mango ought to become /the/ streams standard for D, in which case, no problem - but at least, I need to ask the question. Arcane Jill

Jun 22 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"teqDruid (formerly DemmeGod)"  wrote
 Perhaps, in this case, it would be a better idea for part of mango to move
 to deimos, and for mango to be dependent on deimos.  As I understand it,
 mango is supposed to be a server library, however something like streams
 should be in a standard library, like deimos.

Overall, Mango has a bent towards server-side processing but can be just as useful for client-side also. A case in point is mango.io, which is an independent package from the rest of the Mango family (there's a simple dependency list over here http://svn.dsource.org/svn/projects/mango/trunk/doc/html/index.html). Mango.io could indeed be placed into Demios without issue, if you folks feel that's the right home for it. However, the point about mixing portions of libraries is a good one: sometimes a library must expose some of it's lower layers (perhaps as method arguments, or delegate parameters), and if said lower layer uses mango.io rather than phobos.io then a user might become righteously indignant. Sometimes standards are a good thing, regardless of whatever faults they may have. One way to fix such faults is to change the standard, but doing so in a democratic manner is no easy task <g> On the other hand, it's not uncommon to see a grass-roots shift in practice once there's enough momentum behind a particular approach. Then it becomes easy to make a choice as to which dependency you wish to live with. Getting that initial momentum going is the hard part, regardless of any underlying perceived value. All that aside, I'm personally a big fan of plugable systems. Although the extend of an IO package and the performance factors involved can sometimes make that impractical. It can certainly be done effectively for isolated scenarios. - Kris
Jun 22 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>

Mango.io has some very nice design features and it may well be a candidate for Demios. I've been trying to get stream.d up to date mostly to have an alternate technique available for comparison. I'm interedted in seeing the pros and cons of both methods :) Sean
Jun 22 2004
next sibling parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Sean Kelly" wrote
 I've been trying to get stream.d up to date

Aren't you, Ben, and Walter far too busy doing other things ? <G>
Jun 22 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cbao56$17qa$1 digitaldaemon.com>, Kris says...
"Sean Kelly" wrote
 I've been trying to get stream.d up to date

Aren't you, Ben, and Walter far too busy doing other things ? <G>

Free time is overrated ;) Sean
Jun 22 2004
prev sibling parent John Reimer <brk_6502 NOSP_AM.yahoo.com> writes:
Sean Kelly wrote:

 Mango.io has some very nice design features and it may well be a candidate for
 Demios.  I've been trying to get stream.d up to date mostly to have an
alternate
 technique available for comparison.  I'm interedted in seeing the pros and cons
 of both methods :)
 
 Sean
 
 

My vote is to get mango.io into Demios. If anything's lacking, contributors certainly can start brainstorming some improvements from within Demios. It's an excellent base at the very least. Come on people, let's decide on this one. :-) Later, John
Jun 23 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>

While moving mango.io to Deimos is certainly possible, it wouldn't be my first choice. My first choice would be that mango.io is moved to *PHOBOS*. D absolutely requires a serious and well thought out streams package AS STANDARD. Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't imagine there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but obviously we accept that things have to be "out there in the field" for a while, getting a good testing, getting all the bugs fixed, and so on. You wouldn't want something in Phobos unless it had passed that test. But the thing about mango.io is - it has ALREADY passed that test. It has been out there, in the field, for some time now; all the major bugs have been found and fixed; no-one is planning any reorganization of mango.io in the near future; the class and function names are not going to change. In short, it is stable. What can I say? It's a question of power. Only Walter (so far as I know) has the power to move things to Phobos, but anyone can move things to Deimos. But maybe it's also a question of organization. Maybe mango.io could divorce itself from the rest of Mango and move to Deimos for now (in which case everything else in Deimos will be able to call it without worrying about requiring additional libraries, which would certainly be great). But Deimos itself is nothing special - it's just another library, and there are plenty of them around. True, it's a library full of things that would rather be in Phobos, but Mango already has credibility in its own right. On the other hand, I'd be reluntant to download/import/link against a whole server package, if all I wanted to do was to open a file. Let's be honest here - there are SERIOUS problems with std.stream. It is absurd that calling new File(filename, FileMode.Out) will not delete (truncate) an old file of the same name. It is absurd that, if you want to do this, you require two statements: f = new File(); f.create(...);. It is absurd that if you want to wrap this in a BufferedStream, you have to call create() /before/ you construct the buffered stream (because BufferedStream has no create() function). And it is pretty damn annoying that if you continually read from a BufferedStream which buffers a File, using the readLine() function, then there appears to be no way to distinguish between an empty line and end-of-file (they both return null, and no exceptions are thrown). Why is there no Stream.eof() function? Please - let's just replace it with something that JUST WORKS (and is actually documented). So, personally, I think Walter should simply make us all happy by adding mango.io to Phobos (possibly renaming it to std.io en route) and deprecating std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my second choice. Arcane Jill
Jun 23 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbc1t0$6h7$1 digitaldaemon.com>, Arcane Jill says...
Let's be honest here - there are SERIOUS problems with std.stream. It is absurd
that calling new File(filename, FileMode.Out) will not delete (truncate) an old
file of the same name. It is absurd that, if you want to do this, you require
two statements: f = new File(); f.create(...);. It is absurd that if you want to
wrap this in a BufferedStream, you have to call create() /before/ you construct
the buffered stream (because BufferedStream has no create() function). And it is
pretty damn annoying that if you continually read from a BufferedStream which
buffers a File, using the readLine() function, then there appears to be no way
to distinguish between an empty line and end-of-file (they both return null, and
no exceptions are thrown). Why is there no Stream.eof() function? Please - let's
just replace it with something that JUST WORKS (and is actually documented).

I'm in the process of fixing all of this... just got bogged down writing a real ANSI compliant version of scanf (in my hour or two of free coding time per day). I agree that Mango.io is quite nice, but I don't want it to be chosen simply because there's nothing else comparable. One thing I haven't addressed in stream.d and an area that Mango is also deficient is localized formatted i/o. This perhaps isn't a requirement for 1.0, but it would be nice to see some discussion of what we may need in this area among folks who know. Numeric formatting is pretty easy, but that still leaves character transcoding, doesn't it? Sean
Jun 23 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Sean Kelly" wrote
 One thing I haven't addressed in stream.d and an area that Mango is also
 deficient is localized formatted i/o.  This perhaps isn't a requirement

 but it would be nice to see some discussion of what we may need in this

 among folks who know.  Numeric formatting is pretty easy, but that still

 character transcoding, doesn't it?

I wholeheartedly agree. Didn't Ben post something about Perl formatting? I think the combination of what Ben mentioned together with char transcoding would be a great thing for any IO package. Note that mango.io is also completely deficient regarding unicode. There were too many unknowns at the time of writing, so it was left out until a more formal approach could be introduced. Thanks to Hauke and AJ, the latter appears to be happening (and mango.io will probably integrate their work as an additional flavour of Reader & Writer). In general, how would people like to see formatted IO operate? What about Ben's idea? How should the upcoming unicode library operate within formatted IO? Or should it?
Jun 23 2004
next sibling parent Gold Dragon <dragonwing dragonu.net> writes:
If you want my opinion (which you probably don't) then I pitch that the 
I/O for console and file operate like C++. With the exception of 
replacing the >> and << with ~ and damn the person who can't tell 
whether he is getting input or output. I believe that all the >> and << 
are doing is placing the stuff in a buffer anyway.

You could use modifiers like what C++ uses for formatting and it would 
be nice to have Perl Formatting. This perl formatting:
.	format $header =
.<<<<<<<<	<<<<<<<<	<<<<<<<
.$first, $second, $third
'.'

or something, it has been a while since I have done Perl but the general 
  idea is there. Since one of the problems with Perl's Formatting is 
that you have to init the variables before you use them. I pitch also 
that something like this is done:

.	classname ~ "<<<<<<<<" ~ "||||" ~ ">>>>>>>>>>>>>";

to set the up the formatting parameters and then for placing variables 
into the format:

.	classname ~ variable1 ~ variable2 ~ module::next ~ variable3 ~ variable4

If modifier 'next' is called then place the variables before it (if not 
before another 'next') into the same position else place each into a 
separate position. It would be better than having 'next' after all 
positions. The class should be smart enough to know what is happening. 
Or "Do What I Want" paradigm. I'm sure someone could think of a better 
way but here is my pitch: bow to or curse it.

 Kris wrote:
 In general, how would people like to see formatted IO operate? What about
 Ben's idea? How should the upcoming unicode library operate within formatted
 IO? Or should it?

Jun 23 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbcgcj$u8q$1 digitaldaemon.com>, Kris says...
"Sean Kelly" wrote
 Numeric formatting is pretty easy, but that still
leaves
 character transcoding, doesn't it?

I don't understand. What's character transcoding?
I wholeheartedly agree. Didn't Ben post something about Perl formatting? I
think the combination of what Ben mentioned together with char transcoding
would be a great thing for any IO package.

It's obviously important then. What is it?
Note that mango.io is also completely deficient regarding unicode. There
were too many unknowns at the time of writing, so it was left out until a
more formal approach could be introduced. Thanks to Hauke and AJ, the latter
appears to be happening (and mango.io will probably integrate their work as
an additional flavour of Reader & Writer).

In general, how would people like to see formatted IO operate? What about
Ben's idea? How should the upcoming unicode library operate within formatted
IO? Or should it?

Well, just to clarify what's "upcoming" on the Unicode front: In the next few days, functions to access all Unicode character properties. This is an important first step, because the Unicode algorithms are defined in terms of those properties. Sometime after that, normalization algorithms. This is slightly complicated in that, for completeness, we'd want it continue to work in the event of PUA (Private Use Area) customization. I was planning on stopping after that for a short break and going back to crypto (finish off the hash section). Presumably next on the list for Unicode should come the full casing algorithms - these are actually locale-sensitive (though only rarely, since all locales give identical results apart from "az", "lt" and "tr"). Simply put, the improvement you will see will be gradual. We started with functions like std.ctype.toupper(), defined only for ASCII characters; Hauke gave us utype.toupper() and friends, which extended that to all characters. Before long, we'll have a string toupper-ing function that does full casing. I'm not completely sure I understand the question "how would people like to see formatted IO operate?", or what it has to do with Unicode. Now, there are some really big and powerful algorithms described out there on the Unicode web site, and one of them concerns Unicode Regular Expressions - a staggeringly powerful concept - but probably the most difficult of all Unicode algorithms to implement in practice. You'll probably want that one, but alas I fear it will be one of the last to arrive. Anyway, if I'm way off track, forgive me. Much of localization has to do with replacing text strings (normally English) with a translation into some alternative language. This is usually done at the pre-stream level (in my experience), so by the time it gets to the stream, it's already in French/German/whatever. But there's obviously something I'm not getting here. Ah well. Put it down to my not being as bright as I'd like to be. Jill
Jun 23 2004
next sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <cbd3p3$1tef$1 digitaldaemon.com>, Arcane Jill says...
In article <cbcgcj$u8q$1 digitaldaemon.com>, Kris says...
"Sean Kelly" wrote
 Numeric formatting is pretty easy, but that still
leaves character transcoding, doesn't it?

I don't understand. What's character transcoding?

Say I has a string of dchars and I want to write it to a text file. Is it enough to just do an unformatted write of the string or does it have to be converted to UTF-16 or some such? ie. is there ever any difference between internal character representation and external representation that an i/o lib may care about?
I'm not completely sure I understand the question "how would people like to see
formatted IO operate?", or what it has to do with Unicode.

What I meant was any internationalization. As I mentioned, the issues I know of have to do with numeric formatting, but is there anything else? Sean
Jun 23 2004
next sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...
Say I has a string of dchars and I want to write it to a text file.  Is it
enough to just do an unformatted write of the string or does it have to be
converted to UTF-16 or some such?  ie. is there ever any difference between
internal character representation and external representation that an i/o lib
may care about?

Gotcha. Okay, I do know about that - it's just I've always called it ENcoding (and DEcoding) rather than TRANScoding - but I guess "transcoding" is just as correct, if not more correct. Yeah, well that sounds like a simple enough thing to do. You know that the encoding at one end (inside a D program) is going to be one of the Unicode UTF-s. Presumably, you have some information about the encoding at the other end (the console, a text window, a file, whatever), and is likely to be WINDOWS-1252 on Windows machines in western Europe, etc.. So all you have to do is convert from one to the other. My guess is, you'd want some sort of Transcoder class (see I've leant a new word), so you could do something like: # Transcoder t1 = new Transocoder("ISO-8859-1"); # Transcoder t2 = new Transocoder("SHIFT-JIS"); # Transcoder t3 = new Transocoder("MAC-ROMAN"); # Transcoder t4 = new Transocoder("ASCII"); and so on, and then use this either as a parameter to your stream constructor, or a parameter to a formatted input/output command. A transcoder would simply have to convert from a sequence of dchars to a sequence of ubytes (and vice versa). (Note that the names of encodings are universally understood - we don't have to make them up). Currently, of course, D assumes UTF-8 on the console, which is, in fact, usually wrong, which is why if you printf() a non-ASCII character to an MS-DOS command window, they come out wrong. A transcoder scheme would fix this.
I'm not completely sure I understand the question "how would people like to see
formatted IO operate?", or what it has to do with Unicode.

What I meant was any internationalization. As I mentioned, the issues I know of have to do with numeric formatting, but is there anything else? Sean

Jun 23 2004
parent reply Regan Heath <regan netwin.co.nz> writes:
On Thu, 24 Jun 2004 06:57:13 +0000 (UTC), Arcane Jill 
<Arcane_member pathlink.com> wrote:
 In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...
 Say I has a string of dchars and I want to write it to a text file.  Is 
 it
 enough to just do an unformatted write of the string or does it have to 
 be
 converted to UTF-16 or some such?  ie. is there ever any difference 
 between
 internal character representation and external representation that an 
 i/o lib
 may care about?

Gotcha. Okay, I do know about that - it's just I've always called it ENcoding (and DEcoding) rather than TRANScoding - but I guess "transcoding" is just as correct, if not more correct. Yeah, well that sounds like a simple enough thing to do. You know that the encoding at one end (inside a D program) is going to be one of the Unicode UTF-s. Presumably, you have some information about the encoding at the other end (the console, a text window, a file, whatever), and is likely to be WINDOWS-1252 on Windows machines in western Europe, etc.. So all you have to do is convert from one to the other. My guess is, you'd want some sort of Transcoder class (see I've leant a new word), so you could do something like: # Transcoder t1 = new Transocoder("ISO-8859-1"); # Transcoder t2 = new Transocoder("SHIFT-JIS"); # Transcoder t3 = new Transocoder("MAC-ROMAN"); # Transcoder t4 = new Transocoder("ASCII"); and so on, and then use this either as a parameter to your stream constructor, or a parameter to a formatted input/output command. A transcoder would simply have to convert from a sequence of dchars to a sequence of ubytes (and vice versa). (Note that the names of encodings are universally understood - we don't have to make them up). Currently, of course, D assumes UTF-8 on the console, which is, in fact, usually wrong, which is why if you printf() a non-ASCII character to an MS-DOS command window, they come out wrong. A transcoder scheme would fix this.

I always thought the best way to handle this sort of thing would be to have a pluggable modular system such that you'd have your input or 'source', your output or 'sink', and you could plug any number of transcoders in between them. So for example your source might be a file, and your sink a socket, you plug the transcoders for SSL, UTF-8 to Unicode, or anything else you want in, they all modify the data going thru them such that the data you get from the source arrives at the sink transcoded in those ways. I always wanted to try and write a stream library based on that idea... Regan. -- Using M2, Opera's revolutionary e-mail client: http://www.opera.com/m2/
Jun 24 2004
parent Sean Kelly <sean f4.ca> writes:
In article <opr93b3wzg5a2sq9 digitalmars.com>, Regan Heath says...
I always thought the best way to handle this sort of thing would be to 
have a pluggable modular system such that you'd have your input or 
'source', your output or 'sink', and you could plug any number of 
transcoders in between them.

So for example your source might be a file, and your sink a socket, you 
plug the transcoders for SSL, UTF-8 to Unicode, or anything else you want 
in, they all modify the data going thru them such that the data you get 
 from the source arrives at the sink transcoded in those ways.

I always wanted to try and write a stream library based on that idea...

I've written stream adaptors for this sort of thing. So I might chain together a GZip stream, a Base64 stream, and a socket stream and each would encode the data and push it on to the next layer. But for things like SSL this can get pretty complicated, as you may have to deal with password prompting and other configuration for that specific level. This is one area where Aspect oriented programming has a lot of promise. Sean
Jun 24 2004
prev sibling parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd6v9$21mu$1 digitaldaemon.com>, Sean Kelly says...

As I mentioned, the issues I know of
have to do with numeric formatting,

Unfortunately, Unicode won't help you with numeric formatting. As you know, the decimal point (for example) is encoded as U+002E if you're English, but as U+002C if you're French. At one point, I did suggest to the Consortium folk that they add an UNAMBIGUOUS DECIMAL POINT character, which would render as "." in an English font or "," in a French font, but they didn't like that idea. As far as the Unicode Consortium are concerned, Unicode is a repertoire of characters, not a repertoire of semantic meanings. Therefore, when it comes to this sort of localization issue, we're on our own. In a way, Java does this quite well - at least in the sense that it is sufficiently powerful. Where it falls down is in ease of use. It is not easy to do even simple localization in Java. I'm wondering if that's something else we're stuck with - it is even possible to make it BOTH easy AND powerful? Unfortunately, I only have questions on this one, not answers. Arcane Jill
Jun 24 2004
parent reply Sean Kelly <sean f4.ca> writes:
In article <cbe04d$6ri$1 digitaldaemon.com>, Arcane Jill says...
In a way, Java does this quite well - at least in the sense that it is
sufficiently powerful. Where it falls down is in ease of use. It is not easy to
do even simple localization in Java. I'm wondering if that's something else
we're stuck with - it is even possible to make it BOTH easy AND powerful?

Good question. C++ has a global locale setting and then IIRC you can associate a stream instance with a different locale. Formatting information is stored in a class or set of classes that defines the various separator characters and such. This is quite easy to use but fairly complicated to extend. But the basic idea does seem to work pretty well. How does Java work? Sean
Jun 24 2004
parent Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbesja$1for$1 digitaldaemon.com>, Sean Kelly says...
In article <cbe04d$6ri$1 digitaldaemon.com>, Arcane Jill says...
In a way, Java does this quite well - at least in the sense that it is
sufficiently powerful. Where it falls down is in ease of use. It is not easy to
do even simple localization in Java. I'm wondering if that's something else
we're stuck with - it is even possible to make it BOTH easy AND powerful?

Good question. C++ has a global locale setting and then IIRC you can associate a stream instance with a different locale. Formatting information is stored in a class or set of classes that defines the various separator characters and such. This is quite easy to use but fairly complicated to extend. But the basic idea does seem to work pretty well. How does Java work?

There are basically three big problems to solve in localization. Java solves all three. It doesn't necessarily have the BEST solutions, so we don't necessarily have to rip off the way Java does it, but we do need to solve the same three problems. These are: PROBLEM 1 - HOW TO DEFINE A LOCALE PROBLEM 2 - RESOURCE FILES PROBLEM 3 - INDEXED VARIADIC FUNCTIONS PROBLEM 1 - HOW TO DEFINE A LOCALE The problem is how to define PRECISELY what you mean by "locale". All other locale stuff will depend on this definition. Conceptually, a locale is something like "en-us", a string - but it's convenient not to have to deal with strings directly, because strings are slow, and you have to deal with issues of case, punctuation (hyphen verses underscore), and so on. Java solves this by having a class, Locale, which contains such strings, but it converts them to some standard internal format so it only has to do all that casing stuff ONCE. My own view is that this is too complicated. Especially when you have to deal with the three-letter ISO-639 / ISO-3166 codes (as opposed to the two-letter codes). It strikes me that a simple enum would suffice. Then you could do: # import std.locale; // Just pretend # # Language locale_lang = Language.EN; # Country locale_ctry = Country.US; Bingo. Problem solved. Now whenever you want to pass locale information you only have to pass one or two enum values. In the case of ISO-3166 (countries) the actual enum VALUES are even defined for us by the standard. ISO-639 (languages) defines only the names, so we'd have to make up the values. (And since emums will auto-cast into integers, you could templatize on them. I haven't thought through the implications of this). C++, of course, defines locales by its own internal means, without regard to any ISO standard. I don't recommend we copy this. Way too messy. PROBLEM 2 - RESOURCE FILES This is a very simple problem to define. Given a locale, open a file containing information relevant to that locale. For instance, you have an application, and in the course of execution, it prints stuff. You want it to print in English if the locale-language is Language.EN, French if the locale-language is Language.FR, and so on. So you open the "right" file (there will be one for each supported locale), read in all the text strings into an array, and then spit out the appropriate one at the the appropriate time. The problem? Where do you find the file? See - your app could have been installed anywhere on the user's filesystem. Same goes for your libraries. The current working directory could be anywhere. There is no "obvious" place to look. Traditionally, the Linux answer is to get the "right place to look" from an environment variable. But then you have the problem of environment variable name clashes. Traditionally, the Windows answer is to get the "right place to look" from the registry. No problem with name clashes here, but now you have a different problem - it takes about fifteen thousand lines of code to do the equivalent of getenv(). Enter Java's magnificent solution. Java has a class, ResourceBundle, which does the job. There are two forms, if memory serves correct. In one version, it parses a human-readable text file into a class. In the other version, it deserializes an arbitrary serialized object into a class. But either way, it knows where to look by SEARCHING THE JAVA CLASSPATH. What this means, in simple terms, is that source code is able to locate the file RELATIVE TO THE SOURCE FILE. And this alone is staggeringly powerful. In D terms - you could put a file in the same directory as a source file, and just OPEN IT. Or, if it were in a different directory (in an associated package, say) which could be located RELATIVE TO THE SOURCE FILE then, again, that's problem solved. However, this is a difficult thing to do in D, because there is no classpath. We may have to come up with other solutions to this one. PROBLEM 3 - INDEXED VARIADIC FUNCTIONS Once you've got your strings and your formatting classes, you're very nearly sorted. Just one problem remains - word order. Consider - this will work in standard English: # char[] a = "the server"; # char[] b = "running"; # char[] message = "%.*s is %.*s"; # printf(&message[0], a, b); Prints: the server is running Assume that the first three lines could be substituted with fetching the values from a resource file or some other means, but the fourth line actually a part of your program. Trouble is, it won't work with all languages, because of word order. On planet Degobar (where Yoda comes from), you'd want to say "running, the server is" (rather than "the server is running"), so that printf just won't do. It can't change the word order. Java has SOMETHING LIKE printf for this purpose. It takes a format string containing escape sequences such as %1, %2, %3, and so on, and the numbers tell you WHICH string you're referring to. (This is cumbersome in Java, because you have to use an array). But, basically, it lets you do: # char[] message = "%1 is %2"; // Standard English # char[] message = "%2, %1 is"; // Yoda-speak The printf-like function can now plug the RIGHT values into the right places. Bingo - one string, correctly formatted for an arbitrary locale. I understand that Walter is working on a D-printf() right now. If so, Walter, if you could give us some means of accessing the /nth/ argument (instead of always the /next/ argument), then that problem would be solved immediately. Arcane Jill
Jun 25 2004
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
Just thought of another one.  What constitutes whitespace is locale dependent as
well.  So add that to the localization list.
Jun 23 2004
parent reply Arcane Jill <Arcane_member pathlink.com> writes:
In article <cbd7ac$2284$1 digitaldaemon.com>, Sean Kelly says...
Just thought of another one.  What constitutes whitespace is locale dependent as
well.  So add that to the localization list.

Well, this is not true in Unicode, so - however you define that - you will have to do it without the aid a Unicode Library. Whitespace according to Unicode is defined by the property file http://www.unicode.org/Public/UNIDATA/PropList.txt. The following characters - and ONLY the following characters, are deemed to be whitespace according to Unicode: 0009..000D ; White_Space # Cc [5] <control-0009>..<control-000D> 0020 ; White_Space # Zs SPACE 0085 ; White_Space # Cc <control-0085> 00A0 ; White_Space # Zs NO-BREAK SPACE 1680 ; White_Space # Zs OGHAM SPACE MARK 180E ; White_Space # Zs MONGOLIAN VOWEL SEPARATOR 2000..200A ; White_Space # Zs [11] EN QUAD..HAIR SPACE 2028 ; White_Space # Zl LINE SEPARATOR 2029 ; White_Space # Zp PARAGRAPH SEPARATOR 202F ; White_Space # Zs NARROW NO-BREAK SPACE 205F ; White_Space # Zs MEDIUM MATHEMATICAL SPACE 3000 ; White_Space # Zs IDEOGRAPHIC SPACE My personal opinion (for what it's worth) is that this does not need to be localized. Maybe if a particular application (such as a D compiler, for example) required that (for example) U+00A0 be defined as non-whitespace, then the application could simply work around that - I believe Java does something similar, defining a "Java whitespace" with a slightly different list which excludes U+00A0. But as for localizing for a locale, my personal opinion is that Unicode's definition is good enough. If you're going to use a different list, then you're not using Unicode! Part of the whole point of Unicode is to ease localization issues by simply eliminating them wherever possible. ONE standard. So far as I can tell, the argument that "what is whitespace is locale-dependent" is 8-bit-standard-thinking, and we simply don't need to use that any more. Arcane Jill
Jun 23 2004
parent Sean Kelly <sean f4.ca> writes:
Works for me.  I'll admit that most of what I know of localization has come from
C++, which hasn't quite embraced unicode yet :)

Sean
Jun 24 2004
prev sibling next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cbc1t0$6h7$1 digitaldaemon.com...
 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks


that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's


layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless


whatever faults they may have. One way to fix such faults is to change


standard, but doing so in a democratic manner is no easy task <g>

While moving mango.io to Deimos is certainly possible, it wouldn't be my

 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS

 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't

 there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but
 obviously we accept that things have to be "out there in the field" for a

 getting a good testing, getting all the bugs fixed, and so on. You

 something in Phobos unless it had passed that test. But the thing about

 is - it has ALREADY passed that test. It has been out there, in the field,

 some time now; all the major bugs have been found and fixed; no-one is

 any reorganization of mango.io in the near future; the class and function

 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could

 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But

 itself is nothing special - it's just another library, and there are

 them around. True, it's a library full of things that would rather be in

 but Mango already has credibility in its own right. On the other hand, I'd

 reluntant to download/import/link against a whole server package, if all I
 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream.

I don't know why std.stream is such a whipping boy, but I'll throw in my 2 cents:
 It is absurd
 that calling new File(filename, FileMode.Out) will not delete (truncate)

 file of the same name. It is absurd that, if you want to do this, you

 two statements: f = new File(); f.create(...);.

seems easy enough to fix. I haven't played around with possible solutions but it would require changing just a few lines of code - possibly adding an optional input to the constructor to truncate or another FileMode or something.
 It is absurd that if you want to
 wrap this in a BufferedStream, you have to call create() /before/ you

 the buffered stream (because BufferedStream has no create() function).

I'm not sure what a BufferedStream create() function would do. Do you mean BufferedFile?
 And it is
 pretty damn annoying that if you continually read from a BufferedStream

 buffers a File, using the readLine() function, then there appears to be no

 to distinguish between an empty line and end-of-file (they both return

 no exceptions are thrown). Why is there no Stream.eof() function?

Stream.eof() exists.
 Please - let's
 just replace it with something that JUST WORKS (and is actually

Improving the doc would be nice.
 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and

 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my
 second choice.

 Arcane Jill

Jun 23 2004
parent "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Ben Hinkle" wrote
 I don't know why std.stream is such a whipping boy

I hope that's not the case, but the two certainly have alternate approaches. I don't understand it, but that can sometimes cause heated debate on this NG :-) Personally, I feel std.stream and mango.io are very different animals ~ as such it's hard to compare and contrast. Perhaps it might be useful to clarify what some of the notions are/were behind mango.io, such that any further discussion might be more finely tuned?
Jun 23 2004
prev sibling parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
My opinion is that neither should be in the 1.0 standard library, so we have
freedom to have a proper competitive, tested, comprehensive implementation for
2.0, which will be maximally efficient, usable, comprehensible and robust. The
C++ streams library only manages 1.5 / 4 there, and that took a long time to
develop.

The same goes for GUI.

It may be that Mango ends up being the standard streams for Phobos, or
std.stream, or a refactored of std.stream, or something entirely new.

Since std.stream is in there, I don't expect it to be taken out. But I do think
that the above open-mindedness should be manifest in the community, and clearly
expressed to all users, especially new ones.

D has the faculties to mean that we have the potential to have the best IO
libraries in the C-family world. I hope it will be realised.



"Arcane Jill" <Arcane_member pathlink.com> wrote in message
news:cbc1t0$6h7$1 digitaldaemon.com...
 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks feel
that's the right home for it. However, the point about mixing portions of
libraries is a good one: sometimes a library must expose some of it's lower
layers (perhaps as method arguments, or delegate parameters), and if said
lower layer uses mango.io rather than phobos.io then a user might become
righteously indignant. Sometimes standards are a good thing, regardless of
whatever faults they may have. One way to fix such faults is to change the
standard, but doing so in a democratic manner is no easy task <g>

While moving mango.io to Deimos is certainly possible, it wouldn't be my first choice. My first choice would be that mango.io is moved to *PHOBOS*. D absolutely requires a serious and well thought out streams package AS STANDARD. Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't imagine there's anything in Deimos/etc that wouldn't rather be in Phobos/std, but obviously we accept that things have to be "out there in the field" for a

 getting a good testing, getting all the bugs fixed, and so on. You wouldn't

 something in Phobos unless it had passed that test. But the thing about

 is - it has ALREADY passed that test. It has been out there, in the field, for
 some time now; all the major bugs have been found and fixed; no-one is planning
 any reorganization of mango.io in the near future; the class and function names
 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know) has

 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could divorce
 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But Deimos
 itself is nothing special - it's just another library, and there are plenty of
 them around. True, it's a library full of things that would rather be in

 but Mango already has credibility in its own right. On the other hand, I'd be
 reluntant to download/import/link against a whole server package, if all I
 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is absurd
 that calling new File(filename, FileMode.Out) will not delete (truncate) an old
 file of the same name. It is absurd that, if you want to do this, you require
 two statements: f = new File(); f.create(...);. It is absurd that if you want

 wrap this in a BufferedStream, you have to call create() /before/ you construct
 the buffered stream (because BufferedStream has no create() function). And it

 pretty damn annoying that if you continually read from a BufferedStream which
 buffers a File, using the readLine() function, then there appears to be no way
 to distinguish between an empty line and end-of-file (they both return null,

 no exceptions are thrown). Why is there no Stream.eof() function? Please -

 just replace it with something that JUST WORKS (and is actually documented).

 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and deprecating
 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be my
 second choice.

 Arcane Jill

Jun 23 2004
next sibling parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Matthew" wrote ...
 My opinion is that neither should be in the 1.0 standard library, so we

 freedom to have a proper competitive, tested, comprehensive implementation

 2.0, which will be maximally efficient, usable, comprehensible and robust.

I couldn't agree more Matthew. However, the fact that std.stream is already in phobos gives it a decided advantage. That means the playing field is, unfortunately, not as you idealize it above. Nor will it be regarding DTL, since you already have a percieved advantage via your connection with Walter (obviated by the digitalmars.D.dtl NG). Don't get me wrong ~ there's no sour grapes here. Just making a point that reality is rarely unbiased. Let's not give anyone the wrong impression.
 C++ streams library only manages 1.5 / 4 there, and that took a long time

 develop.

<black humour> As you say; the C++ streams library did indeed take a long time to develop. How long does it usually take to admit that a fundamental approach is tragically flawed? (sorry; sorry ... just a bad joke) </black humour>
 It may be that Mango ends up being the standard streams for Phobos, or
 std.stream, or a refactored of std.stream, or something entirely new.

*mango.io* is but one small independent branch of the Mango Tree, Matthew <g> Perhaps we ought to rename it yet again. Also, as noted elsewhere, mango.io and std.stream are comparable on a superficial basis only. Although they both clearly perform IO of some kind, that's perhaps the extent of their relationship. Where the Phobos library has many vaguely related, and somewhat fractured modules related to IO, mango.io tries hard to be fairly cohesive. Perhaps that is the primary reason why those who use it like it as much as they do? I hope you'll like it too, when you start disseminating the mango.servlet package :-) For the record, let me state that I'm not personlly advocating mango.io should be in Phobos. Someone please correct me if I'm wrong, but I've noted only that I wish it to be reviewed (and perhaps ratified) by the oft-discussed but non-existent "D standard library group committee", along with other members of the Mango Tree family (see old NG for details on the "committee"). The last thing I want (or need) is for mango.io and std.stream to become political footballs; that would be entirely unhelpful to the whole D community. How about that darned DSLG committee? Lars? Walter? - Kris
 Since std.stream is in there, I don't expect it to be taken out. But I do

 that the above open-mindedness should be manifest in the community, and

 expressed to all users, especially new ones.

 D has the faculties to mean that we have the potential to have the best IO
 libraries in the C-family world. I hope it will be realised.



 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cbc1t0$6h7$1 digitaldaemon.com...
 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks



that's the right home for it. However, the point about mixing portions



libraries is a good one: sometimes a library must expose some of it's



layers (perhaps as method arguments, or delegate parameters), and if



lower layer uses mango.io rather than phobos.io then a user might



righteously indignant. Sometimes standards are a good thing, regardless



whatever faults they may have. One way to fix such faults is to change



standard, but doing so in a democratic manner is no easy task <g>

While moving mango.io to Deimos is certainly possible, it wouldn't be my


 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS


 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't


 there's anything in Deimos/etc that wouldn't rather be in Phobos/std,


 obviously we accept that things have to be "out there in the field" for


 while,
 getting a good testing, getting all the bugs fixed, and so on. You


 want
 something in Phobos unless it had passed that test. But the thing about

 is - it has ALREADY passed that test. It has been out there, in the


 some time now; all the major bugs have been found and fixed; no-one is


 any reorganization of mango.io in the near future; the class and


 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)


 the
 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could


 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But


 itself is nothing special - it's just another library, and there are


 them around. True, it's a library full of things that would rather be in

 but Mango already has credibility in its own right. On the other hand,


 reluntant to download/import/link against a whole server package, if all


 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is


 that calling new File(filename, FileMode.Out) will not delete (truncate)


 file of the same name. It is absurd that, if you want to do this, you


 two statements: f = new File(); f.create(...);. It is absurd that if you


 to
 wrap this in a BufferedStream, you have to call create() /before/ you


 the buffered stream (because BufferedStream has no create() function).


 is
 pretty damn annoying that if you continually read from a BufferedStream


 buffers a File, using the readLine() function, then there appears to be


 to distinguish between an empty line and end-of-file (they both return


 and
 no exceptions are thrown). Why is there no Stream.eof() function?


 let's
 just replace it with something that JUST WORKS (and is actually


 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and


 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be


 second choice.

 Arcane Jill


Jun 23 2004
parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
 "Matthew" wrote ...
 My opinion is that neither should be in the 1.0 standard library, so we

 freedom to have a proper competitive, tested, comprehensive implementation

 2.0, which will be maximally efficient, usable, comprehensible and robust.

I couldn't agree more Matthew. However, the fact that std.stream is already in phobos gives it a decided advantage. That means the playing field is, unfortunately, not as you idealize it above.

True. But if std.stream does not cut the mustard, I for one will be voicing for change. I think streaming, and especially object instance serialisation/marshalling/streaming/pickling/whatever-you-want-to-call-it, will be a feature that D *must* do right, in order to curtail comparisons with Java, Python and other languages that do this well on at least one of the four criteria on which I've scored C++ 1.5. I totally agree with Walter that this is something that 1.0 can't wait for. It's also something that will require a very large amount of design, development and refinement, including large amounts of Walter's time - as it'll probably have to involve language as well as library - and lots of input from people with different backgrounds, to get it 100% correct correct. (Well, 95%; there are no 100s.)
 Nor will it be regarding DTL,
 since you already have a percieved advantage via your connection with Walter

Maybe there's an advantage there. But any "connection" with Walter stems from being enthusiastic, and putting as much effort into writing code as into writing NG posts. It's not really anything more than that. (Of course, maybe when our book's more corporeal than corporate schpiel we'll have an irrevocable connection. Pity Walter from that point on. <g>) Regarding DTL. I have had what I think is a good idea, and Walter has agreed thus far. But a combination of compiler bugs (which I've yet to distill) and an all-consuming workload has meant that nothing's really happened from the initial implementation in March. I intend and expect to rectify that by the end of July. But there's a big gap in time. If someone else had a grand idea for DTL then they've had plenty of time to develop/refine it. I've said all along that DTL-MW is not a done deal, and that others should persue alternatives if they have them, and we can evaluate *all* options on their merits.
 (obviated by the digitalmars.D.dtl NG).

<pedantic>"obviated"? I think you mean "exemplified". Obviate means to render unnecessary</pedantic>
 Don't get me wrong ~ there's no sour
 grapes here.

Kris. You don't need to say that. I know you know that you have my undying respect ...
 Just making a point that reality is rarely unbiased. Let's not
 give anyone the wrong impression.

Again. Yes and no. Despite the occasional heat, I think just about everyone here is more interested in D being really good, than in just getting their own opinions swallowed. (At least I hope so!) So I think right will out in most cases, especially the big ones.
 C++ streams library only manages 1.5 / 4 there, and that took a long time

 develop.

<black humour> As you say; the C++ streams library did indeed take a long time to develop. How long does it usually take to admit that a fundamental approach is tragically flawed? (sorry; sorry ... just a bad joke) </black humour>

Well, Java's been with us for nearly a decade ... To be serious, for all that I really don't care for the IOStreams, they do have the localisation issue relatively well handled. That's not something to sniff at, even though it does have high performance costs.
 It may be that Mango ends up being the standard streams for Phobos, or
 std.stream, or a refactored of std.stream, or something entirely new.

*mango.io* is but one small independent branch of the Mango Tree, Matthew <g> Perhaps we ought to rename it yet again. Also, as noted elsewhere, mango.io and std.stream are comparable on a superficial basis only. Although they both clearly perform IO of some kind, that's perhaps the extent of their relationship. Where the Phobos library has many vaguely related, and somewhat fractured modules related to IO, mango.io tries hard to be fairly cohesive. Perhaps that is the primary reason why those who use it like it as much as they do?

I know, I know. Apologies for my abuse: loose use of language. I'm a goose. I'll try and spruce it up, be less abstruse.
 I hope you'll like it too, when you start disseminating the
 mango.servlet package :-)

Yes. I've promised to do that July, and I'll keep it.
 For the record, let me state that I'm not personlly advocating mango.io
 should be in Phobos. Someone please correct me if I'm wrong, but I've noted
 only that I wish it to be reviewed (and perhaps ratified) by the
 oft-discussed but non-existent "D standard library group committee", along
 with other members of the Mango Tree family (see old NG for details on the
 "committee"). The last thing I want (or need) is for mango.io and std.stream
 to become political footballs; that would be entirely unhelpful to the whole
 D community.

Nope. AFAICR, you've not said that. In any case, playing football with a mango would be a huge waste. Mangos must be respectfully, even reverently, consumed.
 How about that darned DSLG committee? Lars? Walter?

 - Kris


 Since std.stream is in there, I don't expect it to be taken out. But I do

 that the above open-mindedness should be manifest in the community, and

 expressed to all users, especially new ones.

 D has the faculties to mean that we have the potential to have the best IO
 libraries in the C-family world. I hope it will be realised.



 "Arcane Jill" <Arcane_member pathlink.com> wrote in message
 news:cbc1t0$6h7$1 digitaldaemon.com...
 In article <cbaj13$10pl$1 digitaldaemon.com>, Kris says...
Mango.io could indeed be placed into Demios without issue, if you folks



that's the right home for it. However, the point about mixing portions



libraries is a good one: sometimes a library must expose some of it's



layers (perhaps as method arguments, or delegate parameters), and if



lower layer uses mango.io rather than phobos.io then a user might



righteously indignant. Sometimes standards are a good thing, regardless



whatever faults they may have. One way to fix such faults is to change



standard, but doing so in a democratic manner is no easy task <g>

While moving mango.io to Deimos is certainly possible, it wouldn't be my


 choice. My first choice would be that mango.io is moved to *PHOBOS*. D
 absolutely requires a serious and well thought out streams package AS


 Now, Deimos is kind of a testing ground for Phobos-wannabes. I don't


 there's anything in Deimos/etc that wouldn't rather be in Phobos/std,


 obviously we accept that things have to be "out there in the field" for


 while,
 getting a good testing, getting all the bugs fixed, and so on. You


 want
 something in Phobos unless it had passed that test. But the thing about

 is - it has ALREADY passed that test. It has been out there, in the


 some time now; all the major bugs have been found and fixed; no-one is


 any reorganization of mango.io in the near future; the class and


 are not going to change. In short, it is stable.

 What can I say? It's a question of power. Only Walter (so far as I know)


 the
 power to move things to Phobos, but anyone can move things to Deimos.

 But maybe it's also a question of organization. Maybe mango.io could


 itself from the rest of Mango and move to Deimos for now (in which case
 everything else in Deimos will be able to call it without worrying about
 requiring additional libraries, which would certainly be great). But


 itself is nothing special - it's just another library, and there are


 them around. True, it's a library full of things that would rather be in

 but Mango already has credibility in its own right. On the other hand,


 reluntant to download/import/link against a whole server package, if all


 wanted to do was to open a file.

 Let's be honest here - there are SERIOUS problems with std.stream. It is


 that calling new File(filename, FileMode.Out) will not delete (truncate)


 file of the same name. It is absurd that, if you want to do this, you


 two statements: f = new File(); f.create(...);. It is absurd that if you


 to
 wrap this in a BufferedStream, you have to call create() /before/ you


 the buffered stream (because BufferedStream has no create() function).


 is
 pretty damn annoying that if you continually read from a BufferedStream


 buffers a File, using the readLine() function, then there appears to be


 to distinguish between an empty line and end-of-file (they both return


 and
 no exceptions are thrown). Why is there no Stream.eof() function?


 let's
 just replace it with something that JUST WORKS (and is actually


 So, personally, I think Walter should simply make us all happy by adding
 mango.io to Phobos (possibly renaming it to std.io en route) and


 std.stream. Failing that, moving mango.io to Deimos (etc.io ?) would be


 second choice.

 Arcane Jill



Jun 23 2004
parent reply "Kris" <someidiot earthlink.dot.dot.dot.net> writes:
"Matthew" wrote
 I think streaming, and especially object instance
 serialisation/marshalling/streaming/pickling/whatever-you-want-to-call-it,
 will be a feature that D *must* do right, in order to curtail comparisons
 with Java, Python and other languages that do this well on at least one
 of the four criteria on which I've scored C++ 1.5

Ah, pickling! There's been so many times we've worked around the predefined Java serialization that I've had team members use an alternative. Especially after tracing just how inefficient it is; not to mention arcane. The Serializable interface is cool, but the hidden methods (private yet somehow public) to intercept default processing are, uhhhh, funky. Then they had to extend the language with keywords such as transient ... it's a solution that looks immediately viable, yet soon runs out of steam. But it works, which is the main thing. It may be naive of me to think that explicitly coding which members of a class are pickled is the better way to go, provided there's an easy, flexible and consistent framework to operate within. But that's where my thoughts lie ~ IMO, v1.0 should absolutely not wait for this to be ground out.
 Maybe there's an advantage there. But any "connection" with Walter stems
 from being enthusiastic, and putting as much effort into writing code as

 writing NG posts. It's not really anything more than that.

If that's all it takes, the myriad projects at dsource.org should be given an equal level of "connection" then! Can they all have sections on the NG too please! <g>
 <pedantic>"obviated"? I think you mean "exemplified". Obviate means to
 render unnecessary</pedantic>

Yeah; it's a bit annoying when one changes the wording of something, and then can't remedy errors after posting.
 <black humour>
 As you say; the C++ streams library did indeed take a long time to


 How long does it usually take to admit that a fundamental approach is
 tragically flawed? (sorry; sorry ... just a bad joke)
 </black humour>

Well, Java's been with us for nearly a decade ...

Too true. Even Bruce Eckel could barely contain disparaging remarks regarding Java IO in his truly excellent book (TIJ). It just goes to show that such things shouldn't be hurried into a standard library for a v1.0 release (re std.streams); poor Java has paid for it in spades (and hacks) ever since, due to backward compatibility. Even NIO is lumbered.
 To be serious, for all that I really don't care for the IOStreams, they do
 have the localisation issue relatively well handled. That's not something
 to sniff at, even though it does have high performance costs.

I think it would be really helpful to expound on this particular note over in that "IO formatting" thread. How about it Matthew? I, for one, would likely learn something anew.
 In any case, playing football with a mango would be a huge waste. Mangos
 must be respectfully, even reverently, consumed.

How about Australian Rugby then?
Jun 23 2004
parent Sean Kelly <sean f4.ca> writes:
In article <cbdfjg$2e4b$1 digitaldaemon.com>, Kris says...
"Matthew" wrote
 Well, Java's been with us for nearly a decade ...

Too true. Even Bruce Eckel could barely contain disparaging remarks regarding Java IO in his truly excellent book (TIJ). It just goes to show that such things shouldn't be hurried into a standard library for a v1.0 release (re std.streams); poor Java has paid for it in spades (and hacks) ever since, due to backward compatibility.

And don't forget the Java GUI API, which was changed completely between 1.0 and 2.0. I think basic i/o is important to have for 1.0, but I'd much rather have the fancy stuff come later than be stuck supporting bad decisions. Sean
Jun 23 2004
prev sibling parent reply John Reimer <brk_6502 NOSP_AM.yahoo.com> writes:
Mango.io may not be ready to go into the D std package as of yet; any 
discussion of the sort might be considered presumptuous.  But I still 
think it should go somewhere where people will give it serious thought. 
  Of late, my general feeling is that mango.io is not getting the chance 
in needs to make an impression on the D community. To improve this maybe 
these ideas should be considered:

1) A web page could be set up for listing several D stream packages for 
D community perusal

2) the page could have official endorsement from the www.digitalmars.com 
site to encourage people to sample, comment, and peruse the competing 
stream packages.

3) Perhaps that web page should indicate: "D Stream Packages to be 
considered for official D Phobos integration"

4) Each package should be well documented and easily extensible... 
(oops.. that just eliminated a few, didn't it?)  ;-)

5) Each package should get a review in some journal, perhaps even in the 
new journal that Walter mentioned in C++.announce.

Most of these ideas depend on how serious the D community is about 
seeing a successful stream library appear.  If we are all that serious, 
then we should be pushing for better publicity for the existing packages.

I really want to see Mango.io given a fair chance.  At the very least, 
mango.io should get consideration for inclusion in the more publically 
developed demios.

Later,

John
Jun 23 2004
parent reply =?ISO-8859-1?Q?Julio_C=E9sar_Carrascal_Urquijo?= writes:
John Reimer wrote:
 (...)
 5) Each package should get a review in some journal, perhaps even in the 
 new journal that Walter mentioned in C++.announce.
 (...)

6) Included in the compiler distribution. -- Julio CÚsar Carrascal Urquijo
Jun 24 2004
parent John Reimer <jjreimer telus.net> writes:
Julio CÚsar Carrascal Urquijo wrote:

 John Reimer wrote:
 (...)
 5) Each package should get a review in some journal, perhaps even in the
 new journal that Walter mentioned in C++.announce.
 (...)

6) Included in the compiler distribution.

Good point!
Jun 24 2004