www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Read Complete File to Array of Lines

reply "Paul" <phshaffer gmail.com> writes:
I would like to read a complete file in one statement and then 
process it line by line.

foreach (line; MyFile)
etc.

Is it possible to read a file into and array of lines?
Thanks
May 11 2012
next sibling parent sclytrack <sclytrack iq87.fr> writes:
On 05/11/2012 05:00 PM, Paul wrote:

 I would like to read a complete file in one statement and then process
 it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks





---SOURCE
import std.stdio;
import std.file;

int main()
{
	writeln("start");
	foreach( line; File("src/main.d").byLine())
	{
		writeln(line);
	}
	writeln("stop");
	return 0;
}

---OUTPUT
start
import std.stdio;
import std.file;

int main()
{
	writeln("start");
	foreach( line; File("src/main.d").byLine())
	{
		writeln(line);
	}
	writeln("stop");
	return 0;
}
stop
May 11 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 11, 2012 at 05:00:16PM +0200, Paul wrote:

 I would like to read a complete file in one statement and then
 process it line by line.
 
 foreach (line; MyFile)
 etc.
 
 Is it possible to read a file into and array of lines?



import std.array;
import std.stdio;

string[] getLines(File f) {
	auto a = appender!string;
	foreach (line; f.byLine()) {
		a.put(line);
	}
	return a.data;
}

void main(string[] args) {
	if (args > 1) {
		auto f = File(args[0]);
		auto lines = getLines(f);
	}
}


T

-- 
"How are you doing?" "Doing what?"
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 15:18:11 UTC, H. S. Teoh wrote:

 On Fri, May 11, 2012 at 05:00:16PM +0200, Paul wrote:

 I would like to read a complete file in one statement and then
 process it line by line.
 
 foreach (line; MyFile)
 etc.
 
 Is it possible to read a file into and array of lines?



 import std.array;
 import std.stdio;

 string[] getLines(File f) {
 	auto a = appender!string;
 	foreach (line; f.byLine()) {
 		a.put(line);
 	}
 	return a.data;
 }



Does that work? I think you mean:

string[] getLines(File f) {
   auto a = appender!(string[]);
   foreach (line; f.byLine()) {
     a.put(line.idup);
   }
   return a.data;
}

You could also write:

string[] getLines(File f)
{
   return array(map!"a.idup"(f.byLine));
}

Or

{
   return f.byLine.map!"a.idup".array;
}


This may be slower than the appender version, which is optimized 
for performance; I'm not sure.

Graham
May 11 2012
prev sibling next sibling parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:

 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



Something like:

import std.file;
import std.string;

void main() {
      foreach(line; readText("file.in").splitLines()) ...
}
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:

 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



If you use the "byLine" approach...

   foreach(line; File("myfile").byLine)) { ... }

or

   File("myfile").byLine.doSomethingWithLines()

...keep in mind that "byLine", as it traverses your file, reuses 
a single buffer to store the current line in. If you're storing 
those lines somewhere for later use, use "line.dup" or 
"line.idup" to make a copy of that buffer. (Or use a different 
approach, like readText/splitLines.)

That was one of the first "gotcha" moments I had with D. :)

Graham
May 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> wrote:


 I would like to read a complete file in one statement and then process  
 it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



Would something like this work?

auto arr = array(map!"a.idup"(File("file.txt").byLine()));

-Steve
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 18:57:52 UTC, Steven Schveighoffer 
wrote:

 On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> 
 wrote:


 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



 Would something like this work?

 auto arr = array(map!"a.idup"(File("file.txt").byLine()));



It sure would. I suspect that Jesse's approach...

   readText("file.in").splitLines()

...would be the most efficient way if you need an actual array: 
slurp the whole file at once, then create an array of 
memory-sharing slices.

I look forward to the great std.stdio/std.file unification of 
201x, when I won't have to look in two modules for file-reading 
functions. :)

Graham
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:

 It sure would. I suspect that Jesse's approach...

   readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual array: 
 slurp the whole file at once, then create an array of 
 memory-sharing slices.



  Doesn't sound hard.. I could likely write a quick line splitting 
ranges fairly quickly, assuming \r and \n are the newlines, if 
there's new ones I'm unaware of then it may not work quite as 
well as you want :P

  Let's see....
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 20:06:45 UTC, Era Scarecrow wrote:

 On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:

 It sure would. I suspect that Jesse's approach...

  readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual  
 array: slurp the whole file at once, then create an array of  
 memory-sharing slices.






   Actually re-reading that and glancing at splitLines, I see it's 
already done... Mmmm... ignore my post(s) I guess.
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 20:06:45 UTC, Era Scarecrow wrote:

 On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:

 It sure would. I suspect that Jesse's approach...

  readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual 
 array: slurp the whole file at once, then create an array of 
 memory-sharing slices.



  Doesn't sound hard.. I could likely write a quick line 
 splitting ranges fairly quickly, assuming \r and \n are the 
 newlines, if there's new ones I'm unaware of then it may not 
 work quite as well as you want :P

  Let's see....



Hold on! It already exists:

std.file.readText
std.string.splitLines

Graham
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:

 On Friday, 11 May 2012 at 18:57:52 UTC, Steven Schveighoffer 
 wrote:

 On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> 
 wrote:


 I would like to read a complete file in one statement and 
 then process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



 Would something like this work?

 auto arr = array(map!"a.idup"(File("file.txt").byLine()));



 It sure would. I suspect that Jesse's approach...

   readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual array: 
 slurp the whole file at once, then create an array of 
 memory-sharing slices.

 I look forward to the great std.stdio/std.file unification of 
 201x, when I won't have to look in two modules for file-reading 
 functions. :)

 Graham



question?
I will try the readText.splitLines().  Thanks to all!
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:

 On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:

 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks



 Something like:

 import std.file;
 import std.string;

 void main() {
      foreach(line; readText("file.in").splitLines()) ...
 }



Thanks Jesse.
I'm finding that I can't just substitute args[1] for a text 
string.  Is there a clever way to use args[] in place of your 
"file.in"?
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 20:40:23 UTC, Paul wrote:

 On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:

 void main() {
     foreach(line; readText("file.in").splitLines()) ...
 }



 Thanks Jesse.
 I'm finding that I can't just substitute args[1] for a text 
 string.  Is there a clever way to use args[] in place of your 
 "file.in"?



Why not?

void main(string[] args) {
     if(args.length > 1) {
         foreach(line; readText(args[1]).splitLines()) ...
     }
}
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 20:43:47 UTC, Era Scarecrow wrote:

 On Friday, 11 May 2012 at 20:40:23 UTC, Paul wrote:

 On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:

 void main() {
    foreach(line; readText("file.in").splitLines()) ...
 }



 Thanks Jesse.
 I'm finding that I can't just substitute args[1] for a text 
 string.  Is there a clever way to use args[] in place of your 
 "file.in"?



 Why not?

 void main(string[] args) {
     if(args.length > 1) {
         foreach(line; readText(args[1]).splitLines()) ...
     }
 }



This is my program:

import std.stdio,std.string,std.file;

// Main function
void main(string[] args){
     if (args.length > 1){
         foreach(line; readText(args[1]).splitLines()) {
             writefln("line : %s", line);
             //if (n > 10) break;
         }
     }
}

I get this:

std.utf.UTFException std\utf.d(644): Invalid UTF-8 sequence (at 
index 1)
----------------
41E424
41E29B
4020F3
4020C1
402042
4077A4
4077E3
4073F3
438781
----------------

These are the first few lines of my text file:
NAME   = XPAW01_STA
   TYPE   = COMPND
   DESCRP =
   PERIOD = 1
   PHASE  = 0
   ON     = 0
   INITON = 2
   CINHIB = 0
   GR1DV1 =
   GR1DV2 =
   GR1DV3 =
   GR1DV4 =
May 11 2012
prev sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 21:13:41 UTC, Paul wrote:

 std.utf.UTFException std\utf.d(644): Invalid UTF-8 sequence (at 
 index 1)



   What are you reading? If it's regular text (0-127) then you
shouldn't have an issue. However 128-255 (or, -1 to -127) are
treated differently. D by default is UTF-8 or unicode encoded,
meaning it wants to see valid encodings that follow that
structure.

   Unfortunately I don't have good references of how to simply
convert old Ascii to a utf-8 (within Phobos). I have my own
workaround...
May 11 2012