www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Read Complete File to Array of Lines

reply "Paul" <phshaffer gmail.com> writes:
I would like to read a complete file in one statement and then 
process it line by line.

foreach (line; MyFile)
etc.

Is it possible to read a file into and array of lines?
Thanks
May 11 2012
next sibling parent sclytrack <sclytrack iq87.fr> writes:
On 05/11/2012 05:00 PM, Paul wrote:
 I would like to read a complete file in one statement and then process
 it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

---SOURCE import std.stdio; import std.file; int main() { writeln("start"); foreach( line; File("src/main.d").byLine()) { writeln(line); } writeln("stop"); return 0; } ---OUTPUT start import std.stdio; import std.file; int main() { writeln("start"); foreach( line; File("src/main.d").byLine()) { writeln(line); } writeln("stop"); return 0; } stop
May 11 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 11, 2012 at 05:00:16PM +0200, Paul wrote:
 I would like to read a complete file in one statement and then
 process it line by line.
 
 foreach (line; MyFile)
 etc.
 
 Is it possible to read a file into and array of lines?

import std.array; import std.stdio; string[] getLines(File f) { auto a = appender!string; foreach (line; f.byLine()) { a.put(line); } return a.data; } void main(string[] args) { if (args > 1) { auto f = File(args[0]); auto lines = getLines(f); } } T -- "How are you doing?" "Doing what?"
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 15:18:11 UTC, H. S. Teoh wrote:
 On Fri, May 11, 2012 at 05:00:16PM +0200, Paul wrote:
 I would like to read a complete file in one statement and then
 process it line by line.
 
 foreach (line; MyFile)
 etc.
 
 Is it possible to read a file into and array of lines?

import std.array; import std.stdio; string[] getLines(File f) { auto a = appender!string; foreach (line; f.byLine()) { a.put(line); } return a.data; }

Does that work? I think you mean: string[] getLines(File f) { auto a = appender!(string[]); foreach (line; f.byLine()) { a.put(line.idup); } return a.data; } You could also write: string[] getLines(File f) { return array(map!"a.idup"(f.byLine)); } Or { return f.byLine.map!"a.idup".array; } This may be slower than the appender version, which is optimized for performance; I'm not sure. Graham
May 11 2012
prev sibling next sibling parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:
 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

Something like: import std.file; import std.string; void main() { foreach(line; readText("file.in").splitLines()) ... }
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:
 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

If you use the "byLine" approach... foreach(line; File("myfile").byLine)) { ... } or File("myfile").byLine.doSomethingWithLines() ...keep in mind that "byLine", as it traverses your file, reuses a single buffer to store the current line in. If you're storing those lines somewhere for later use, use "line.dup" or "line.idup" to make a copy of that buffer. (Or use a different approach, like readText/splitLines.) That was one of the first "gotcha" moments I had with D. :) Graham
May 11 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> wrote:

 I would like to read a complete file in one statement and then process  
 it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

Would something like this work? auto arr = array(map!"a.idup"(File("file.txt").byLine())); -Steve
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 18:57:52 UTC, Steven Schveighoffer 
wrote:
 On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> 
 wrote:

 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

Would something like this work? auto arr = array(map!"a.idup"(File("file.txt").byLine()));

It sure would. I suspect that Jesse's approach... readText("file.in").splitLines() ...would be the most efficient way if you need an actual array: slurp the whole file at once, then create an array of memory-sharing slices. I look forward to the great std.stdio/std.file unification of 201x, when I won't have to look in two modules for file-reading functions. :) Graham
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:
 It sure would. I suspect that Jesse's approach...

   readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual array: 
 slurp the whole file at once, then create an array of 
 memory-sharing slices.

Doesn't sound hard.. I could likely write a quick line splitting ranges fairly quickly, assuming \r and \n are the newlines, if there's new ones I'm unaware of then it may not work quite as well as you want :P Let's see....
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 20:06:45 UTC, Era Scarecrow wrote:
 On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:
 It sure would. I suspect that Jesse's approach...

  readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual  
 array: slurp the whole file at once, then create an array of  
 memory-sharing slices.


Actually re-reading that and glancing at splitLines, I see it's already done... Mmmm... ignore my post(s) I guess.
May 11 2012
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Friday, 11 May 2012 at 20:06:45 UTC, Era Scarecrow wrote:
 On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:
 It sure would. I suspect that Jesse's approach...

  readText("file.in").splitLines()

 ...would be the most efficient way if you need an actual 
 array: slurp the whole file at once, then create an array of 
 memory-sharing slices.

Doesn't sound hard.. I could likely write a quick line splitting ranges fairly quickly, assuming \r and \n are the newlines, if there's new ones I'm unaware of then it may not work quite as well as you want :P Let's see....

Hold on! It already exists: std.file.readText std.string.splitLines Graham
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 19:24:49 UTC, Graham Fawcett wrote:
 On Friday, 11 May 2012 at 18:57:52 UTC, Steven Schveighoffer 
 wrote:
 On Fri, 11 May 2012 11:00:16 -0400, Paul <phshaffer gmail.com> 
 wrote:

 I would like to read a complete file in one statement and 
 then process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

Would something like this work? auto arr = array(map!"a.idup"(File("file.txt").byLine()));

It sure would. I suspect that Jesse's approach... readText("file.in").splitLines() ...would be the most efficient way if you need an actual array: slurp the whole file at once, then create an array of memory-sharing slices. I look forward to the great std.stdio/std.file unification of 201x, when I won't have to look in two modules for file-reading functions. :) Graham

question? I will try the readText.splitLines(). Thanks to all!
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:
 On Friday, 11 May 2012 at 15:00:18 UTC, Paul wrote:
 I would like to read a complete file in one statement and then 
 process it line by line.

 foreach (line; MyFile)
 etc.

 Is it possible to read a file into and array of lines?
 Thanks

Something like: import std.file; import std.string; void main() { foreach(line; readText("file.in").splitLines()) ... }

Thanks Jesse. I'm finding that I can't just substitute args[1] for a text string. Is there a clever way to use args[] in place of your "file.in"?
May 11 2012
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 20:40:23 UTC, Paul wrote:
 On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:
 void main() {
     foreach(line; readText("file.in").splitLines()) ...
 }

Thanks Jesse. I'm finding that I can't just substitute args[1] for a text string. Is there a clever way to use args[] in place of your "file.in"?

Why not? void main(string[] args) { if(args.length > 1) { foreach(line; readText(args[1]).splitLines()) ... } }
May 11 2012
prev sibling next sibling parent "Paul" <phshaffer gmail.com> writes:
On Friday, 11 May 2012 at 20:43:47 UTC, Era Scarecrow wrote:
 On Friday, 11 May 2012 at 20:40:23 UTC, Paul wrote:
 On Friday, 11 May 2012 at 18:02:54 UTC, Jesse Phillips wrote:
 void main() {
    foreach(line; readText("file.in").splitLines()) ...
 }

Thanks Jesse. I'm finding that I can't just substitute args[1] for a text string. Is there a clever way to use args[] in place of your "file.in"?

Why not? void main(string[] args) { if(args.length > 1) { foreach(line; readText(args[1]).splitLines()) ... } }

This is my program: import std.stdio,std.string,std.file; // Main function void main(string[] args){ if (args.length > 1){ foreach(line; readText(args[1]).splitLines()) { writefln("line : %s", line); //if (n > 10) break; } } } I get this: std.utf.UTFException std\utf.d(644): Invalid UTF-8 sequence (at index 1) ---------------- 41E424 41E29B 4020F3 4020C1 402042 4077A4 4077E3 4073F3 438781 ---------------- These are the first few lines of my text file: NAME = XPAW01_STA TYPE = COMPND DESCRP = PERIOD = 1 PHASE = 0 ON = 0 INITON = 2 CINHIB = 0 GR1DV1 = GR1DV2 = GR1DV3 = GR1DV4 =
May 11 2012
prev sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Friday, 11 May 2012 at 21:13:41 UTC, Paul wrote:
 std.utf.UTFException std\utf.d(644): Invalid UTF-8 sequence (at 
 index 1)

What are you reading? If it's regular text (0-127) then you shouldn't have an issue. However 128-255 (or, -1 to -127) are treated differently. D by default is UTF-8 or unicode encoded, meaning it wants to see valid encodings that follow that structure. Unfortunately I don't have good references of how to simply convert old Ascii to a utf-8 (within Phobos). I have my own workaround...
May 11 2012