www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Bug in csv or byLine ?

reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
$ cat debug.csv
timestamp,curr_property
2015-12-01 06:07:55,7035
----------------------------------------------------------------
$ cat process.d
import std.stdio;
import std.csv;
import std.algorithm;
import std.file;

void main(string[] args) {
   version (Fail) {
     File(args[1], "r").byLine.joiner("\n").csvReader.each!writeln;
   } else {
     readText(args[1]).csvReader.each!writeln;
   }
}
----------------------------------------------------------------
$ dmd -run ./process.d debug.csv
["timestamp", "curr_property"]
["2015-12-01 06:07:55", "7035"]
----------------------------------------------------------------
$ dmd -version=Fail -run ./process.d debug.csv
["timestamp", "curr_property"]
["2015-12-01 06:07:55", "7035"]
core.exception.AssertError std/algorithm/iteration.d(2027): 
Assertion failure
----------------
??:? _d_assert [0x4633d3]
??:? void std.algorithm.iteration.__assert(int) [0x46d770]
??:? pure  property  safe dchar 
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, 
char).ByLine, 
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, 
char).ByLine, immutable(char)[]).Result.front() [0x44eaf0]
??:? void std.csv.CsvReader!(immutable(char)[], 1, 
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, 
char).ByLine, 
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, 
char).ByLine, immutable(char)[]).Result, dchar, 
immutable(char)[][]).CsvReader.popFront() [0x44f7fc]
??:? void 
std.algorithm.iteration.__T4eachS183std5stdio7writelnZ.each!(std.csv.CsvReader
(immutable(char)[], 1,
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).Result, dchar,
immutable(char)[][]).CsvReader).each(std.csv.CsvReader!(immutable(char)[], 1,
std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine,
immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader) [0x4608f7]
??:? _Dmain [0x44bc93]


Any idea ?
Jan 08
parent reply Tobi G. <gruen_tobias web.de> writes:
On Friday, 8 January 2016 at 09:59:26 UTC, Guillaume Chatelet 
wrote:
 Any idea ?
No, sorry. Under Windows DMD v2.069.2 it works perfectly in both cases. Which compiler do you use? You could run DMD with the -g option. This will print often more useful output, if it fails. togrue
Jan 08
parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:
 No, sorry. Under Windows DMD v2.069.2 it works perfectly in 
 both cases.

 Which compiler do you use?
- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0) So if it works on windows I guess it's a problem with the File implementation.
 You could run DMD with the -g option. This will print often 
 more useful output, if it fails.
-g didn't bring much. core.exception.AssertError std/algorithm/iteration.d(2027): Assertion failure ---------------- ??:? _d_assert [0x4a9c33] ??:? void std.algorithm.iteration.__assert(int) [0x4b8048] /usr/include/dmd/phobos/std/algorithm/iteration.d:2027 pure property safe dchar std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result.front() [0x495330] /usr/include/dmd/phobos/std/csv.d:1018 void std.csv.CsvReader!(immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader.popFront() [0x49608c] /usr/include/dmd/phobos/std/algorithm/iteration.d:881 void std.algorithm.iteration.__T4eachS183std5stdio7writelnZ.each!(std.csv.CsvReader (immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader).each(std.csv.CsvReader!(immutable(char)[], 1, std.algorithm.iteration.joiner!(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).joiner(std.stdio.File.ByLine!(char, char).ByLine, immutable(char)[]).Result, dchar, immutable(char)[][]).CsvReader) [0x4a5063] ./process.d:8 _Dmain [0x49226c]
Jan 08
parent reply Tobi G. <gruen_tobias web.de> writes:
On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet 
wrote:
 On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:
 No, sorry. Under Windows DMD v2.069.2 it works perfectly in 
 both cases.

 Which compiler do you use?
- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrue
Jan 08
parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Friday, 8 January 2016 at 13:22:40 UTC, Tobi G. wrote:
 On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet 
 wrote:
 On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:
 No, sorry. Under Windows DMD v2.069.2 it works perfectly in 
 both cases.

 Which compiler do you use?
- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrue
Indeed there's an empty line at the end of the csv. Interestingly enough if I try with DMD64 D Compiler v2.069, the Fail version runs fine but the normal version returns: std.csv.CSVException /usr/include/dlang/dmd/std/csv.d(1246): Row 3's length 1 does not match previous length of 2.
Jan 08
parent reply Keywan Ghadami <keywan.ghadami googlemail.com> writes:
On Friday, 8 January 2016 at 13:53:06 UTC, Guillaume Chatelet 
wrote:
 On Friday, 8 January 2016 at 13:22:40 UTC, Tobi G. wrote:
 On Friday, 8 January 2016 at 12:13:59 UTC, Guillaume Chatelet 
 wrote:
 On Friday, 8 January 2016 at 12:07:05 UTC, Tobi G. wrote:
 No, sorry. Under Windows DMD v2.069.2 it works perfectly in 
 both cases.

 Which compiler do you use?
- DMD64 D Compiler v2.069.2 on Linux. - LDC 0.16.1 (DMD v2.067.1, LLVM 3.7.0)
I ran it now under Linux/Ubuntu DMD64 D Compiler v2.069.2 But both still worked.. Are there some characters in your input data which are invalid and not displayed in the forum? (multiple empty lines after the actual csv data for example) togrue
Indeed there's an empty line at the end of the csv. Interestingly enough if I try with DMD64 D Compiler v2.069, the Fail version runs fine but the normal version returns: std.csv.CSVException /usr/include/dlang/dmd/std/csv.d(1246): Row 3's length 1 does not match previous length of 2.
I still do not understand half of the syntax(still learning) but my guess is that it is a bug in the csv reader: In https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e 32c82f785/std/csv.d line 1020 calls popfront on the input without checking that it input is empty. But this only happen in special case if the last line is \r\n if i understand it correctly. I might be totally wrong because it does not match your stackstrace and i did not yet understand everything about this.
Jan 10
parent reply Tobi G. <gruen_tobias web.de> writes:
On Sunday, 10 January 2016 at 09:41:16 UTC, Keywan Ghadami wrote:
 On Friday, 8 January 2016 at 13:53:06 UTC, Guillaume Chatelet 
 wrote:
 I still do not understand half of the syntax(still learning) 
 but my guess is that it is a bug in the csv reader:
 In 
 https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e
32c82f785/std/csv.d line 1020 calls popfront on the input without checking that
it input is empty.
 But this only happen in special case if the last line is \r\n 
 if i understand it correctly.
 I might be totally wrong because it does not match your 
 stackstrace and i did not yet understand everything about this.
Yes. It looks like there is no case to handle '\r' - line endings correctly. https://github.com/D-Programming-Language/phobos/blob/67c95e6de21d5d627e3c57128b4d6e332c82f785/std/csv.d?L=1020#L1020 If '\r' is the last character it will fail, (because it tryes to read the next character). enum data = "name, surname, age Joe, Joker, 99\r"; // WILL FAIL import std.csv, std.stdio; void main(string[] args) { auto reader = csvReader(data); foreach(entry; reader) { writeln(entry); } } togrue
Jan 10
parent reply Tobi G. <gruen_tobias web.de> writes:
The bug has been fixed...
Jan 10
parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:
 The bug has been fixed...
Do you have a link for the fix? Is there a BugZilla entry?
Jan 10
parent reply Tobi G. <gruen_tobias web.de> writes:
On Sunday, 10 January 2016 at 19:07:52 UTC, Jesse Phillips wrote:
 On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:
 The bug has been fixed...
Do you have a link for the fix? Is there a BugZilla entry?
Yes sure.. https://issues.dlang.org/show_bug.cgi?id=15545 and the fix at github https://github.com/D-Programming-Language/phobos/pull/3917 togrue
Jan 10
parent Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Sunday, 10 January 2016 at 19:50:15 UTC, Tobi G. wrote:
 On Sunday, 10 January 2016 at 19:07:52 UTC, Jesse Phillips 
 wrote:
 On Sunday, 10 January 2016 at 18:09:23 UTC, Tobi G. wrote:
 The bug has been fixed...
Do you have a link for the fix? Is there a BugZilla entry?
Yes sure.. https://issues.dlang.org/show_bug.cgi?id=15545 and the fix at github https://github.com/D-Programming-Language/phobos/pull/3917 togrue
Thx for the fix !
Jan 11