www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Reading bzipped files

reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
Have anybody cooked up any range adaptors for on the fly decoding 
of bzipped files? Preferable compatible with phobos standard 
interfaces for file io.

Should probably be built on top of

http://code.dlang.org/packages/bzip2
May 02 2015
next sibling parent reply "tom" <tom srtnwz.com> writes:
On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:

 Have anybody cooked up any range adaptors for on the fly 
 decoding of bzipped files? Preferable compatible with phobos 
 standard interfaces for file io.

 Should probably be built on top of

 http://code.dlang.org/packages/bzip2


i use Stephan Schiffels code from

http://forum.dlang.org/thread/djhteyhpcnaskpabxijj forum.dlang.org?page=2
May 02 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:

 i use Stephan Schiffels code from
 http://forum.dlang.org/thread/djhteyhpcnaskpabxijj forum.dlang.org?page=2


See polished version at:

https://github.com/nordlow/justd/blob/master/zio.d
May 03 2015
parent reply "monty" <monty python.org> writes:
On Sunday, 3 May 2015 at 14:37:32 UTC, Per Nordlöw wrote:

 On Saturday, 2 May 2015 at 20:37:44 UTC, tom wrote:

 i use Stephan Schiffels code from
 http://forum.dlang.org/thread/djhteyhpcnaskpabxijj forum.dlang.org?page=2


 See polished version at:

 https://github.com/nordlow/justd/blob/master/zio.d


cool. btw (at least GzipByLine) its sloooow if you compare it to 
gzcat and pipe it into stdin and use ByLineFast.
i think its mainly the buffer appending that  suboptimal.
May 04 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Monday, 4 May 2015 at 20:53:27 UTC, monty wrote:

 cool. btw (at least GzipByLine) its sloooow if you compare it 
 to gzcat and pipe it into stdin and use ByLineFast.
 i think its mainly the buffer appending that  suboptimal.


Could anyone please point out if this is possible to fix somehow? 
Is the solution to avoid the GC and do buffer reuse somehow?
May 04 2015
parent reply "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:

 Could anyone please point out if this is possible to fix 
 somehow? Is the solution to avoid the GC and do buffer reuse 
 somehow?


I *really* need this for reading Gigabytes of DBpedia data...
May 04 2015
parent reply "monty" <monty python.org> writes:
On Tuesday, 5 May 2015 at 06:50:12 UTC, Per Nordlöw wrote:

 On Tuesday, 5 May 2015 at 06:48:36 UTC, Per Nordlöw wrote:

 Could anyone please point out if this is possible to fix 
 somehow? Is the solution to avoid the GC and do buffer reuse 
 somehow?


 I *really* need this for reading Gigabytes of DBpedia data...


my approach would be to more or less port the ByLineFast approach.
i just haven't had any time to do this yet.

maybe this could work too though:
https://github.com/biod/BioD/tree/master/bio/core/bgzf


also check out:
https://github.com/agordon/fileslurp
https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d
May 05 2015
parent "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Tuesday, 5 May 2015 at 07:11:36 UTC, monty wrote:

 also check out:
 https://github.com/agordon/fileslurp
 https://github.com/biod/BioD/blob/master/bio/core/utils/bylinefast.d


Thanks!
May 05 2015
prev sibling parent "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Saturday, 2 May 2015 at 13:50:10 UTC, Per Nordlöw wrote:

 Should probably be built on top of

 http://code.dlang.org/packages/bzip2


Docs here:

http://www.bzip.org/1.0.3/html/hl-interface.html
May 05 2015