www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Trouble with std.zlib

reply Andrew Wiley <wiley.andrew.j gmail.com> writes:
--0016e68deca820525804a867941c
Content-Type: text/plain; charset=ISO-8859-1

I'm using std.zlib in an application, and I'm having trouble decompressing a
file. The file is gzipped, and I can easily see the contents with `cat file
| gzip -d` but this code doesn't work:
auto arr = read("file.gz");
ubyte[] realarr = cast(ubyte[])uncompress(arr);

output:
std.zlib.ZlibException std/zlib.d(59): data error

From the header file, it seems like zlib should be detecting a gzip header

compressed format, and eventually the code I'm writing will be used in-memory instead of on files, so I'm going to have to uncompress it somehow. Has anyone encountered this before? Thanks, Andrew --0016e68deca820525804a867941c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I&#39;m using std.zlib in an application, and I&#39;m having trouble decomp= ressing a file. The file is gzipped, and I can easily see the contents with= `cat file | gzip -d` but this code doesn&#39;t work:<div><div>auto arr =3D= read(&quot;file.gz&quot;);</div> <div>ubyte[] realarr =3D cast(ubyte[])uncompress(arr);</div></div><div><br>= </div><div>output:</div><div>std.zlib.ZlibException std/zlib.d(59): data er= ror</div><div><br></div><div>From the header file, it seems like zlib shoul= d be detecting a gzip header and *just working*, but, well, it isn&#39;t. I= have to store data in a compressed format, and eventually the code I&#39;m= writing will be used in-memory instead of on files, so I&#39;m going to ha= ve to uncompress it somehow.</div> <div>Has anyone encountered this before?</div><div><br></div><div>Thanks,</= div><div>Andrew</div> --0016e68deca820525804a867941c--
Jul 19 2011
next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Andrew Wiley Wrote:

 I'm using std.zlib in an application, and I'm having trouble decompressing a
 file. The file is gzipped, and I can easily see the contents with `cat file
 | gzip -d` but this code doesn't work:
 auto arr = read("file.gz");
 ubyte[] realarr = cast(ubyte[])uncompress(arr);

The zlib format is different from the gzip format. http://www.zlib.net/zlib_faq.html#faq18 One is made for file storage and the other internal data transfer. D does not provide a wrapper to the gzip functions at this time but they are available in std.c.zlib (with the C interface).
Jul 19 2011
prev sibling next sibling parent reply Andrew Wiley <wiley.andrew.j gmail.com> writes:
--001636c924e8454db904a86e56cc
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jul 19, 2011 at 7:11 AM, Jesse Phillips
<jessekphillips+D gmail.com>wrote:

 Andrew Wiley Wrote:

 I'm using std.zlib in an application, and I'm having trouble

 file. The file is gzipped, and I can easily see the contents with `cat

 | gzip -d` but this code doesn't work:
 auto arr = read("file.gz");
 ubyte[] realarr = cast(ubyte[])uncompress(arr);

The zlib format is different from the gzip format. http://www.zlib.net/zlib_faq.html#faq18 One is made for file storage and the other internal data transfer. D does not provide a wrapper to the gzip functions at this time but they are available in std.c.zlib (with the C interface).

I know, but if you look at the docs for the inflate functions, zlib is supposed to be autodetecting the gzip header and uncompressing the payload correctly, if I understand the documentation (which I probably don't at this point). --001636c924e8454db904a86e56cc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 7:11 AM, Jesse Phillips = <span dir=3D"ltr">&lt;<a href=3D"mailto:jessekphillips%2BD gmail.com">jesse= kphillips+D gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x;"> <div class=3D"im">Andrew Wiley Wrote:<br> <br> &gt; I&#39;m using std.zlib in an application, and I&#39;m having trouble d= ecompressing a<br> &gt; file. The file is gzipped, and I can easily see the contents with `cat= file<br> &gt; | gzip -d` but this code doesn&#39;t work:<br> &gt; auto arr =3D read(&quot;file.gz&quot;);<br> &gt; ubyte[] realarr =3D cast(ubyte[])uncompress(arr);<br> <br> </div>The zlib format is different from the gzip format.<br> <br> <a href=3D"http://www.zlib.net/zlib_faq.html#faq18" target=3D"_blank">http:= //www.zlib.net/zlib_faq.html#faq18</a><br> <br> One is made for file storage and the other internal data transfer. D does n= ot provide a wrapper to the gzip functions at this time but they are availa= ble in std.c.zlib (with the C interface).<br></blockquote><div><br></div> <div>I know, but if you look at the docs for the inflate functions, zlib is= supposed to be autodetecting the gzip header and uncompressing the payload= correctly, if I understand the documentation (which I probably don&#39;t a= t this point).=A0</div> </div><br> --001636c924e8454db904a86e56cc--
Jul 19 2011
parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Andrew Wiley Wrote:

 I know, but if you look at the docs for the inflate functions, zlib is
 supposed to be autodetecting the gzip header and uncompressing the payload
 correctly, if I understand the documentation (which I probably don't at this
 point).

Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."
Jul 19 2011
prev sibling next sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--0016369fa2ba19f94c04a86f40b0
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips
<jessekphillips+D gmail.com>wrote:

 Andrew Wiley Wrote:

 I know, but if you look at the docs for the inflate functions, zlib is
 supposed to be autodetecting the gzip header and uncompressing the

 correctly, if I understand the documentation (which I probably don't at

 point).

Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."

Ah, I stand corrected: "inflate() can decompress and check either zlib-wrapped or gzip-wrapped deflate data. The header type is detected automatically, if requested when initializing with inflateInit2()." --0016369fa2ba19f94c04a86f40b0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips = <span dir=3D"ltr">&lt;<a href=3D"mailto:jessekphillips%2BD gmail.com">jesse= kphillips+D gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x;"> <div class=3D"im">Andrew Wiley Wrote:<br> <br> &gt; I know, but if you look at the docs for the inflate functions, zlib is= <br> &gt; supposed to be autodetecting the gzip header and uncompressing the pay= load<br> &gt; correctly, if I understand the documentation (which I probably don&#39= ;t at this<br> &gt; point).<br> <br> </div>Not sure what documentation you are talking of. But it doesn&#39;t so= und like it does auto detection, an answer to one of the question:<br> <br> &quot;You can request that deflate write the gzip format instead of the zli= b format using deflateInit2(). You can also request that inflate decode the= gzip format using inflateInit2(). Read zlib.h for more details.&quot;<br> </blockquote></div><br><div>Ah, I stand corrected:</div><div>&quot;inflate(= ) can decompress and check either zlib-wrapped or gzip-wrapped</div><div>= =A0 deflate data. =A0The header type is detected automatically, if requeste= d when</div> <div>=A0 initializing with inflateInit2().&quot;</div> --0016369fa2ba19f94c04a86f40b0--
Jul 19 2011
prev sibling parent Andrew Wiley <wiley.andrew.j gmail.com> writes:
--0016e68ee0b138b3a904a86f5207
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jul 19, 2011 at 10:14 AM, Andrew Wiley <wiley.andrew.j gmail.com>wrote:

 On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips <
 jessekphillips+D gmail.com> wrote:

 Andrew Wiley Wrote:

 I know, but if you look at the docs for the inflate functions, zlib is
 supposed to be autodetecting the gzip header and uncompressing the

 correctly, if I understand the documentation (which I probably don't at

 point).

Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."

Ah, I stand corrected: "inflate() can decompress and check either zlib-wrapped or gzip-wrapped deflate data. The header type is detected automatically, if requested when initializing with inflateInit2()."

Looks like this is the solution: ubyte[] realarr = cast(ubyte[])uncompress(arr, 0, 47); The 47 makes zlib autodetect the header (because the 32 bit is set; if you wanted to force gzip, you'd set the 16 bit). --0016e68ee0b138b3a904a86f5207 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 10:14 AM, Andrew Wiley <= span dir=3D"ltr">&lt;<a href=3D"mailto:wiley.andrew.j gmail.com">wiley.andr= ew.j gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <div><div></div><div class=3D"h5"><div class=3D"gmail_quote">On Tue, Jul 19= , 2011 at 9:57 AM, Jesse Phillips <span dir=3D"ltr">&lt;<a href=3D"mailto:j= essekphillips%2BD gmail.com" target=3D"_blank">jessekphillips+D gmail.com</= a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div>Andrew Wiley Wrote:<br> <br> &gt; I know, but if you look at the docs for the inflate functions, zlib is= <br> &gt; supposed to be autodetecting the gzip header and uncompressing the pay= load<br> &gt; correctly, if I understand the documentation (which I probably don&#39= ;t at this<br> &gt; point).<br> <br> </div>Not sure what documentation you are talking of. But it doesn&#39;t so= und like it does auto detection, an answer to one of the question:<br> <br> &quot;You can request that deflate write the gzip format instead of the zli= b format using deflateInit2(). You can also request that inflate decode the= gzip format using inflateInit2(). Read zlib.h for more details.&quot;<br> </blockquote></div><br></div></div><div>Ah, I stand corrected:</div><div>&q= uot;inflate() can decompress and check either zlib-wrapped or gzip-wrapped<= /div><div>=A0 deflate data. =A0The header type is detected automatically, i= f requested when</div> <div>=A0 initializing with inflateInit2().&quot;</div> </blockquote></div><br><div>Looks like this is the solution:</div><div>ubyt= e[] realarr =3D cast(ubyte[])uncompress(arr, 0, 47);</div><div>The 47 makes= zlib autodetect the header (because the 32 bit is set; if you wanted to fo= rce gzip, you&#39;d set the 16 bit).</div> <div><br></div> --0016e68ee0b138b3a904a86f5207--
Jul 19 2011