digitalmars.D - Trouble with std.zlib
- Andrew Wiley <wiley.andrew.j gmail.com> Jul 19 2011
- Jesse Phillips <jessekphillips+D gmail.com> Jul 19 2011
- Andrew Wiley <wiley.andrew.j gmail.com> Jul 19 2011
- Jesse Phillips <jessekphillips+D gmail.com> Jul 19 2011
- Andrew Wiley <wiley.andrew.j gmail.com> Jul 19 2011
- Andrew Wiley <wiley.andrew.j gmail.com> Jul 19 2011
--0016e68deca820525804a867941c
Content-Type: text/plain; charset=ISO-8859-1
I'm using std.zlib in an application, and I'm having trouble decompressing a
file. The file is gzipped, and I can easily see the contents with `cat file
| gzip -d` but this code doesn't work:
auto arr = read("file.gz");
ubyte[] realarr = cast(ubyte[])uncompress(arr);
output:
std.zlib.ZlibException std/zlib.d(59): data error
From the header file, it seems like zlib should be detecting a gzip header
compressed format, and eventually the code I'm writing will be used
in-memory instead of on files, so I'm going to have to uncompress it
somehow.
Has anyone encountered this before?
Thanks,
Andrew
--0016e68deca820525804a867941c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
I'm using std.zlib in an application, and I'm having trouble decomp=
ressing a file. The file is gzipped, and I can easily see the contents with=
`cat file | gzip -d` but this code doesn't work:<div><div>auto arr =3D=
read("file.gz");</div>
<div>ubyte[] realarr =3D cast(ubyte[])uncompress(arr);</div></div><div><br>=
</div><div>output:</div><div>std.zlib.ZlibException std/zlib.d(59): data er=
ror</div><div><br></div><div>From the header file, it seems like zlib shoul=
d be detecting a gzip header and *just working*, but, well, it isn't. I=
have to store data in a compressed format, and eventually the code I'm=
writing will be used in-memory instead of on files, so I'm going to ha=
ve to uncompress it somehow.</div>
<div>Has anyone encountered this before?</div><div><br></div><div>Thanks,</=
div><div>Andrew</div>
--0016e68deca820525804a867941c--
Jul 19 2011
Jesse Phillips <jessekphillips+D gmail.com> Andrew Wiley Wrote:I'm using std.zlib in an application, and I'm having trouble decompressing a file. The file is gzipped, and I can easily see the contents with `cat file | gzip -d` but this code doesn't work: auto arr = read("file.gz"); ubyte[] realarr = cast(ubyte[])uncompress(arr);
The zlib format is different from the gzip format. http://www.zlib.net/zlib_faq.html#faq18 One is made for file storage and the other internal data transfer. D does not provide a wrapper to the gzip functions at this time but they are available in std.c.zlib (with the C interface).
Jul 19 2011
--001636c924e8454db904a86e56cc Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jul 19, 2011 at 7:11 AM, Jesse Phillips <jessekphillips+D gmail.com>wrote:Andrew Wiley Wrote:I'm using std.zlib in an application, and I'm having trouble
file. The file is gzipped, and I can easily see the contents with `cat
| gzip -d` but this code doesn't work: auto arr = read("file.gz"); ubyte[] realarr = cast(ubyte[])uncompress(arr);
The zlib format is different from the gzip format. http://www.zlib.net/zlib_faq.html#faq18 One is made for file storage and the other internal data transfer. D does not provide a wrapper to the gzip functions at this time but they are available in std.c.zlib (with the C interface).
I know, but if you look at the docs for the inflate functions, zlib is supposed to be autodetecting the gzip header and uncompressing the payload correctly, if I understand the documentation (which I probably don't at this point). --001636c924e8454db904a86e56cc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 7:11 AM, Jesse Phillips = <span dir=3D"ltr"><<a href=3D"mailto:jessekphillips%2BD gmail.com">jesse= kphillips+D gmail.com</a>></span> wrote:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x;"> <div class=3D"im">Andrew Wiley Wrote:<br> <br> > I'm using std.zlib in an application, and I'm having trouble d= ecompressing a<br> > file. The file is gzipped, and I can easily see the contents with `cat= file<br> > | gzip -d` but this code doesn't work:<br> > auto arr =3D read("file.gz");<br> > ubyte[] realarr =3D cast(ubyte[])uncompress(arr);<br> <br> </div>The zlib format is different from the gzip format.<br> <br> <a href=3D"http://www.zlib.net/zlib_faq.html#faq18" target=3D"_blank">http:= //www.zlib.net/zlib_faq.html#faq18</a><br> <br> One is made for file storage and the other internal data transfer. D does n= ot provide a wrapper to the gzip functions at this time but they are availa= ble in std.c.zlib (with the C interface).<br></blockquote><div><br></div> <div>I know, but if you look at the docs for the inflate functions, zlib is= supposed to be autodetecting the gzip header and uncompressing the payload= correctly, if I understand the documentation (which I probably don't a= t this point).=A0</div> </div><br> --001636c924e8454db904a86e56cc--
Jul 19 2011
Andrew Wiley Wrote:I know, but if you look at the docs for the inflate functions, zlib is supposed to be autodetecting the gzip header and uncompressing the payload correctly, if I understand the documentation (which I probably don't at this point).
Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."
Jul 19 2011
--0016369fa2ba19f94c04a86f40b0 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips <jessekphillips+D gmail.com>wrote:Andrew Wiley Wrote:I know, but if you look at the docs for the inflate functions, zlib is supposed to be autodetecting the gzip header and uncompressing the
correctly, if I understand the documentation (which I probably don't at
point).
Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."
Ah, I stand corrected: "inflate() can decompress and check either zlib-wrapped or gzip-wrapped deflate data. The header type is detected automatically, if requested when initializing with inflateInit2()." --0016369fa2ba19f94c04a86f40b0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips = <span dir=3D"ltr"><<a href=3D"mailto:jessekphillips%2BD gmail.com">jesse= kphillips+D gmail.com</a>></span> wrote:<br><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x;"> <div class=3D"im">Andrew Wiley Wrote:<br> <br> > I know, but if you look at the docs for the inflate functions, zlib is= <br> > supposed to be autodetecting the gzip header and uncompressing the pay= load<br> > correctly, if I understand the documentation (which I probably don'= ;t at this<br> > point).<br> <br> </div>Not sure what documentation you are talking of. But it doesn't so= und like it does auto detection, an answer to one of the question:<br> <br> "You can request that deflate write the gzip format instead of the zli= b format using deflateInit2(). You can also request that inflate decode the= gzip format using inflateInit2(). Read zlib.h for more details."<br> </blockquote></div><br><div>Ah, I stand corrected:</div><div>"inflate(= ) can decompress and check either zlib-wrapped or gzip-wrapped</div><div>= =A0 deflate data. =A0The header type is detected automatically, if requeste= d when</div> <div>=A0 initializing with inflateInit2()."</div> --0016369fa2ba19f94c04a86f40b0--
Jul 19 2011
--0016e68ee0b138b3a904a86f5207 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Jul 19, 2011 at 10:14 AM, Andrew Wiley <wiley.andrew.j gmail.com>wrote:On Tue, Jul 19, 2011 at 9:57 AM, Jesse Phillips < jessekphillips+D gmail.com> wrote:Andrew Wiley Wrote:I know, but if you look at the docs for the inflate functions, zlib is supposed to be autodetecting the gzip header and uncompressing the
correctly, if I understand the documentation (which I probably don't at
point).
Not sure what documentation you are talking of. But it doesn't sound like it does auto detection, an answer to one of the question: "You can request that deflate write the gzip format instead of the zlib format using deflateInit2(). You can also request that inflate decode the gzip format using inflateInit2(). Read zlib.h for more details."
Ah, I stand corrected: "inflate() can decompress and check either zlib-wrapped or gzip-wrapped deflate data. The header type is detected automatically, if requested when initializing with inflateInit2()."
Looks like this is the solution: ubyte[] realarr = cast(ubyte[])uncompress(arr, 0, 47); The 47 makes zlib autodetect the header (because the 32 bit is set; if you wanted to force gzip, you'd set the 16 bit). --0016e68ee0b138b3a904a86f5207 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 19, 2011 at 10:14 AM, Andrew Wiley <= span dir=3D"ltr"><<a href=3D"mailto:wiley.andrew.j gmail.com">wiley.andr= ew.j gmail.com</a>></span> wrote:<br><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"> <div><div></div><div class=3D"h5"><div class=3D"gmail_quote">On Tue, Jul 19= , 2011 at 9:57 AM, Jesse Phillips <span dir=3D"ltr"><<a href=3D"mailto:j= essekphillips%2BD gmail.com" target=3D"_blank">jessekphillips+D gmail.com</= a>></span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div>Andrew Wiley Wrote:<br> <br> > I know, but if you look at the docs for the inflate functions, zlib is= <br> > supposed to be autodetecting the gzip header and uncompressing the pay= load<br> > correctly, if I understand the documentation (which I probably don'= ;t at this<br> > point).<br> <br> </div>Not sure what documentation you are talking of. But it doesn't so= und like it does auto detection, an answer to one of the question:<br> <br> "You can request that deflate write the gzip format instead of the zli= b format using deflateInit2(). You can also request that inflate decode the= gzip format using inflateInit2(). Read zlib.h for more details."<br> </blockquote></div><br></div></div><div>Ah, I stand corrected:</div><div>&q= uot;inflate() can decompress and check either zlib-wrapped or gzip-wrapped<= /div><div>=A0 deflate data. =A0The header type is detected automatically, i= f requested when</div> <div>=A0 initializing with inflateInit2()."</div> </blockquote></div><br><div>Looks like this is the solution:</div><div>ubyt= e[] realarr =3D cast(ubyte[])uncompress(arr, 0, 47);</div><div>The 47 makes= zlib autodetect the header (because the 32 bit is set; if you wanted to fo= rce gzip, you'd set the 16 bit).</div> <div><br></div> --0016e68ee0b138b3a904a86f5207--
Jul 19 2011