www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Embed files into binary.

reply Pie? <AmericanPie gmail.com> writes:
I saw somewhere someone explaining how to embed resources into a 
binary using the import keyword.

I believe the essentially converted the file into a ubyte or 
something and then wrote that out to a temp file and read in the 
temp file... this seems a bit of a kludge to me.

Is it possible to do the same, but create a sort of "in memory" 
file layout?

While I can modify my own routines to read the "file" from 
memory, I can't do that easily with outside code(I could modify 
the binaries).

Basically reading a file has to read it to memory, and if the 
file data already exists in memory, there is no point to read, 
just get it direct.

Any thoughts on this?

Because D allows such an embedding feature, maybe the file system 
should allow working with this concept? That way, it becomes VERY 
easy to embed files into the binary and work with them like they 
wernt. Also, going between the two different versions(embedded vs 
not) could be done with version (Release).
Jun 05 2016
next sibling parent reply docandrew <x x.com> writes:
On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I saw somewhere someone explaining how to embed resources into 
 a binary using the import keyword.

 I believe the essentially converted the file into a ubyte or 
 something and then wrote that out to a temp file and read in 
 the temp file... this seems a bit of a kludge to me.

 Is it possible to do the same, but create a sort of "in memory" 
 file layout?

 While I can modify my own routines to read the "file" from 
 memory, I can't do that easily with outside code(I could modify 
 the binaries).

 Basically reading a file has to read it to memory, and if the 
 file data already exists in memory, there is no point to read, 
 just get it direct.

 Any thoughts on this?

 Because D allows such an embedding feature, maybe the file 
 system should allow working with this concept? That way, it 
 becomes VERY easy to embed files into the binary and work with 
 them like they wernt. Also, going between the two different 
 versions(embedded vs not) could be done with version (Release).


I'm not sure about import, but one option is to put the external 
files directly in your binary as part of the linker step. Check 
out:

http://www.linuxjournal.com/content/embedding-file-executable-aka-hello-world-version-5967

On Windows I think there's a way to embed files in .DLLs and then 
link those into your executable, but I'm ignorant of the 
steps/tools required.

-Jon
Jun 05 2016
parent Pie? <AmericanPie gmail.com> writes:
On Monday, 6 June 2016 at 02:18:48 UTC, docandrew wrote:

 On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I saw somewhere someone explaining how to embed resources into 
 a binary using the import keyword.

 I believe the essentially converted the file into a ubyte or 
 something and then wrote that out to a temp file and read in 
 the temp file... this seems a bit of a kludge to me.

 Is it possible to do the same, but create a sort of "in 
 memory" file layout?

 While I can modify my own routines to read the "file" from 
 memory, I can't do that easily with outside code(I could 
 modify the binaries).

 Basically reading a file has to read it to memory, and if the 
 file data already exists in memory, there is no point to read, 
 just get it direct.

 Any thoughts on this?

 Because D allows such an embedding feature, maybe the file 
 system should allow working with this concept? That way, it 
 becomes VERY easy to embed files into the binary and work with 
 them like they wernt. Also, going between the two different 
 versions(embedded vs not) could be done with version (Release).


 I'm not sure about import, but one option is to put the 
 external files directly in your binary as part of the linker 
 step. Check out:

 http://www.linuxjournal.com/content/embedding-file-executable-aka-hello-world-version-5967

 On Windows I think there's a way to embed files in .DLLs and 
 then link those into your executable, but I'm ignorant of the 
 steps/tools required.

 -Jon


Check out https://p0nce.github.io/d-idioms/ for embedding dll's.

It's much easier than using the link you give.

The problem is the same though, How to use the in memory data 
rather than re-reading it? It's not a problem of embedding but 
that the routines that use the data expect files, not memory 
backed virtual files.
Jun 05 2016
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I believe the essentially converted the file into a ubyte or 
 something and then wrote that out to a temp file and read in 
 the temp file... this seems a bit of a kludge to me.


They might do that for certain special cases, but 
import("file.whatever") just drops the file content in memory and 
you can then access it as an array.


 Because D allows such an embedding feature, maybe the file 
 system should allow working with this concept?


Why do you need it through the file system? If you're writing the 
code, just use the array in memory. If it is external, see if the 
library offers something like that.

DLL and exe are a special case, most things don't need to be 
physical files.
Jun 05 2016
parent reply Pie? <AmericanPie gmail.com> writes:
On Monday, 6 June 2016 at 02:34:07 UTC, Adam D. Ruppe wrote:

 On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I believe the essentially converted the file into a ubyte or 
 something and then wrote that out to a temp file and read in 
 the temp file... this seems a bit of a kludge to me.


 They might do that for certain special cases, but 
 import("file.whatever") just drops the file content in memory 
 and you can then access it as an array.


 Because D allows such an embedding feature, maybe the file 
 system should allow working with this concept?


 Why do you need it through the file system? If you're writing 
 the code, just use the array in memory. If it is external, see 
 if the library offers something like that.

 DLL and exe are a special case, most things don't need to be 
 physical files.


Because, as I said, if I'm working with pre-exiting modules that 
work with file, I have to provide a file or modify the source.

e.g., how could I do this easily with your read in your png 
module? It takes a file..

/// Easily reads a png file into a MemoryImage
MemoryImage readPng(string filename) {
	import std.file;
	return imageFromPng(readPng(cast(ubyte[]) read(filename)));

recognize the code?

Of course, like I said, it can be modified in this case, but that 
means nothing in general.
Jun 05 2016
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 06/06/2016 3:11 PM, Pie? wrote:

 On Monday, 6 June 2016 at 02:34:07 UTC, Adam D. Ruppe wrote:

 On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I believe the essentially converted the file into a ubyte or
 something and then wrote that out to a temp file and read in the temp
 file... this seems a bit of a kludge to me.


 They might do that for certain special cases, but
 import("file.whatever") just drops the file content in memory and you
 can then access it as an array.


 Because D allows such an embedding feature, maybe the file system
 should allow working with this concept?


 Why do you need it through the file system? If you're writing the
 code, just use the array in memory. If it is external, see if the
 library offers something like that.

 DLL and exe are a special case, most things don't need to be physical
 files.


 Because, as I said, if I'm working with pre-exiting modules that work
 with file, I have to provide a file or modify the source.

 e.g., how could I do this easily with your read in your png module? It
 takes a file..

 /// Easily reads a png file into a MemoryImage
 MemoryImage readPng(string filename) {
     import std.file;
     return imageFromPng(readPng(cast(ubyte[]) read(filename)));

 recognize the code?

 Of course, like I said, it can be modified in this case, but that means
 nothing in general.



ubyte[] theArray = import("...");
MemoryImage img = imageFromPng(readPng(theArray));

And that's just from the snippet you provided.
Yay overloads!
Jun 05 2016
parent reply Pie? <AmericanPie gmail.com> writes:
On Monday, 6 June 2016 at 03:23:02 UTC, rikki cattermole wrote:

 On 06/06/2016 3:11 PM, Pie? wrote:

 On Monday, 6 June 2016 at 02:34:07 UTC, Adam D. Ruppe wrote:

 On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I believe the essentially converted the file into a ubyte or
 something and then wrote that out to a temp file and read in 
 the temp
 file... this seems a bit of a kludge to me.


 They might do that for certain special cases, but
 import("file.whatever") just drops the file content in memory 
 and you
 can then access it as an array.


 Because D allows such an embedding feature, maybe the file 
 system
 should allow working with this concept?


 Why do you need it through the file system? If you're writing 
 the
 code, just use the array in memory. If it is external, see if 
 the
 library offers something like that.

 DLL and exe are a special case, most things don't need to be 
 physical
 files.


 Because, as I said, if I'm working with pre-exiting modules 
 that work
 with file, I have to provide a file or modify the source.

 e.g., how could I do this easily with your read in your png 
 module? It
 takes a file..

 /// Easily reads a png file into a MemoryImage
 MemoryImage readPng(string filename) {
     import std.file;
     return imageFromPng(readPng(cast(ubyte[]) read(filename)));

 recognize the code?

 Of course, like I said, it can be modified in this case, but 
 that means
 nothing in general.



 ubyte[] theArray = import("...");
 MemoryImage img = imageFromPng(readPng(theArray));

 And that's just from the snippet you provided.
 Yay overloads!


Yes, I figured it could be done because of the MemoryImage, but 
that wasn't the point.  You are delving in to code that you might 
not normally have access or might not exist.

Adam wrote the MemoryImage that essentially does this but that is 
not the general case.

Image if all you access to was readPng(string filename), then 
what would you say? That was my point of posting the code, I 
didn't mean for you to go look and see if it was possible to 
achieve what I was saying in arsd(Formally known as Adam Druppe) 
git repository ;)

If that was the case I would have titled this something similar 
not but maybe not exactly equivalent(Just being pedantic since we 
seem to be having a communications gap in how accurate and 
precise our expressions are to be interpreted): "Possible to get 
arsd(Formally known as Adam Druppe) readpng lib to read from 
import(filename)?".

Since I was desiring to arrive at a more general solution to 
potential future programming goals, I phrased my question in more 
general terms. I assumed that this would be understood. (i.e., 
vague questions generally gets/desires vague results, intelligent 
questions generally get/desires intelligent results, etc.)

In fact, I posted the most generic(and possibly vague) question I 
could:

"Embed files into binary."  I stated, but not exclaimed or 
questioned(as these are in theory, mutually exclusive), in the 
subject line!

But this is no question! It is a riddle my friend! By stating it 
in such a way as to only imply the logical consequent "How to use 
embedded files in the binary"[Of course, with an infinite number 
of assumptions mixed in for spice]!

Why is this a necessary forgone conclusion, as Einstein would 
some times say(not to me in particular, mind you)?  Because there 
is no point to embed files in a binary if one can not use them!!!

Naturally, this is why I further said, in a round about way, why 
exporting them to temp files is not really an option. This was 
further to exclude, if you are keeping track with your Venn 
Diagram, something akin to the infinite monkey theorem... or a 
breach in the space time continuum.... or just several responses 
about along the lines "...save the data to a temp file and use 
that".

Note my retort:

">>>> Because D allows such an embedding feature, maybe the file

 system
 should allow working with this concept?"








Asking about future events and possibly space-time paradoxes!

For example, what if I derive some source code from some 
unexpected places and they have some complex way of using D's 
std.file routines and offers no interface to use memory arrays as 
an input? Suppose further that it is very contorted and only 
Steven Hawkins has figured out how to unravel the complexities of 
the law of import/export. We are not privy to his knowledge or 
intellectual properties. Would it not be intelligent and 
forthright, mimicking Steven Hawkins in a sort of motherly 
daughterly bond kind of way, to see that we must write new code 
to solve the problem... this timing mimicking God himself as he 
bought forth the earth and heaven?


Does that make sense?
Jun 05 2016
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 06/06/2016 5:07 PM, Pie? wrote:

 On Monday, 6 June 2016 at 03:23:02 UTC, rikki cattermole wrote:

 On 06/06/2016 3:11 PM, Pie? wrote:

 On Monday, 6 June 2016 at 02:34:07 UTC, Adam D. Ruppe wrote:

 On Monday, 6 June 2016 at 02:05:09 UTC, Pie? wrote:

 I believe the essentially converted the file into a ubyte or
 something and then wrote that out to a temp file and read in the temp
 file... this seems a bit of a kludge to me.


 They might do that for certain special cases, but
 import("file.whatever") just drops the file content in memory and you
 can then access it as an array.


 Because D allows such an embedding feature, maybe the file system
 should allow working with this concept?


 Why do you need it through the file system? If you're writing the
 code, just use the array in memory. If it is external, see if the
 library offers something like that.

 DLL and exe are a special case, most things don't need to be physical
 files.


 Because, as I said, if I'm working with pre-exiting modules that work
 with file, I have to provide a file or modify the source.

 e.g., how could I do this easily with your read in your png module? It
 takes a file..

 /// Easily reads a png file into a MemoryImage
 MemoryImage readPng(string filename) {
     import std.file;
     return imageFromPng(readPng(cast(ubyte[]) read(filename)));

 recognize the code?

 Of course, like I said, it can be modified in this case, but that means
 nothing in general.



 ubyte[] theArray = import("...");
 MemoryImage img = imageFromPng(readPng(theArray));

 And that's just from the snippet you provided.
 Yay overloads!


 Yes, I figured it could be done because of the MemoryImage, but that
 wasn't the point.  You are delving in to code that you might not
 normally have access or might not exist.

 Adam wrote the MemoryImage that essentially does this but that is not
 the general case.

 Image if all you access to was readPng(string filename), then what would
 you say? That was my point of posting the code, I didn't mean for you to
 go look and see if it was possible to achieve what I was saying in
 arsd(Formally known as Adam Druppe) git repository ;)


Then I would say go get a new image library as that one isn't a very 
good one.

..snip..

Right got rid of all that text.
If you want to make the filesystem appear to have files it doesn't, 
you'll need a virtual file system library.

Welcome to the next issue, getting libraries to support it ;)
Jun 05 2016
parent reply Pie? <AmericanPie gmail.com> writes:
On Monday, 6 June 2016 at 05:30:12 UTC, rikki cattermole wrote:

 On 06/06/2016 5:07 PM, Pie? wrote:

 [...]


 Then I would say go get a new image library as that one isn't a 
 very good one.

 ..snip..

 Right got rid of all that text.
 If you want to make the filesystem appear to have files it 
 doesn't, you'll need a virtual file system library.

 Welcome to the next issue, getting libraries to support it ;)


My point exactly! Glad you could finally support my conclusion! ;)
Jun 05 2016
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 06/06/2016 6:06 PM, Pie? wrote:

 On Monday, 6 June 2016 at 05:30:12 UTC, rikki cattermole wrote:

 On 06/06/2016 5:07 PM, Pie? wrote:

 [...]


 Then I would say go get a new image library as that one isn't a very
 good one.

 ..snip..

 Right got rid of all that text.
 If you want to make the filesystem appear to have files it doesn't,
 you'll need a virtual file system library.

 Welcome to the next issue, getting libraries to support it ;)


 My point exactly! Glad you could finally support my conclusion! ;)


Finally was many many months ago[0].
I'm not happy with my implementation. Either way, somebody has to go 
ahead and write one for Phobos. Although good luck with getting that in.

[0] 
https://github.com/rikkimax/alphaPhobos/tree/master/source/std/experimental/vfs
Jun 05 2016
parent reply Pie? <AmericanPie gmail.com> writes:
On Monday, 6 June 2016 at 06:10:35 UTC, rikki cattermole wrote:

 On 06/06/2016 6:06 PM, Pie? wrote:

 On Monday, 6 June 2016 at 05:30:12 UTC, rikki cattermole wrote:

 On 06/06/2016 5:07 PM, Pie? wrote:

 [...]


 Then I would say go get a new image library as that one isn't 
 a very
 good one.

 ..snip..

 Right got rid of all that text.
 If you want to make the filesystem appear to have files it 
 doesn't,
 you'll need a virtual file system library.

 Welcome to the next issue, getting libraries to support it ;)


 My point exactly! Glad you could finally support my 
 conclusion! ;)


 Finally was many many months ago[0].
 I'm not happy with my implementation. Either way, somebody has 
 to go ahead and write one for Phobos. Although good luck with 
 getting that in.

 [0] 
 https://github.com/rikkimax/alphaPhobos/tree/master/source/std/experimental/vfs



Would this, say, allow an application to embed a "file system" 
into it's run time and then virtually dispatch file IO? If so, 
what about write access(the binary would grow in size when 
written to)?

What about seemlessly shifting between a real file system and the 
virtual one by simple flag toggling and rebuilding? If so, then D 
would definitely benefit from it.  This way one can create 
binaries and back them by a real file system for debugging 
purposes(easy access, no extracting from binaries or having to 
store them) then for release, one can turn that in to a virtual 
file system.

If this is what your vfs does or aims at then hats off! Cheerio! 
Where do I sign up? If not, would you mind expanding it to do 
such a thing?

I am essentially coding up something like that as we speak... 
Using static importing the binary and modulating all my file IO 
into a routine that chooses to use the physical file system when 
in debug mode and the imported data in release(it's quite easy, 
just dispatch to std.file io in debug and simply return the 
memory of the requested data in release).
Jun 05 2016
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 06/06/2016 6:45 PM, Pie? wrote:

 On Monday, 6 June 2016 at 06:10:35 UTC, rikki cattermole wrote:

 On 06/06/2016 6:06 PM, Pie? wrote:

 On Monday, 6 June 2016 at 05:30:12 UTC, rikki cattermole wrote:

 On 06/06/2016 5:07 PM, Pie? wrote:

 [...]


 Then I would say go get a new image library as that one isn't a very
 good one.

 ..snip..

 Right got rid of all that text.
 If you want to make the filesystem appear to have files it doesn't,
 you'll need a virtual file system library.

 Welcome to the next issue, getting libraries to support it ;)


 My point exactly! Glad you could finally support my conclusion! ;)


 Finally was many many months ago[0].
 I'm not happy with my implementation. Either way, somebody has to go
 ahead and write one for Phobos. Although good luck with getting that in.

 [0]
 https://github.com/rikkimax/alphaPhobos/tree/master/source/std/experimental/vfs



 Would this, say, allow an application to embed a "file system" into it's
 run time and then virtually dispatch file IO? If so, what about write
 access(the binary would grow in size when written to)?


Its in the name, "virtual" file system.
Write access would depend on where the "file" is.


 What about seemlessly shifting between a real file system and the
 virtual one by simple flag toggling and rebuilding? If so, then D would
 definitely benefit from it.  This way one can create binaries and back
 them by a real file system for debugging purposes(easy access, no
 extracting from binaries or having to store them) then for release, one
 can turn that in to a virtual file system.


Nope, virtual file system is its own representation. Can't switch all 
the c functions to use it. Its a almost full rewrite to use it.


 If this is what your vfs does or aims at then hats off! Cheerio! Where
 do I sign up? If not, would you mind expanding it to do such a thing?


Again, I'm not happy with my one.
Jun 05 2016
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 6 June 2016 at 03:11:32 UTC, Pie? wrote:

 e.g., how could I do this easily with your read in your png 
 module? It takes a file..

 /// Easily reads a png file into a MemoryImage
 MemoryImage readPng(string filename) {
 	import std.file;
 	return imageFromPng(readPng(cast(ubyte[]) read(filename)));

 recognize the code?



The string version just forwards to the byte array version. So 
you can use it directly with the array by just casting to ubyte[].

rikki cattermole got it right, except I *think* the cast(ubyte[]) 
needs to be explicit on the import line, so

ubyte[] theArray = cast(ubyte[]) import("...");

(Actually, the string one is an afterthought, I originally wrote 
it to only work with arrays (well, technically, input ranges of 
bytes), leaving the file reading to an outside function, but 
since loading a file from a filename is so common, I added the 
overload for it.)
Jun 05 2016