www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Getting a safe path for a temporary file

reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
Is it currently possible to get the path to a safe temporary 
file, i.e. one that is guaranteed to be freshly created and will 
not override another existing file?

There's `std.file.tempDir`, which doesn't create a unique file. 
Then there's `std.stdio.tmpfile()`, which does, but it returns a 
`File` object, and its `name` property is `null`.

Did I miss something? IMO this is very import functionality. One 
use case is passing these names as command line arguments to an 
external program that doesn't support stdin/stdout.
Jan 17 2015
next sibling parent reply "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote:
 Is it currently possible to get the path to a safe temporary 
 file, i.e. one that is guaranteed to be freshly created and 
 will not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique file. 
 Then there's `std.stdio.tmpfile()`, which does, but it returns 
 a `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. 
 One use case is passing these names as command line arguments 
 to an external program that doesn't support stdin/stdout.
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Jan 17 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote:
 On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote:
 Is it currently possible to get the path to a safe temporary 
 file, i.e. one that is guaranteed to be freshly created and 
 will not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique 
 file. Then there's `std.stdio.tmpfile()`, which does, but it 
 returns a `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. 
 One use case is passing these names as command line arguments 
 to an external program that doesn't support stdin/stdout.
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
Jan 17 2015
next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote:
 On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc 
 wrote:
 On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz 
 wrote:
 Is it currently possible to get the path to a safe temporary 
 file, i.e. one that is guaranteed to be freshly created and 
 will not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique 
 file. Then there's `std.stdio.tmpfile()`, which does, but it 
 returns a `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. 
 One use case is passing these names as command line arguments 
 to an external program that doesn't support stdin/stdout.
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
You're looking for core.sys.posix.stdlib : mkstemp. I think that should be used by std.stdio.File as well, care to create an enhancement request in bugzilla?
Jan 17 2015
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath
wrote:
 On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote:
 On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc 
 wrote:
 On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz 
 wrote:
 Is it currently possible to get the path to a safe temporary 
 file, i.e. one that is guaranteed to be freshly created and 
 will not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique 
 file. Then there's `std.stdio.tmpfile()`, which does, but it 
 returns a `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. 
 One use case is passing these names as command line 
 arguments to an external program that doesn't support 
 stdin/stdout.
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
You're looking for core.sys.posix.stdlib : mkstemp. I think that should be used by std.stdio.File as well, care to create an enhancement request in bugzilla?
But it's POSIX only :-(
Jan 18 2015
prev sibling parent reply Shriramana Sharma writes:
On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath 
wrote:
 You're looking for core.sys.posix.stdlib : mkstemp.

 I think that should be used by std.stdio.File as well, care to 
 create an enhancement request in bugzilla?
Though this thread is old, I ran into the issue when wanting to create a temporary file in my D program and so filed this: https://issues.dlang.org/show_bug.cgi?id=17926 For my program right now I'm using a souped-up version using a static array: import std.stdio; struct TempFile { string name; private File handle; alias handle this; } TempFile tempFileOpen() { char[20] name = "/tmp/XXXXXX"; File handle; import core.sys.posix.stdlib: mkstemp; handle.fdopen(mkstemp(name.ptr), "w"); import std.string: fromStringz; return TempFile(fromStringz(name.ptr).idup, handle); } void main() { TempFile tfile; tfile = tempFileOpen(); writeln(tfile.name); tfile = tempFileOpen(); writeln(tfile.name); }
Oct 22 2017
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Sunday, October 22, 2017 15:21:37 Shriramana Sharma via Digitalmars-d-
learn wrote:
 On Saturday, 17 January 2015 at 17:16:41 UTC, Tobias Pankrath

 wrote:
 You're looking for core.sys.posix.stdlib : mkstemp.

 I think that should be used by std.stdio.File as well, care to
 create an enhancement request in bugzilla?
Though this thread is old, I ran into the issue when wanting to create a temporary file in my D program and so filed this: https://issues.dlang.org/show_bug.cgi?id=17926 For my program right now I'm using a souped-up version using a static array: import std.stdio; struct TempFile { string name; private File handle; alias handle this; } TempFile tempFileOpen() { char[20] name = "/tmp/XXXXXX"; File handle; import core.sys.posix.stdlib: mkstemp; handle.fdopen(mkstemp(name.ptr), "w"); import std.string: fromStringz; return TempFile(fromStringz(name.ptr).idup, handle); } void main() { TempFile tfile; tfile = tempFileOpen(); writeln(tfile.name); tfile = tempFileOpen(); writeln(tfile.name); }
We did have a function for creating a temporary file and returning it as a File, but it quickly got canned, because adding that functionality to std.stdio resulted in "hello world" increasing in size, which apparently was considered unacceptable: https://issues.dlang.org/show_bug.cgi?id=14599 I did recently create this enhancement request: https://issues.dlang.org/show_bug.cgi?id=17912 and this PR: https://github.com/dlang/phobos/pull/5788 It creates a function in std.file that acts similar to std.file.write but which writes to a randomly generated file name instead of the one you tell it. So, you don't get an open file out of the deal, but you can write data to it when it's first opened, and you can reopen it afterwards (which is usually what I've seen the need for anyway - e.g. a test where you need to pass a filename to a function that you're testing so that it can open it and do whatever it does normally). - Jonathan M Davis
Oct 22 2017
prev sibling parent reply Shriramana Sharma writes:
On Sunday, 22 October 2017 at 15:21:37 UTC, Shriramana Sharma 
wrote:
 For my program right now I'm using a souped-up version using a 
 static array:

     char[20] name = "/tmp/XXXXXX";
Hmm I was wondering if I needed it to be static, and verily, substituting: char[] name = "/tmp/XXXXXX".dup; instead gives a proper output *some* of the time but mostly gives the error: std.exception.ErrnoException std/stdio.d(630): (Bad file descriptor) ---------------- ??:? safe shared(core.stdc.stdio._IO_FILE)* std.exception.errnoEnforce!(shared(core.stdc.stdio._IO_FILE)*, "std/stdio.d", 630uL).errnoEnforce(shared(core.stdc.stdio._IO_FILE)*, lazy immutable(char)[]) [0x44bd31] ??:? trusted void std.stdio.File.fdopen(int, const(char[]), immutable(char)[]) [0x44294a] ??:? safe void std.stdio.File.fdopen(int, const(char[])) [0x4428b1] ??:? <src>.TempFile <src>.tempFileOpen() [0x43d4fd] ??:? _Dmain [0x43d5ce] Is it because the D slice is subject to relocation and C is occasionally not able to access the proper pointer?
Oct 22 2017
parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 10/22/2017 06:41 PM, Shriramana Sharma wrote:
 On Sunday, 22 October 2017 at 15:21:37 UTC, Shriramana Sharma wrote:
 For my program right now I'm using a souped-up version using a static
 array:

     char[20] name = "/tmp/XXXXXX";
Literal strings have a '\0' attached, which does not become a part of the slice.
 Hmm I was wondering if I needed it to be static, and verily, 
substituting:
 char[] name = "/tmp/XXXXXX".dup;
However, .dup is not be aware of the special literal treatment so it does not copy the '\0' character.
 Is it because the D slice is subject to relocation and C is occasionally
 not able to access the proper pointer?
As long as you have a reference on the D side, the slice will remain valid. The actual issue is the missing '\0'. So, consider toStringz in this case: https://dlang.org/library/std/string/to_stringz.html Ali
Oct 24 2017
parent reply Shriramana Sharma writes:
On Wednesday, 25 October 2017 at 00:35:29 UTC, Ali Çehreli wrote:
 char[] name = "/tmp/XXXXXX".dup;
remain valid. The actual issue is the missing '\0'. So, consider toStringz in this case: https://dlang.org/library/std/string/to_stringz.html
Thanks for your reply, but can you clarify exactly I should use this? char[] name = "/tmp/XXXXXX".toStringz; gives <src>(13): Error: cannot implicitly convert expression `toStringz("/tmp/XXXXXX")` of type `immutable(char)*` to `char[]` So I tried: char[] name = "/tmp/XXXXXX".toStringz.dup; which gives <src>(13): Error: template object.dup cannot deduce function from argument types !()(immutable(char)*), candidates are: /usr/include/dmd/druntime/import/object.d(1943): object.dup(T : V[K], K, V)(T aa) /usr/include/dmd/druntime/import/object.d(1979): object.dup(T : V[K], K, V)(T* aa) /usr/include/dmd/druntime/import/object.d(3764): object.dup(T)(T[] a) if (!is(const(T) : T)) /usr/include/dmd/druntime/import/object.d(3780): object.dup(T)(const(T)[] a) if (is(const(T) : T)) And: char[] name = "/tmp/XXXXXX".dup.toStringz; gives the error <src>(13): Error: cannot implicitly convert expression `toStringz(dup("/tmp/XXXXXX"))` of type `immutable(char)*` to `char[]` So I'm not sure what to do?!
Oct 27 2017
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 28, 2017 02:46:00 Shriramana Sharma via Digitalmars-d-
learn wrote:
 On Wednesday, 25 October 2017 at 00:35:29 UTC, Ali Çehreli wrote:
 char[] name = "/tmp/XXXXXX".dup;
remain valid. The actual issue is the missing '\0'. So, consider toStringz in this case: https://dlang.org/library/std/string/to_stringz.html
Thanks for your reply, but can you clarify exactly I should use this? char[] name = "/tmp/XXXXXX".toStringz; gives <src>(13): Error: cannot implicitly convert expression `toStringz("/tmp/XXXXXX")` of type `immutable(char)*` to `char[]` So I tried: char[] name = "/tmp/XXXXXX".toStringz.dup; which gives <src>(13): Error: template object.dup cannot deduce function from argument types !()(immutable(char)*), candidates are: /usr/include/dmd/druntime/import/object.d(1943): object.dup(T : V[K], K, V)(T aa) /usr/include/dmd/druntime/import/object.d(1979): object.dup(T : V[K], K, V)(T* aa) /usr/include/dmd/druntime/import/object.d(3764): object.dup(T)(T[] a) if (!is(const(T) : T)) /usr/include/dmd/druntime/import/object.d(3780): object.dup(T)(const(T)[] a) if (is(const(T) : T)) And: char[] name = "/tmp/XXXXXX".dup.toStringz; gives the error <src>(13): Error: cannot implicitly convert expression `toStringz(dup("/tmp/XXXXXX"))` of type `immutable(char)*` to `char[]` So I'm not sure what to do?!
Well, for starters, toStringz returns a pointer, not a dynamic array. If you want a dynamic array that's null-terminated, then you'll need to explicitly put a null character on the end unless you're just going to use a string literal (which would mean a string, not a char[], so that won't work here). Also, toStringz specifically returns an immutable(char)* - though looking it over right now, I'd say that that's a bug for the overload that takes const(char)[] instead of string. It really should return const(char)* in that case. But regardless, that means that even changing char[] to char* wouldn't cut it if you're using toStringz, since you're starting with a string. If you want a specific constness, then you can use the more general function, std.utf.toUTFz, which works with multiple character types and differing constness rather than being designed specifically for string like toStringz is. However, something to take into account is that toStringz and toUTFz don't always return the same string (and in fact, they probably never should, because the trick they use to check for the null character one past the end of the string doesn't always work correctly). So, if you're passing a string to a C function that's then going to mutate it, you don't want toStringz or toUTFZ. In that case, it's better to just manually put the null terminator at the end of the string. So, you probably would end up with something like char[] name = "/tmp/XXXXXX\0".dup; auto fd = mkstemp(name.ptr); Then when you return the name, you do something like name[0 .. $ - 1].idup; Alternatively, you could use a static array, but either way, you'd want to put the null terminator in there manually. And fromStringz really isn't necessary, since mkstemp is just going to fill in the array that you gave it, and you know exactly where the null terminator is going to be, since it's just filling in the X's rather than doing something that could end up putting a null terminator anywhere in the array. You can just slice the array to chop off the null terminator, and then dup it or idup if it's a static array (so that you don't return a slice of a local variable), or if it's a dynamic array, then either return it as-is or idup it if you want a string; you could even use std.exception.assumeUnique to just cast it to string if you know that no other references to that data exists (which they wouldn't if you allocated the string inside of the function). Also, you _really_ wouldn't want to use fromStringz if you used a static array, since fromStringz always returns a slice of the original input: inout(char)[] fromStringz(inout(char)* cString) nogc system pure nothrow { import core.stdc.string : strlen; return cString ? cString[0 .. strlen(cString)] : null; } But if I were you, I'd just create a char[] with an explicit null terminator, and then afterwards, slice off the null character, and pass it to assumeUnique to get a string, since then you allocate only once and don't have to copy the contents of the array. - Jonathan M Davis
Oct 27 2017
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Friday, October 27, 2017 21:38:55 Jonathan M Davis via Digitalmars-d-
learn wrote:
 Also, toStringz specifically returns an immutable(char)* - though looking
 it over right now, I'd say that that's a bug for the overload that takes
 const(char)[] instead of string. It really should return const(char)* in
 that case.
Actually, looking it over again, the return type is fine. That's the overload that always allocates a new string (since it assumes that the array could not have been from a string literal, since that would be a string and would go to the other overload), so even though the argument could have been const or mutable, what's returned is a pointer to a string, so immutable(char)* is correct. But https://issues.dlang.org/show_bug.cgi?id=15136 is still a problem for the other overload. - Jonathan M Davis
Oct 27 2017
prev sibling parent reply "Laeeth Isharc" <Laeethnospam nospam.laeeth.com> writes:
On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote:
 On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc 
 wrote:
 On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz 
 wrote:
 Is it currently possible to get the path to a safe temporary 
 file, i.e. one that is guaranteed to be freshly created and 
 will not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique 
 file. Then there's `std.stdio.tmpfile()`, which does, but it 
 returns a `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. 
 One use case is passing these names as command line arguments 
 to an external program that doesn't support stdin/stdout.
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid?
Jan 17 2015
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 18 January 2015 at 00:51:37 UTC, Laeeth Isharc wrote:
 I don't follow why a collision attack is applicable in this 
 case.
  Your stage 1 of generating unique names: how is this different 
 from using a random uuid?
It's not different, and if you're still doing the O_EXCL open afterwards, it's safe. I just assumed you were going to use the generated filename without a further check. This is then unsafe, no matter how the UUID is generated, and depending on the RNG that's been used, they can be quite predictable. Granted, the risk is low, but still...
Jan 18 2015
parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 18 January 2015 at 11:21:52 UTC, Marc Schütz wrote:
 It's not different, and if you're still doing the O_EXCL open 
 afterwards, it's safe. I just assumed you were going to use the 
 generated filename without a further check. This is then 
 unsafe, no matter how the UUID is generated, and depending on 
 the RNG that's been used, they can be quite predictable. 
 Granted, the risk is low, but still...
tmpfile is more predictable: it generates sequential file names.
Jan 18 2015
parent Cym13 <cpicard openmailbox.org> writes:
On Sunday, 18 January 2015 at 16:00:32 UTC, Kagamin wrote:
 On Sunday, 18 January 2015 at 11:21:52 UTC, Marc Schütz wrote:
 It's not different, and if you're still doing the O_EXCL open 
 afterwards, it's safe. I just assumed you were going to use 
 the generated filename without a further check. This is then 
 unsafe, no matter how the UUID is generated, and depending on 
 the RNG that's been used, they can be quite predictable. 
 Granted, the risk is low, but still...
tmpfile is more predictable: it generates sequential file names.
Being predictable is only an issue if the file is wrongly used (ie: no check that it might already exist, or be a symlink or check at the wrong time leaving an exploitable time frame etc). Sequential file names are a good way to provide uniqueness over a single system after all.
Oct 25 2017
prev sibling parent Cym13 <cpicard openmailbox.org> writes:
On Sunday, 18 January 2015 at 00:51:37 UTC, Laeeth Isharc wrote:
 On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote:
 On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc 
 wrote:
 On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz 
 wrote:
 [...]
I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name.
I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid?
UUIDs are defined to be unique, not unpredictable. UUID that use random number generation (UUID4) should use a cryptographically secure random number generator but are not required to. Therefore it shouldn't be blindly trusted against someone actively trying to get a collision.
Oct 25 2017
prev sibling parent reply Jonathan M Davis via Digitalmars-d-learn writes:
On Saturday, January 17, 2015 13:47:37 via Digitalmars-d-learn wrote:
 Is it currently possible to get the path to a safe temporary
 file, i.e. one that is guaranteed to be freshly created and will
 not override another existing file?

 There's `std.file.tempDir`, which doesn't create a unique file.
 Then there's `std.stdio.tmpfile()`, which does, but it returns a
 `File` object, and its `name` property is `null`.

 Did I miss something? IMO this is very import functionality. One
 use case is passing these names as command line arguments to an
 external program that doesn't support stdin/stdout.
The _only_ way to this write is to randomly generate a file name and then open the file with O_CREAT | O_EXCL (or more likely, O_RDWR | O_CREAT | O_EXCL), and then retry with a new name if the creation fails (good enough random name generation would likely require only two attempts at most). Simply randomly generating a file name is not enough, because it's still technically possible for the file to already exist (even if it's unlikely), and even checking for the file's existence prior to opening it isn't enough, because technically, the file could be created by another program in the small amount of time between when you checked for the file's existence and tried to create it. POSIX actually has mkstemp for doing this for you, but on some operating systems, it restricts the number of random files that it can generate to as little as 26 (at least, that's what I recall the number being). I don't think that any of the POSIX systems that we currently support have an implementation of mkstemp that's quite that bad, but all in all, I don't think that using mkstemp is a good idea. The problem is solved simply enough by randomly generating a file name (e.g. with rndGen()) and then using the correct flags with open. And I actually have code that does this that I was working on getting into Phobos, but the problem was getting access to the correct function on Windows (_wsopen_s, I believe). It wasn't available in druntime, and I didn't get around to fixing that (IIRC, because I started looking into the whole problem of how to deal with windows bindings in druntime in general and going down a rat hole that I didn't have time for). So, I never finished that pull request, and I really should get back to it. But I think that what we need is a function in std.stdio (e.g tempFile insteaf of tmpfile) which returns an open File with a randomly generated name and gives you access to its name rather than using C's tmpfile, which does not give you access to the name and deletes the file on you when it's closed. IMHO, tmpfile is pretty useless - especially when it comes to unit tests. - Jonathan M Davis
Jan 17 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 17 January 2015 at 21:32:18 UTC, Jonathan M Davis 
via Digitalmars-d-learn wrote:
 But I think that what we need is a function in std.stdio (e.g 
 tempFile
 insteaf of tmpfile) which returns an open File with a randomly 
 generated
 name and gives you access to its name rather than using C's 
 tmpfile, which
 does not give you access to the name and deletes the file on 
 you when it's
 closed.
Right - I overlooked this fact. The bad thing is that you might even be forced to close the file before another program can open it, if either of the programs wants to open it exclusively (probably most likely to happen on Windows).
Jan 18 2015