www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Encapsulating Locked File Appends

reply dsimcha <dsimcha yahoo.com> writes:
Is there an easy way to write a function, or a function already written for
me, that will allow for a file shared between processes to be appended to
safely?  This should be done in a way that will make it impossible for two
processes to write to it at the same time, using locking, and should be
platform-independent.

Performance does not matter because, in the use case I have, we're only
talking about one update every few minutes.  Simplicity, however, does matter.
 All I'm trying to do is run a simulation thousands of times on a bunch of
different computers sharing an NFS file system and have all of the results end
up in one nice plain text file instead of having each instance write to its
own file and having to keep track of them and piece them together by hand.
Mar 10 2009
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
dsimcha wrote:
 Is there an easy way to write a function, or a function already written for
 me, that will allow for a file shared between processes to be appended to
 safely?  This should be done in a way that will make it impossible for two
 processes to write to it at the same time, using locking, and should be
 platform-independent.
 
 Performance does not matter because, in the use case I have, we're only
 talking about one update every few minutes.  Simplicity, however, does matter.
  All I'm trying to do is run a simulation thousands of times on a bunch of
 different computers sharing an NFS file system and have all of the results end
 up in one nice plain text file instead of having each instance write to its
 own file and having to keep track of them and piece them together by hand.

Use a second file to represent the lock, with the contents as the lock owner. While the file exists, poll. When it's not there, create it and write the process id into it, then unlink the file when you're done.
Mar 10 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Sean Kelly (sean invisibleduck.org)'s article
 dsimcha wrote:
 Is there an easy way to write a function, or a function already written for
 me, that will allow for a file shared between processes to be appended to
 safely?  This should be done in a way that will make it impossible for two
 processes to write to it at the same time, using locking, and should be
 platform-independent.

 Performance does not matter because, in the use case I have, we're only
 talking about one update every few minutes.  Simplicity, however, does matter.
  All I'm trying to do is run a simulation thousands of times on a bunch of
 different computers sharing an NFS file system and have all of the results end
 up in one nice plain text file instead of having each instance write to its
 own file and having to keep track of them and piece them together by hand.

owner. While the file exists, poll. When it's not there, create it and write the process id into it, then unlink the file when you're done.

Wouldn't you have to somehow atomically poll and create the file? What if some other process created the lock file between your call of exists() and write()?
Mar 10 2009
parent reply Sean Kelly <sean invisibleduck.org> writes:
dsimcha wrote:
 == Quote from Sean Kelly (sean invisibleduck.org)'s article
 dsimcha wrote:
 Is there an easy way to write a function, or a function already written for
 me, that will allow for a file shared between processes to be appended to
 safely?  This should be done in a way that will make it impossible for two
 processes to write to it at the same time, using locking, and should be
 platform-independent.

 Performance does not matter because, in the use case I have, we're only
 talking about one update every few minutes.  Simplicity, however, does matter.
  All I'm trying to do is run a simulation thousands of times on a bunch of
 different computers sharing an NFS file system and have all of the results end
 up in one nice plain text file instead of having each instance write to its
 own file and having to keep track of them and piece them together by hand.

owner. While the file exists, poll. When it's not there, create it and write the process id into it, then unlink the file when you're done.

Wouldn't you have to somehow atomically poll and create the file? What if some other process created the lock file between your call of exists() and write()?

If you use the "create only" flag when opening the file then it should fail if the file was already created by someone else. Unless NFS doesn't provide a sufficiently reliable synchronization mechanism for this to work, that is (I really don't know).
Mar 10 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Sean Kelly wrote:
 dsimcha wrote:
 == Quote from Sean Kelly (sean invisibleduck.org)'s article
 dsimcha wrote:
 Is there an easy way to write a function, or a function already 
 written for
 me, that will allow for a file shared between processes to be 
 appended to
 safely?  This should be done in a way that will make it impossible 
 for two
 processes to write to it at the same time, using locking, and should be
 platform-independent.

 Performance does not matter because, in the use case I have, we're only
 talking about one update every few minutes.  Simplicity, however, 
 does matter.
  All I'm trying to do is run a simulation thousands of times on a 
 bunch of
 different computers sharing an NFS file system and have all of the 
 results end
 up in one nice plain text file instead of having each instance write 
 to its
 own file and having to keep track of them and piece them together by 
 hand.

owner. While the file exists, poll. When it's not there, create it and write the process id into it, then unlink the file when you're done.

Wouldn't you have to somehow atomically poll and create the file? What if some other process created the lock file between your call of exists() and write()?

If you use the "create only" flag when opening the file then it should fail if the file was already created by someone else. Unless NFS doesn't provide a sufficiently reliable synchronization mechanism for this to work, that is (I really don't know).

I've worked a lot with NFS and have the scars and learned the curses to prove it. NFS is as non-deterministic as it gets when it comes about concurrent writes. There is next to no guarantee. The append problem is an absolute classic on NFS. I tried about five different schemes, all failed under mysterious circumstances. What I do now and suggest you do too is to have each different process create its own file. After all processes have ended, have a master process assemble all small files into one. It's really the only thing I got to work. Andrei
Mar 10 2009
parent reply BCS <ao pathlink.com> writes:
Reply to Andrei,

 NFS is as non-deterministic as it gets when it comes about concurrent
 writes. There is next to no guarantee. The append problem is an
 absolute classic on NFS. I tried about five different schemes, all
 failed under mysterious circumstances. What I do now and suggest you
 do too is to have each different process create its own file. After
 all processes have ended, have a master process assemble all small
 files into one. It's really the only thing I got to work.
 
 Andrei
 

IIRC there is a lockd process on most NFS systems. I think it does something for this issue but I don't know what.
Mar 10 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from BCS (ao pathlink.com)'s article
 Reply to Andrei,
 NFS is as non-deterministic as it gets when it comes about concurrent
 writes. There is next to no guarantee. The append problem is an
 absolute classic on NFS. I tried about five different schemes, all
 failed under mysterious circumstances. What I do now and suggest you
 do too is to have each different process create its own file. After
 all processes have ended, have a master process assemble all small
 files into one. It's really the only thing I got to work.

 Andrei

for this issue but I don't know what.

Thanks, but ideally I'd like to do this in a way that it doesn't _have_ to be NFS, since this is a general problem I have and only this instance is on NFS. Ideally, I'd like to write a generic function called lockedAppend that works on both Linux and Windows and is filesystem agnostic. As far as I can tell, this is close to impossible, so maybe I'm better off having all my processes write to separate files. There is no reason why they absolutely _have_ to all write to the same file, it would just be more convenient if they did.
Mar 10 2009
prev sibling parent =?UTF-8?B?QWxleGFuZGVyIFDDoW5law==?= writes:
dsimcha wrote:
 Is there an easy way to write a function, or a function already written for
 me, that will allow for a file shared between processes to be appended to
 safely?  This should be done in a way that will make it impossible for two
 processes to write to it at the same time, using locking, and should be
 platform-independent.
 
 Performance does not matter because, in the use case I have, we're only
 talking about one update every few minutes.  Simplicity, however, does matter.
  All I'm trying to do is run a simulation thousands of times on a bunch of
 different computers sharing an NFS file system and have all of the results end
 up in one nice plain text file instead of having each instance write to its
 own file and having to keep track of them and piece them together by hand.

Ideally you’d use a clustered file system for that, namely GFS (Global File System), OCFS(2) (Oracle, I think) or Lustre (Sun)... but I don’t know what your use-case is so you might not really need it. 0.02€
Mar 10 2009