none
MoveFileEx(MOVEFILE_REPLACE_EXISTING) + NTFS + same volume = atomic? RRS feed

  • Question

  • I would like to know, if MoveFileEx can provide atomic behaviour if certain conditions are met. The MSDN documentation does not say anything about this one way or the other.

    This question arises often when atomic modifications to files are required, but my searching the internet didn't yield any conclusive results. Some people claim that MoveFileEx can *never* provide atomic behaviour, no matter what circumstances, but usually without any references/proof. Other people claim it can, but again without references/proof.

    Now, I *think* that MoveFileEx should indeed be atomic, if the following conditions are met:

    • dwFlags = MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH
    • lpExistingFileName and lpNewFileName point to files on the same volume
    • that volume uses NTFS

    MOVEFILE_WRITE_THROUGH might not be required, and maybe additional constraints have to be met.

    The only reason that I could imagine why it would not be atomic, is if the "delete" and "rename" parts of the operation are indeed performed individually. And it would really surprise me if that were the case.

    Or maybe, if a true atomicity guarantee cannot be provided, there could be a somewhat weaker guarantee. The only really important thing in most situations is, that no matter what happens (power loss etc.), a file called "lpNewFileName" will always exist after the operation (if it existed before), and that the contents/attributes/ACLs/... of "lpNewFileName" will be that of either one of the involved files.

    It would be great if someone from Microsoft could answer this, and if possible, also update the MSDN documentation for MoveFileEx. Even a guarantee under very restrictive conditions would be far better than no information. Or, if no such guarantee can be provided, a definitive statement to that fact would also help greatly.

    Friday, September 2, 2011 6:42 PM

All replies

  • I am confused a bit.  I believe the file will be locked when you operate on it.  For example, if you open the file for reading in an application, a lock is in place that disallows deletion or moving, for example.  The converse, AFAIK, is also true:  If you are moving a file, a lock is in place so nobody can access it because it is changing locations.  If this is the case, doesn't this, in practice, meet your need to be atomic?

    Or am I completely off here??


    MCP
    Friday, September 2, 2011 8:48 PM
  • With "atomic" I mean "all or nothing" behaviour in face of things like a sudden powerloss or other unexpected system failure (BSOD or what not).

    • Edited by Paul Groke Friday, September 2, 2011 11:16 PM
    Friday, September 2, 2011 11:16 PM
  • Ah, I see.  I don't think that is ever possible to be guaranteed.  The moving of a file consists of two basic operations:  Copy and delete.  Even bypassing the Windows write cache and writing directly to the disk, the disk itself could be caching data and that is beyond the control of Windows.  Furthermore, power loss could occur at the exact time the copy operation completes but the delete is about to be issued.  But that's just my opinion.

    I would also like to hear from someone @ Microsoft.  Guys!!!


    MCP
    Saturday, September 3, 2011 12:42 AM
  • NTFS journals metadata changes, so it would be entirely possible that MoveFileEx is atomic. That takes care of the disk caches and what not. If the "delete" and the "rename" part are committed to the journal together, then MoveFileEx will be atomic. If they are committed one after another, then it will not be.
    Saturday, September 3, 2011 11:05 PM
  • pgroke wrote:
    >
    >With "atomic" I mean "all or nothing" behaviour in face of things like
    >a sudden powerloss or other unexpected system failure (BSOD or what not).
     
    When you move a file within the same volume, it is done as a directory
    update.  The name is added to the new directory, and removed from the old
    directory.  (You can demonstrate that by moving an extremely large file. It
    happens instantaneously, which could not be the case if it were copying
    contents.)
     
    Because NTFS is a journalling file system, even if there is a power
    failure, you'll either get both updates or neither update.
     
    So, although I doubt you'll ever find a Microsoft employee willing to admit
    it in public, the behavior is atomic.
     
    Now, if there are unauthorized file system filters in place, that might
    ruin things, but in an environment where you care about these things, that
    won't happen.
    --
    Tim Roberts, timr@probo.com
    Providenza & Boekelheide, Inc.
     

    Tim Roberts, DDK MVP Providenza & Boekelheide, Inc.
    Saturday, September 3, 2011 11:24 PM
  • I have similar concern and wondering why MS provides MoveFileTransacted function where from what I can see MoveFileEx appears to be an update to metadata (MFT table).
    I find this quote "NTFS does have recovery at the metadata level - in other words, updates concerning file system metadata are always atomic." in this article  How to do atomic writes in a file - Antimail - Site Home - MSDN Blogs .

    I ran a quick test calling MoveFileEx("C:\\18GB.vhd", "C:\\test\foo.vhd", MOVEFILE_REPLACE_EXISTING) and it finished in milliseconds. Seems like a metadata update and not an actual delete and copy file operation.

     

     

    Thursday, September 8, 2011 6:14 AM
  • When you move a file within the same volume, it is done as a directory
    update.  The name is added to the new directory, and removed from the old
    directory.  (You can demonstrate that by moving an extremely large file. It
    happens instantaneously, which could not be the case if it were copying
    contents.)
     
    Because NTFS is a journalling file system, even if there is a power
    failure, you'll either get both updates or neither update.
     
    So, although I doubt you'll ever find a Microsoft employee willing to admit
    it in public, the behavior is atomic.


    The thing is: a replacing move can be done at one or two operations at the metadata-level. Either "replacing rename" in one atomic step, or "delete" and "rename" in two distinct steps. Being that it's not documented anywhere, you cannot just assume that it's done in one atomic "replacing rename", only because NTFS does metadata-journalling.

    Metadata-journalling only guarantees that the file system will always be in a consistent state. However it would also be a "consistent state" if both files are gone, or the "target" file is deleted while the "source" file still exists with it's old name. An "unexpected" state yes, an undesireable one sure, but not inconsistent.

    At least that's my understanding of the matter.

    Thursday, September 8, 2011 7:57 PM
  • @RPrep

    With TxF you can do much more than just "single" atomic operations: you can make multiple operations on the file system atomic. So, using MoveFileTransacted, you can rename a dozen files in one atomic operation. With MoveFileEx you surely cannot.


    ps: I also found the "How to do atomic writes in a file" link, and it does not use MoveFileEx like I intend to. The "protocol" they're using requires explicit recovery steps, which if skipped, will result in a missing file. That makes it a no-go for what I want to do, because I have to guarantee that the "target" file always exists with it's proper name.

    Doing just one single MoveFileEx to replace the target file with a new one requires no recovery, if one can accept that the temp-file can linger until the next update operation (which would be perfectly fine with me)

    • Edited by Paul Groke Thursday, September 8, 2011 8:04 PM
    Thursday, September 8, 2011 7:59 PM
  • If the MoveFile stays on the same volume and no incompatible flags are specified then MoveFile will TRY to fulfill the request by calling ZwSetInformationFile(Rename), which may or may not be atomic depending on the filesystem involved. But if ZwSetInformationFile fails or if your MoveFile request is incompatible with ZwSetInformationFile for any reason, MoveFile will silently fall back to the normal CopyFile + DeleteFile, which is not atomic. That means you can never rely on MoveFile to be atomic. If it succeeds, it probably was atomic, but it might fail halfway.

    It is very possible for MoveFile to be atomic, but it is also possible for something to go wrong and for it to fall back to a non-atomic behavior even if the underlying filesystem supports atomic rename.

    If you were to use the unsupported and semi-documented NtSetInformationFile function directly, you could avoid any possibility of falling back to CopyFile + DeleteFile. You would still be at the mercy of the filesystem's implementation.

    I would guess that all major filesystems (including NTFS and FAT) implement rename-within-the-same-directory as a single metadata operation. Metadata operations are always atomic (they either happen, do not happen, or corrupt the filesystem). Note that NTFS differs from FAT only in that it promises to not corrupt the filesystem in the case of an interrupted metadata operation.


    Tuesday, February 21, 2012 4:13 AM
  • So based on this logic and https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/nf-ntifs-ntsetinformationfile and https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/content/ntifs/ns-ntifs-_file_rename_information , a rename with overwrite can be considered an atomic operation by NTFS. Just the docs for MoveFileEx() don’t say that it uses that (even though that’s probably the most likely operation on NTFS).
    Tuesday, February 20, 2018 4:52 PM