none
Windows CopyFile Function Bug

    質問

  • Issue

    The Windows API functions CopyFile() and CopyFileEx() intermittently fail when both of the following conditions are true:

    1. The source file is read-only
    2. The target file already exists (even if it is only a stub 0-byte file).

    The frequency of occurrence varies depending on different machines. On two machines at my company, it occurs ~3K iterations of CopyFile(). On my dev machine (and VMWare on my dev machine), it occurs ~150K iterations. We know that the issue occurs on both Vista and Windows 7 (XP not tested). Also, when CopyFile() file fails under these conditions, it returns Error_Access_Denied and it also modifies the target file by adding the read-only attribute to it (This means that successive CopyFile() attempts will fail every time since CopyFile is designed to fail when the target is read-only. Therefore, you must mark the file as not read-only before retrying). My guess as to the  cause of this bug is that before the copy is occurring, the target file is, for some reason, intermittently being marked as read-only. This, of course, causes the copy to fail.

    Perhaps this is related to the following Microsoft KB: BUG: CopyFile Fails with Read-Only Files? Note, however, that we are reproducing this on normal use case of Windows 7 and Vista, with no SMB server or any other special setup. I haven't found any other reference to this issue online.

    Steps to Reproduce
    1. Put a read-only file named setup.exe on a Vista or Win 7 machine at the root directory of a hard drive.
    2. Run copyfiletest.exe* with the following command-line:
      copyfiletest.exe [drive letter where you placed setup.exe at the root] > [full path of log file]
      For example:
      copyfiletest.exe c > c:\log.txt
    3. Let it run for 4 - 24 hours (at least 1 million iterations, where each iteration is a line number in the log).
    4. Open the log, and search for "BINGO". If you find it, you reproduced the failure.

    Workaround

    If CopyFile fails with Error_Access_Denied, then retry up to 5 times. I've never seen this failure two times in a row. 

    *Here is the source to copyfiletest.exe:

     

    // copyfiletest.cpp : Defines the entry point for the console application.
    //
    
    #include "stdafx.h"
    #include "atlstr.h"
    #include <iostream>
    
    using namespace std;
    
    int _tmain(int argc, _TCHAR* argv[])
    {
    	for (int i=0; true; i++)
    	{
    		wcout << "(" << i << ")";
    		CString tempFolder, tempFilePath;
    		tempFolder.GetBufferSetLength(260);
    		tempFilePath.GetBufferSetLength(260);
    		if (!GetTempPath(260, tempFolder.GetBuffer()))
    			wcout << "GetTempPath returned failure!" << endl;
    		if (!GetTempFileName((LPCTSTR)tempFolder, _T("nii"), 0, tempFilePath.GetBuffer()))
    			wcout << "GetTempFileName returned failure!" << endl;
    		tempFilePath.ReleaseBuffer();
    		wcout << "About to copy file: " << tempFilePath.GetString() << endl;
    		CString fileToCopy(argv[1]);
    		fileToCopy.Append(_T(":\\setup.exe"));
    		if (!CopyFile(fileToCopy, tempFilePath, FALSE))
    		{
    			wcout << "BINGO!  " << "Last error is:  " << GetLastError() << endl;
    		}
    		SetFileAttributes(tempFilePath, FILE_ATTRIBUTE_NORMAL);
    		DeleteFile(tempFilePath);
    		
    	}
    	return 0;
    }
    
    

     



    • 編集済み DavidG79 2011年9月30日 20:35
    2011年9月30日 19:08

すべての返信

  • Hi,

    I tested CopyFileEx and I succeded in copying a "readonly" file to a target that is already existing. Perhaps you're experiencing something else, like security exceptions. Check your error with GetLastError function to see what's the real problem is, or use SHFileOperation function if you're using Windows Vista and Up.

    HTH,

    -chris


    Mark the best replies as Answers! | Blog: http://devpinoy.org/blogs/cvega
    2011年10月3日 9:14
  • Hi Chris,

    Per my post, it can take up to 150K iterations to occur on my dev box (Win 7). Also, GetLastError returns Error_Access_Denied when this occurs (even though it succeeded the previous 149,999 times under the exact same circumstances).


    david goldberg
    2011年10月3日 12:49
  • Hi David,

    Sorry, I didn't read your repro that you need to do it in 150k iterations.

    I think what you're experiencing is not a bug, but a delay write problem (access denied). This usually happen when the file you're trying to write-to is being used by another process/thread. Perhaps along the lines of those 150k iterations, the destination file for FileCopy is still in use by the previous FileCopy call (handle is still open) hence the access denied.

    My suggestion, maybe before doing the FileCopy operation, test if the file is not in use, by opening the file with GENERIC_WRITE access. If you catched a Win32 error, then most likely the file is is use (or missing).

    HTH,
    -chris


    Mark the best replies as Answers! | Blog: http://devpinoy.org/blogs/cvega
    2011年10月4日 3:23
  • Hi Chris,

    • I created a program which called handle.exe (and wrote it's output to a separate text file) right before every CopyFile() call. Numerous times I reproduced the CopyFile error, while the handle.exe output showed no open handles to the target file.
    • Notice in my program above that I'm calling DeleteFile before I redo my iteration.
    • Instead of CopyFile, I called CreateFile() over the top of the existing target file and never reproduced the issue.
    • I called DeleteFile instead of CopyFile() and never had a problem over an extreme number of iterations relative to the reproduction rate.
    • If it is an open handle issue, why does this only occur when the source is read-only?

    Thanks,

    David


    david goldberg
    2011年10月4日 13:04
  • If there are caches to flush, why doesn't the OS do it before CopyFile completes? At what number of iterations do we have to start worrying about managing the OS cache? Ehen our users hit the error, we were only copying one file. The only reason I'm doing the iterations is to reliably reproduce the issue.

    david goldberg
    2011年10月4日 13:06
  • Hi David,

    If it is an open handle issue, why does this only occur when the source is read-only?

    Open handle issue is just a guess, with large number of iteration and stuff. I wish I have answers to your questions, as I'm interested about this myself. We have a file management software in my company that rely heavily on CopyFileEx function (file system snapshot/restore/setup etc) and thankfully we haven't experience this bug you're talking about, other than occasional security faults.

    If you are just dealing with small files, perhaps you can write a quick file copy operation using std::copy. There is a discussion about this here.

    Cheers,

    -chris


    Mark the best replies as Answers! | Blog: http://devpinoy.org/blogs/cvega
    2011年10月5日 2:40
  • Hello,

    CopyFileEx on MSDN is very clear : http://msdn.microsoft.com/en-us/library/aa363852(VS.85).aspx

    CopyFileEx exists since a long time now. but you need to GetLastError() very carefully.

    All your problems are documented under the remark section.

     

    Remarks

    This function preserves extended attributes, OLE structured storage, NTFS file system alternate data streams, security attributes, and file attributes.

    Windows 7, Windows Server 2008 R2, Windows Server 2008, Windows Vista, Windows Server 2003, and Windows XP:  Security attributes for the existing file are not copied to the new file until Windows Developer Preview and Windows Server Developer Preview. To copy security attributes, use the SHFileOperation function.

    This function fails with ERROR_ACCESS_DENIED if the destination file already exists and has the FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_READONLY attribute set.

    When encrypted files are copied using CopyFileEx, the function attempts to encrypt the destination file with the keys used in the encryption of the source file. If this cannot be done, this function attempts to encrypt the destination file with default keys. If both of these methods cannot be done, CopyFileEx fails with an ERROR_ENCRYPTION_FAILED error code. If you want CopyFileEx to complete the copy operation even if the destination file cannot be encrypted, include the COPY_FILE_ALLOW_DECRYPTED_DESTINATION as the value of the dwCopyFlags parameter in your call to CopyFileEx.

     

    If COPY_FILE_COPY_SYMLINK is specified, the following rules apply:

    • If the source file is a symbolic link, the symbolic link is copied, not the target file.
    • If the source file is not a symbolic link, there is no change in behavior.
    • If the destination file is an existing symbolic link, the symbolic link is overwritten, not the target file.
    • If COPY_FILE_FAIL_IF_EXISTS is also specified, and the destination file is an existing symbolic link, the operation fails in all cases.

    If COPY_FILE_COPY_SYMLINK is not specified, the following rules apply:

    • If COPY_FILE_FAIL_IF_EXISTS is also specified, and the destination file is an existing symbolic link, the operation fails only if the target of the symbolic link exists.
    • If COPY_FILE_FAIL_IF_EXISTS is not specified, there is no change in behavior.

    If you are writing an application that is optimizing file copy operations across a LAN, consider using the CopyFile2 function. CopyFile2 supports high-performance network transfers. CopyFile2 was introduced with Windows Developer Preview and Windows Server Developer Preview.

    Windows 7, Windows Server 2008 R2, Windows Server 2008, Windows Vista, Windows Server 2003, and Windows XP:  If you are writing an application that is optimizing file copy operations across a LAN, consider using the TransmitFile function from Windows Sockets (Winsock). TransmitFile supports high-performance network transfers and provides a simple interface to send the contents of a file to a remote computer. To use TransmitFile, you must write a Winsock client application that sends the file from the source computer as well as a Winsock server application that uses other Winsock functions to receive the file on the remote computer.

    Christophe Pichaud
    2011年10月5日 9:56
  • Hi Christophe,

    If you are referring to the following section of the MSDN doc:

    This function fails with ERROR_ACCESS_DENIED if the destination file already exists and has the FILE_ATTRIBUTE_HIDDEN or FILE_ATTRIBUTE_READONLY attribute set.


    Then that isn't relevant. Again, the issue I'm seeing is:

    The Windows API functions CopyFile() and CopyFileEx() intermittently fail when both of the following conditions are true:

    1. The source file is read-only
    2. The target file already exists (even if it is only a stub 0-byte file).

     


    david goldberg
    2011年10月5日 15:48
  • The Windows API functions CopyFile() and CopyFileEx() intermittently fail when both of the following conditions are true:
    1. The source file is read-only 2. The target file already exists (even if it is only a stub 0-byte file).

    How does your test program reproduce that latter condition - since it
    creates a new temp file name each iteration as far as I can see?

    Dave

    2011年10月6日 12:54
  • GetTempFileName function:

    Creates a name for a temporary file. If a unique file name is generated, an empty file is created and the handle to it is released; otherwise, only a file name is generated.


    david goldberg
    2011年10月6日 13:32
  • GetTempFileName function:

    Creates a name for a temporary file. If a unique file name is generated, an empty file is created and the handle to it is released; otherwise, only a file name is generated.

    OK :)

    I'm trying to reproduce it, but it's not happened so far.

    Dave

    2011年10月6日 14:48
  • Thanks, Dave. I wouldn't start being concerned until you get to 1 or 2 million iterations since the frequency seems to be machine dependent. We have two (slightly older) machines where it happens consistently about every 1K iterations. On my new dev box, it occurs about every 150K iterations.
    david goldberg
    2011年10月6日 15:00
  • >Thanks, Dave. I wouldn't start being concerned until you get to 1 or 2 million iterations since the frequency seems to be machine dependent. We have two (slightly older) machines where it happens consistently about every 1K iterations. On my new dev box, it occurs about every 150K iterations.

    It's now occurred twice for me in a Win7 VM at 52,236 and 52,270
    iterations, but a Win8 VM is still going fine after 560,000. The W7 VM
    is currently OK at 192,800 iterations.

    I've no idea what the conclusion to this is. Clearly it's not a good
    situation to have something fail like this and be so inconsistent - it
    makes developing/testing any file handling code nearly impossible as
    you can't be sure you've tested it thoroughly.

    BTW, from your own testing, is there any significance to the fact that
    the source file is:

    1. named Setup.exe
    2. read-only

    ... I've not tried changing them yet.

    Dave

    2011年10月6日 15:45
  • The W7 VM is currently OK at 192,800 iterations.


    Do you mean that it hasn't yet happened a third time on the W7 VM?

    This is interesting information about it not yet occurring on W8. I'd be interested to see this go to 1-3 million iterations.

    Yes, I no longer have warm fuzzies about using even the most basic Windows API calls.

    The only significance that I'm aware of is that the source must be read-only for this error to occur. I've confirmed it won't occur (with what I deemed to be ample number of iterations) if the source is not read-only. I'm not sure if I tested with another source filename, but I would be shocked if that made a difference.

     


    david goldberg
    2011年10月6日 15:58
  • The W7 VM is currently OK at 192,800 iterations.
    Do you mean that it hasn't yet happened a third time on the W7 VM?

    It hadn't then - but it did fail at 227,409.

    This is interesting information about it not yet occurring on W8. I'd be interested to see this go to 1-3 million iterations.

    I'll leave it going :)

    Yes, I no longer have warm fuzzies about using even the most basic Windows API calls.

    It is indeed very worrying.

    I've come across some spurious/random file handling oddities caused by
    AV products in the past, but they've usually been more easily
    reproduced than this, and my W7 & W8 VM's have no such 3'rd party AV
    installed.

    This is the sort of thing we could do with SysInternals Mark
    Russinovich having a look into.

    The only significance that I'm aware of is that the source must be read-only for this error to occur. I've confirmed it won't occur (with what I deemed to be ample number of iterations) if the source is not read-only. I'm not sure if I tested with another source filename, but I would be shocked if that made a difference.

    So far the W7 VM is ok with a non-R/O file at 425,000 iterations. I'll
    leave that going as well.

    Dave

    2011年10月6日 16:35
  • Thanks, Dave!

    This is the sort of thing we could do with SysInternals Mark
    Russinovich having a look into.

    Wow, yeah. He's a celebrity! How do we make that happen? :)


    david goldberg
    2011年10月6日 16:48
  • >Wow, yeah. He's a celebrity! How do we make that happen? :)

    No idea :)

    FWIW, the W7 VM is currently at iteration 2,793,200, so, with a non
    read-only file it would seem to be behaving fine - though you never
    can tell unless you know what the real cause of the problem is!

    The W8 VM is still going strong at 3,767,300 iterations with no
    problem.

    Does anyone else monitoring this thread think this looks suspiciously
    like an OS issue/bug (and one that MS may have resolved for W8 based
    on my tests)?

    Dave

    2011年10月6日 20:23
  • Thanks for the info, Dave! I certainly didn't get up to 3 million iterations in my non-readonly source testing, so this is a good data point. And it is also useful information that we have good reason to believe this problem doesn't exist in W8.


    david goldberg
    2011年10月6日 20:33
  • Latest results:

    W7 with a writeable source file ran to 11,083,400 iterations before I
    stopped it. I then changed the source file to read-only and it failed
    again at (only) 52,515 iterations.

    W8 was OK with a read-only file to 11,531,300 iterations before I
    stopped it because I realised it was configured as a single processor
    VM whereas my W7 was 2 processors.
    Having reconfigured the W8 VM to 2 procs (to be the same as the W7
    one), it's currently still fine with a read-only source file at
    1,704,300 iterations.

    Based on my results here it looks suspiciously like something MS have
    fixed in W8 (they have done good improvements to file copying
    already), or there's some component not running in W8 that is the
    cause of the problem in W7.

    Dave

    2011年10月7日 10:41
  • Thanks for the info, Dave! I agree with your analysis.

    Does anyone know how we get Microsoft to officially respond to this bug? Seems like they should still push out a Hotfix for it. It will be a long time before my company can drop support for W7 ...


    david goldberg
    2011年10月7日 13:21
  • >Does anyone know how we get Microsoft to officially respond to this bug?

    For "official" support you'd need to open a support call - hopefully
    it is a bug and you wouldn't be charged for it.

    I'll try to see if I can raise some MS response, but don't hold your
    breath.

    Dave

    2011年10月7日 14:12
  • This issue has been raised with Windows Serviceability. We'll take a look and see if we can reproduce the issue locally.
    2011年10月7日 20:06
  • Thanks, Martin! Please let me know what you find.


    david goldberg
    2011年10月7日 20:14
  • >This issue has been raised with Windows Serviceability. We'll take a look and see if we can reproduce the issue locally.

    Hi Martin,

    Have you managed to reproduce it?

    Dave

    2011年10月12日 6:59
  • Any update Martin?
    2011年10月17日 19:36
  • Hi Martin,

    Have you managed to reproduce it?

    The silence is deafening :(

    So, this has either fallen down a black hole to be lost forever, or
    someone's managed to reproduce it and had an "oh s***" moment and
    maybe we'll get a Windows update at some point that fixes the problem.

    To the other David who started this thread... I guess that if you need
    to follow up on this, the only way will be through an official MS
    support case. If you do that and hear something, please let us know.

    Dave

    2011年10月25日 9:18
  • I found the following information from others in my company. This is definitely a related issue, and maybe the larger issue behind the one I reported. However, if it is the same issue, then I don't understand why the CopyFile issue only occurs when the source is read-only. Regardless, anyone interested in the CopyFile issue would be interested in this:

    We are currently investigating an issue on Windows 7 (also on Vista, but harder to reproduce) where an error can occur when you try to create a file with either the CreateFile or fopen function if a file with the same path was recently deleted with either the DeleteFile or remove function. The error occurs only when the attempt to create the file occurs relatively soon after the deletion. When fopen fails, errno is set to EACCES. When CreateFile fails, GetLastError returns ERROR_ACCESS_DENIED. When this error occurs the function GetFileAttributes returns INVALID_FILE_ATTRIBUTES and GetLastError returns ERROR_ACCESS_DENIED.

    This issue could cause problems in any code that tries to create a file after deleting it.

    Workarounds include the following:

    1. Wait after deleting the file before trying to create it. You can call GetFileAttributes repeatedly until GetLastError returns ERROR_FILE_NOT_FOUND. Be careful not to get into an infinite loop though as there are other legitimate reasons why you can continually get ERROR_ACCESS_DENIED. It's probably best to loop only for a fixed amount of time.
    2. Avoid creating the same file that was deleted. Use a different path for the creation and deletion.
    3. Rename the file you are about to delete to the name of a file that does not exist, and then copy onto the original name

    Issues we are still investigating:

    1. We've seen this happen on Vista too, though it happens much less frequently and is less reproducible.
    2. It seems to happen more often if the file is being written to the desktop, though that's not a requirement.

    Additional Information:
    Microsoft Connect: http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=498145


    Also we have just opened a support incident with Microsoft.

    I've been hearing from several groups about similar problems with deletion and recreation or renaming of files to the same name as the deleted file and that they've seen these problems on Vista and XP as well. So I just wanted to give you all an update.

    One group pointed out that the documentation for CreateFile says the following:
    "If you call CreateFile on a file that is pending deletion as a result of a previous call to DeleteFile, the function fails. The operating system delays file deletion until all handles to the file are closed. GetLastError returns ERROR_ACCESS_DENIED."

    My current theory is that, although deletion is probably a synchronous operation (otherwise we'd be seeing the problem occurring a lot more easily than we are), if any other program has the file open, the delete is delayed until that program closes its file handle. In the case of newer OSes like Windows 7, there are a lot more background processes potentially opening a file as soon as it's created. For example, anyone of the following might be opening the file in our test case and resulting in the error we are seeing:

    1. Windows File Explorer - Needs to open files briefly to get the data needed to display them (e.g. icon info, version info, etc.). This also explains why we see the problem more easily when writing the file to the desktop as the Windows File Explorer is probably refreshing data for files visible on the desktop more aggressively.
    2. Windows Prefetching and disk caching services - The algorithm for this might have changed in Windows 7 making it more likely for the file to be open when we delete and recreate it than it was in earlier OSes.
    3. Virus and spyware scanners
    4. file system defragmentation software

    Really anyone of the above or even all of them could be leading to the problem we are seeing. I have a support issue with Microsoft and am planning on having them track down what the cause is in this particular situation. If my theory is correct, it's unlikely that they can or will do anything about this issue so we will likely still have to work around it.

    I reported this to Microsoft. The result is the following:

    1. This can happen on older OSes too. Basically it can happen if some external program such as a virus scanner, malware scanner, or other program is running in the background opening files. It's expected behavior of Windows that if another process has a file open and you delete it, that the file is in some weird indeterminate state where a file with the same path cannot be recreated until all handles to the original file are closed. Sadly, no one expects other programs like virus scanners to be accessing their data files so this issue comes up unexpectedly.
    2. In the particular case we were running into, it was Microsoft's newest anti-malware scanner (msmpeng.exe) that had the file open. It seems like this is more likely to happen with the version of the scanner than comes with Windows 7 at least for our particular test case.
    3. I heard from many groups that ran into this issue in the past (even on earlier OSes than Windows 7). And Microsoft said they themselves work around this issue in places.
    4. I suggested to Microsoft that it would be better if the OS or file system drivers treated deleted files in a special way such that a new file with the same path could be created even when the file that previously had that path was still pending for delete. They could even implement this by renaming files to a GUID when they are deleted. I explained it would be better to workaround the issue at the OS level rather than have every single application which deletes and recreates files have to workaround it.
    5. I got the impression it is very unlikely anything will ever be done regarding this issue and it is considered expected behavior and up to application developers to workaround.
    6. We searched for all code in in our project which deletes files and reproduced the problem in several other cases and added code to workaround it (mostly by retrying for a finite number of tries). 
    7. Sadly there is no good way to tell the difference between getting access denied because of this issue or because of a more legitimate reason (e.g. a real permission or network issue), thus any retry workaround better timeout eventually or it could cause an endless loop.
    2011年10月28日 12:39
  • Apologies for the lack of updates. Please engage Microsoft customer support if you require a fix for Windows 7 or earlier versions.
    2012年2月23日 21:08
  • I am having the same problem with concurrent file copies. It seems that PrivCopyFileExW has a bug. This is the internal implementation of CopyFile inside kernel32.dll. I have posted a repro on SO along with process monitor output and stack traces which shows the issue. It is not related tof virus scanner or anything else. It seems that adding a file to a directory during a copy operation does lock the directory exclusivly which is rather nasty to say the least.

    Yours,

       Alois Kraus

    2012年3月20日 8:31
  • >I am having the same problem with concurrent file copies. It seems that PrivCopyFileExW has a bug. This is the internal implementation of CopyFile inside kernel32.dll. I have posted a reproon SO along with process monitor output  <http://stackoverflow.com/questions/9772970/privcopyfileexw-bug-in-windows>and stack traces which shows the issue. It is not related tof virus scanner or anything else. It seems that adding a file to a directory during a copy operation does lock the directory exclusivly which is rather nasty to say the least.
    I'd be intrigued to know if you can reproduce the issue with the
    Windows 8 beta since MS have done some work in the copy files area.

    I think the only way you'll get some progress on this matter is if you
    contact MS support (by phone).

    Dave

    2012年3月20日 9:47
  • I suggested to Microsoft that it would be better if the OS or file system drivers treated deleted files in a special way such that a new file with the same path could be created even when the file that previously had that path was still pending for delete. They could even implement this by renaming files to a GUID when they are deleted. I explained it would be better to workaround the issue at the OS level rather than have every single application which deletes and recreates files have to workaround it.

    Unfortunately this won't fix the problem either.  A recursive delete still has the problem that all parent directories are blocked since they are not empty.  We encountered a similar problem in Cygwin when calling `rm -r' on big directory trees.  Sometimes an already deleted directory still lingers for some reason a few milliseconds, even if the (supposedly) only process accessing the dir is rm, and the handle to the directory was already closed.  This in turn disallows to delete the parent directory, and rm fails.  Our workaround is to add a loop with a short Sleep and then to try again a small number of times.  This now works always, but the necessity to do so alone is disappointing.

    The right fix would be to change the filesystem drivers to behave like filesystem drivers on POSIX OSes:  When the last link (==directory entry) to a file or directory gets deleted, delete the link, even if the file is still in use.  The processes still using the file can do so, there's just no directory entry left which refers to the file.  This allows both, deleting the parent dir, or reusing the same filename again immediately, even if anopther process is still accessing the file.

    Corinna

    2012年3月20日 9:56
  • I realize there is a continued discussion on the nature of the problem and how to fix it. I don't want to discourage the discussion however I wanted to point out , if you do require a fix, please contact Microsoft support (http://support.microsoft.com/ then select get help now) to actually request a fix.

    Regards,
    Martin

    2012年3月27日 21:22
  • @David: The issue can be reproduced on Windows 8 as well with no problems.

    @Martin: Yes I have opened a support request at MS. They are looking into it but I fear that it will turn out as a will not fix thing since I am quite sure MS does know about this bug for quite some time. I am certainly not the first one who did find this thing.

    Yours, 

        Alois Kraus

    2012年3月29日 6:19