none
FileSystemWatcher RRS feed

  • Question

  • I have developed a Windows service, utilising the FileSystemWatcher, to monitor a directory structure for new files.

    On reciept of a file, it will validate the file and the extension against a setup, persisted in a MSSql database, and on succesful validation, initialise a SSIS package that will persist the contend to a database table.

    The service is deployed to a SQL server, monitoring a seperate file server, after being stress tested in exess of the amount/size of files recieved in the production domain.

    Worth noting, that the problem to be described occurs during low as well as high volume file changes.

    Synopsis: The solution works fine, but every now and again (seemingly in a random manner) will not fire events.  Checking the service, it shows as running. 

    On stopping and starting the service, and placing the file in the folder, it will then import the file, and carry on doing so for a while, till, all of a sudden, it siezes operations again.

    Any ideas or pointers to what I can focus on for investigation.

    Much appreciated

    Tuesday, September 18, 2012 12:10 PM

Answers

  • I suspect that because it is monitoring a separate file server, there is some break in the file share connection that disables the FileSystemWatcher events. My advice is to try to run a test service on the local system to see if the problem goes away.

    Dan Randolph - My Code Samples List

    • Marked as answer by code(monkey) Wednesday, September 26, 2012 12:06 PM
    Tuesday, September 18, 2012 10:50 PM

All replies

  • I suspect that because it is monitoring a separate file server, there is some break in the file share connection that disables the FileSystemWatcher events. My advice is to try to run a test service on the local system to see if the problem goes away.

    Dan Randolph - My Code Samples List

    • Marked as answer by code(monkey) Wednesday, September 26, 2012 12:06 PM
    Tuesday, September 18, 2012 10:50 PM
  • Monitoring of files using FileSystemWatcher (which uses the Win32 API ReadDirectoryChangesW under the covers) is not 100% reliable.  There are a variety of scenarios that can cause notifications to be lost, especially when monitoring a folder on another computer.

    Typically you must have a backup-plan to exhaustively check the monitored folder(s) on a schedule to look for files that you missed. 


    -cd Mark the best replies as answers!

    Tuesday, September 18, 2012 11:53 PM
    Moderator
  • @Dan, even before reading your reply, I came to the same conclusion last night as during functional testing, which was exhaustive, everything was done on a local box, and we never had the problem.

    @Carl, thanks, I also read this worthwhile thread: http://social.msdn.microsoft.com/Forums/en-US/netfxbcl/thread/4465cafb-f4ed-434f-89d8-c85ced6ffaa8/ , but the way my issues manifested themselves, was not the same. Worth noting is the "not 100% reliable" like you stated.

    I will for reference update the thread with my findings.

    Wednesday, September 19, 2012 6:30 AM
  • Hi Code,

    Welcome to the MSDN Forum.

    Did you notice this: http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx

    Note that a FileSystemWatcher may miss an event when the buffer size is exceeded. To avoid missing events, follow these guidelines:

    • Increase the buffer size by setting the InternalBufferSize property.

    • Avoid watching files with long file names, because a long file name contributes to filling up the buffer. Consider renaming these files using shorter names.

    • Keep your event handling code as short as possible.

    Best regards,


    Mike Feng
    MSDN Community Support | Feedback to us
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, September 19, 2012 10:36 AM
    Moderator
  • Others have described the long standing problems with ReadDirectoryChangesW, the underpinning of FileSystemWatcher.  I won't add to that.  So, what should you do about it?

    IMO, the way to go is to use both a timer and FileSystemWatcher.  When FileSystemWatcher detects what you want to detect, have it manipulate the timer to cause an immediate tick event.  In the tick event, examine the directory and process everything that you see, and then reset the timer interval to some reasonable value (eg something between 10sec and 10min).  In this way, you get a workaround for the dropped events.  You will get quick response when FileSystemWatcher does raise an event, you will get slower response when it fails.  You will get robustness in that you will always eventually process every new file that comes your way.

    Wednesday, September 19, 2012 12:02 PM
  • @Mike..we did experiment with an increased buffer size, but the issue happens even when a single, small file (75kB) gets submitted.

    The one consistency is that it will happen after a period of inactivity (day or two)

    The one thing that I take away from your post is the short filename size. Most of the files are generated by users from other applications, and they do tend to name it e.g. Underlying Fund Manager returns by Joe Doe on 20120331.csv.

    This naming "convention" did not  form part of our test pack.

    Thank you.

    Wednesday, September 19, 2012 12:27 PM
  • Hi Code(monkey),

    How about your issue now?

    Does the file name convention help you?

    Best regards,


    Mike Feng
    MSDN Community Support | Feedback to us
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Monday, September 24, 2012 1:40 PM
    Moderator
  • Some feedback, based on a week of Business as Usual testing:

    Moved the source folder onto the server that hosts the Filewatcher service, and to date, not a single issue. 

    Budgetary constraints prevents a full investigation as to what the issue was (those that hold the purse strings, pull it tight when stuff works)

    @Mike, for testing we did not implement shorter filename convention, and had no issues, but the files are small, and the volumes low.  I did simulate a high volume, small file, long filename test, and had one failure, but could not reproduce it again.

    We are now following a convention of shorter filenames as sound business practice. 

    Crucial to a filewatcher service imho is sound error trapping and handling. For complete redundancy a timer as described by @amercer would be the way to go.

    Wednesday, September 26, 2012 12:06 PM