none
backup tool (revisited) continued... RRS feed

All replies

  • Hi Frank

    slight issue with FileTypeFilters - needs a slight amendment to ignore case

    e.g. if I want to include say just . jpg then it works but doesn't do .JPG


    Darren Rose

    Sunday, February 26, 2017 7:07 PM
  • Hi Frank

    slight issue with FileTypeFilters - needs a slight amendment to ignore case

    e.g. if I want to include say just . jpg then it works but doesn't do .JPG


    Darren Rose

    My mother-in-law is on her way here to see the new place (yea ... my thoughts exactly).

    I'll have a look at that later this afternoon or tomorrow.


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 7:09 PM
  • My mother-in-law is on her way here to see the new place (yea ... my thoughts exactly).

    I'll have a look at that later this afternoon or tomorrow.

    Enjoy..... ;)

    Thanks


    Darren Rose

    Sunday, February 26, 2017 7:10 PM
  • Hi Frank

    slight issue with FileTypeFilters - needs a slight amendment to ignore case

    e.g. if I want to include say just . jpg then it works but doesn't do .JPG


    Darren Rose

    Darren,

    I *think* this will do what you want. If not then I'll look more deeply tomorrow:

    http://www.fls-online.com/VBNet_Forum/02-26-17/LocalFileCopier.zip


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 7:44 PM
  • Hi Frank

    slight issue with FileTypeFilters - needs a slight amendment to ignore case

    e.g. if I want to include say just . jpg then it works but doesn't do .JPG


    Darren Rose

    Darren,

    I *think* this will do what you want. If not then I'll look more deeply tomorrow:

    http://www.fls-online.com/VBNet_Forum/02-26-17/LocalFileCopier.zip


    "One who has no vices also has no virtues..."

    Thanks Frank - that works perfectly

    I have one odd issue - if I use the Test app then it copies empty directories fine if I have it set as below, but in my app using the latest DLL even with it set as below it doesn't do them - a long shot I know but any thought as to why?

     fileCopier = New FileCopier() _
                        With {.CompletionEstimateDelaySeconds = 30,
                              .EmptyDirectories = e.EmptyDirectories}
    

    As you can see now it is in my app it is really get tested extensively - I have been testing every possible combination of options all day now just to make sure everything perfect - hence these late findings :)


    Darren Rose

    Sunday, February 26, 2017 8:10 PM
  • Darren,

    You mean that it's not creating the empty directories, right?

    I'm not 100% sure what you mean, honestly.


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 8:16 PM
  • Darren,

    You mean that it's not creating the empty directories, right?

    I'm not 100% sure what you mean, honestly.


    "One who has no vices also has no virtues..."

    Yes that is correct even with option set it doesn't create the empty directories

    Is strange as compared settings between my app and your test app and am using EXACTLY the same settings, even compared the XML_Results.xml files to confirm both my app and your app showing same paths etc - but with Test_LocalFileCopier it works, but with my app using the DLL you supplied it doesn't


    Darren Rose

    Sunday, February 26, 2017 8:23 PM

  • Yes that is correct even with option set it doesn't create the empty directories

    Is strange as compared settings between my app and your test app and am using EXACTLY the same settings, even compared the XML_Results.xml files to confirm both my app and your app showing same paths etc - but with Test_LocalFileCopier it works, but with my app using the DLL you supplied it doesn't


    Darren Rose

    Sounds like fun to find out what's going on, so I'll look at it later in the week as time permits.

    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 8:25 PM
  • Sounds like fun to find out what's going on, so I'll look at it later in the week as time permits.


    "One who has no vices also has no virtues..."

    Okay I have managed to "recreate" the problem with the test app as well

    The empty folder that it was not creating wasn't actually empty! - it had a hidden file in it called desktop.ini and I had that add as a .FilenamesToExclude - so even though I thought the folder was empty the program didn't

    Not sure if this will be fixable as it is obviously confusing the system

    To recreate when you have time - create a folder with one file in it and then add that file as an exclusion

    For example these were settings I used below and my folder structure in source was a couple of pictures (1 .jpg and 1 .JPG to test other fix) and two folders, one full of photos and one with just a desktop.ini file in it

      .AddDirectoryPath("C:\users\darren rose\pictures", SearchOption.AllDirectories, "z:\backup")
    
                    .FilenamesToExclude.Add("desktop.ini")
    
                    .PerformDuplicateCheck = False
    
                    .AlwaysOverwriteTarget = False
    
                    .CreatePaths()

    Anyway I shall leave you alone now to enjoy your visitor ;)


    Darren Rose

    Sunday, February 26, 2017 8:55 PM
  • Darren,

    If it's hidden or a system file then that's exactly why -- it's never found and you don't want to open that Pandora's box!


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 9:04 PM
  • Actually nothing to do with hidden or system - in my case it was hidden hence why I didn't initially realise

    But if you create a folder and put any file in it (not hidden/system) and then exclude files of that name - then it doesn't see folder as empty and thus doesn't recreate the folder in the targer


    Darren Rose

    Sunday, February 26, 2017 9:11 PM
  • Actually nothing to do with hidden or system - in my case it was hidden hence why I didn't initially realise

    But if you create a folder and put any file in it (not hidden/system) and then exclude files of that name - then it doesn't see folder as empty and thus doesn't recreate the folder in the targer


    Darren Rose

    Oh.

    Boy - I don't know off-hand. I'll look into it later this week then.


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 9:15 PM

  • But if you create a folder and put any file in it (not hidden/system) and then exclude files of that name - then it doesn't see folder as empty and thus doesn't recreate the folder in the targer


    Darren Rose

    The more that I think about it, the more I disagree:

    That directory doesn't match any of the criteria, so why would it be re-created? It's not empty.


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 9:26 PM

  • But if you create a folder and put any file in it (not hidden/system) and then exclude files of that name - then it doesn't see folder as empty and thus doesn't recreate the folder in the targer


    Darren Rose

    The more that I think about it, the more I disagree:

    That directory doesn't match any of the criteria, so why would it be re-created? It's not empty.


    "One who has no vices also has no virtues..."


    Okay fair point - its not empty, but it may as well be in this case as only contains a useless desktop.ini which I exclude - but I get what you mean

    Darren Rose

    Sunday, February 26, 2017 9:43 PM
  • p.s. on that last DLL (which fixed the uppercase/lowercase file extensions) you might want to increment version number?

    Either way it works fine now which is good, so please let me have project download again with that fixed so I can keep my files up to date

    Thanks again for looking - always appreciated - hope I didn't drag you away from visitor too long...... :)


    Darren Rose

    Sunday, February 26, 2017 9:52 PM
  • p.s. on that last DLL (which fixed the uppercase/lowercase file extensions) you might want to increment version number?

    Either way it works fine now which is good, so please let me have project download again with that fixed so I can keep my files up to date

    Thanks again for looking - always appreciated - hope I didn't drag you away from visitor too long...... :)


    Darren Rose

    I'll do that in the morning.

    *****

    No, as it turns out (I should have thought it through, given the date), the ulterior motive was the time of year: Elaine (my wife) did her mom's taxes and filed them online and all that.

    ~whatever~

    ;-)


    "One who has no vices also has no virtues..."

    Sunday, February 26, 2017 10:00 PM
  • Darren,

    Here ya go:

    http://www.fls-online.com/VBNet_Forum/02-27-17/Test_LocalFileCopier.zip


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 11:54 AM
  • Darren,

    Here ya go:

    http://www.fls-online.com/VBNet_Forum/02-27-17/Test_LocalFileCopier.zip


    "One who has no vices also has no virtues..."


    Thanks Frank

    Darren Rose

    Monday, February 27, 2017 11:57 AM


  • Thanks Frank

    Darren Rose

    You're welcome. :)

    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 12:05 PM
  • Hi Frank

    Here is an odd one for you - using your test app I have the following set

                With pathCreator
                    .AddDirectoryPath("e:\photos\photos 3", SearchOption.AllDirectories, "z:\backup")
    
                    .FileFilter.Filters.Add(".jpg")
    
                    .FileFilterType = FileAndDirectoryPaths.FileFilterMeaning.Inclusions
    
                    .PerformDuplicateCheck = False
    
                    .AlwaysOverwriteTarget = False
    
                    .CreatePaths()
                End With

    But it stops at following line

     If e.OperationException IsNot Nothing Then
                Stop
            Else

    With this strange error - any ideas??


    Darren Rose

    Monday, February 27, 2017 1:47 PM
  • Hi Frank

    Here is an odd one for you - using your test app I have the following set

                With pathCreator
                    .AddDirectoryPath("e:\photos\photos 3", SearchOption.AllDirectories, "z:\backup")
    
                    .FileFilter.Filters.Add(".jpg")
    
                    .FileFilterType = FileAndDirectoryPaths.FileFilterMeaning.Inclusions
    
                    .PerformDuplicateCheck = False
    
                    .AlwaysOverwriteTarget = False
    
                    .CreatePaths()
                End With

    But it stops at following line

     If e.OperationException IsNot Nothing Then
                Stop
            Else

    With this strange error - any ideas??


    Darren Rose

    It looks like it didn't find anything to backup?

    I'm guessing here - I don't know off-hand.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 1:52 PM
  • P.S. error only happens if I am doing file filter inclusions - works fine if not

    Darren Rose

    Monday, February 27, 2017 1:52 PM
  • P.S. error only happens if I am doing file filter inclusions - works fine if not

    Darren Rose


    I'll try to duplicate it later on and if I can then I'll see what's going on with it.

    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 1:58 PM
  • It looks like it didn't find anything to backup?

    I'm guessing here - I don't know off-hand.


    "One who has no vices also has no virtues..."

    Definitely content in that folder - and if I try and backup an empty location I get different error anyway

    Darren Rose

    Monday, February 27, 2017 2:05 PM

  • Definitely content in that folder - and if I try and backup an empty location I get different error anyway

    Darren Rose

    Give this a try please?

    http://www.fls-online.com/VBNet_Forum/02-27-17/LocalFileCopier.zip

    Use the same settings and run it to the end and let's go from there.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 2:19 PM
  • That seems to work fine and now I don't get error and files copy okay - what was the problem?

    *****

    BUT when using the filefilter there is a discrepancy in the files copied

    Let me explain

    In that folder I have 357 jpeg files but only 355 of them get copied with filefilter set to include *.jpg

    If I do same copy without the filefilter set then all 357 are copied

    If I look at the XML file then for example in directory below are two jpg files - with filters on it shows just one in XML log

    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image3.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image3.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>

    but with filters off it shows two?

    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image3.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image3.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>
    
    
    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image4.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image4.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>

    As you can see they are both .JPG files? so should be included with the filefilter type inclusion

    N.B. Had this issue before last changes but was testing it on multiple sources before mentioning it

    I can replicate it on other folders containing photos, for some reason with filters on it just misses the odd one or two files


    Darren Rose

    Monday, February 27, 2017 2:37 PM
  • That seems to work fine and now I don't get error and files copy okay - what was the problem?

    *****

    BUT when using the filefilter there is a discrepancy in the files copied

    Let me explain

    In that folder I have 357 jpeg files but only 355 of them get copied with filefilter set to include *.jpg

    If I do same copy without the filefilter set then all 357 are copied

    If I look at the XML file then for example in directory below are two jpg files - with filters on it shows just one in XML log

    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image3.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image3.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>

    but with filters off it shows two?

    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image3.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image3.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>
    
    
    -<Item>
    
    <SourceFilePath>E:\photos\photos 3\Maybank\April 2016\image4.JPG</SourceFilePath>
    
    <TargetFilePath>Z:\backup\photos\photos 3\Maybank\April 2016\image4.JPG</TargetFilePath>
    
    <CopyAction>CopyIfSourceIsNewer</CopyAction>
    
    </Item>

    As you can see they are both .JPG files? so should be included with the filefilter type inclusion

    N.B. Had this issue before last changes but was testing it on multiple sources before mentioning it

    I can replicate it on other folders containing photos, for some reason with filters on it just misses the odd one or two files


    Darren Rose

    I have no idea how that could happen.

    When did it change? Was it after the casing of the file filters that I changed over the weekend?


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 2:50 PM
  • Darren,

    So far I haven't been able to duplicate it here and I've tested it on several folders with a variety of .jpg/.JPG files in various folders.

    Is it always the same file(s) that it skips?


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 3:23 PM
  • I have tried it on a variety of source files and it happens with most of them - and yes always seems to be the same file/files it misses

    Have checked permissions, attributes, read-only etc and all the same as the files it successfully finds

    Have created a zip of photos which wherever I copy them from fails every time for 1 file - so perhaps download and try it see if we can replicate it your end?

    The settings I used were

       With pathCreator
                    .AddDirectoryPath("D:\Photos 4\Maybank", SearchOption.AllDirectories, "z:\backup")
    
                    .FileFilter.Filters.Add(".JPG")
    
                    .FileFilterType = FileAndDirectoryPaths.FileFilterMeaning.Inclusions
    
                    .PerformDuplicateCheck = False
    
                    .AlwaysOverwriteTarget = False
    
                    .CreatePaths()
                End With

    The file is here = http://www.pcassistonline.co.uk/TEST/photos 4.zip

    The file it always seems to miss is D:\Photos 4\Maybank\April 2016\image4.JPG

    http://www.pcassistonline.co.uk/TEST/xml_results.xml


    Darren Rose

    Monday, February 27, 2017 3:44 PM
  • Darren,

    I have the zip file downloaded so you can delete that if you need to.

    I'll get to it today but work has to come first. It has me curious too though so I'll let you know.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 3:56 PM
  • Darren,

    Try this version please and let me know? If this works - I think it will - then I'll have to rework another section that I'll explain, but let me know first:

    http://www.fls-online.com/VBNet_Forum/02-27-17/LocalFileCopier_v2.zip


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 4:28 PM
  • yes that works :)

    What was the issue?


    Darren Rose

    Monday, February 27, 2017 4:55 PM
  • yes that works :)

    What was the issue?


    Darren Rose

    That's a temporary fix just to be sure - I'm working on a permanent fix now.

    When you set up the filter, you proved out something that I added after the fact: The part that adds empty folders. Remember that this part doesn't make the assessment; it adds a "fake path" and therein lies the problem because I hadn't thought through all the combinations.

    I'm sure that you know your images and the directory structure so this is familiar I'm sure:

    The file that magically disappeared is the last one in the directory that I have highlighted. The next directory looked at (August 2011) is empty in that there are no files and it then added the "fake file path".

    It then next went to the fltering section which normally isn't on but you set it on so ok now it looks and REMOVES it by telling it not to proceed to the next part. That's what happened; it added it then a few steps later, removed it!

    *****

    This is what I'm testing now:

    Dim fakeSourceFilePath As String = Combine(dirPath, Utilities.PlaceholderFileName) ' ================================================== ' Feb 27, 2017: Added the optional parameter to skip ' filtering for these "fake" entries. ' ================================================== AddToSourceAndTarget(fakeSourceFilePath, dcs.TargetDirectoryPath, dcs.TargetPathMethod, True)


    That way the "fake file path" will always be added, no matter whether you've told it to filter or not.

    So far so good but I want to test more.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 5:04 PM
  • yes that make sense - and I can confirm similar layout of directories in my other test sources as well!

    Darren Rose

    Monday, February 27, 2017 5:15 PM
  • yes that make sense - and I can confirm similar layout of directories in my other test sources as well!

    Darren Rose

    I'm glad that you hit on the right combination.

    I wasn't even thinking about the empty folder part until I realized that of course my "fake file" didn't pass the criteria of a file filter because my fake file doesn't HAVE a file extension! (Do remember that there is no file - it's just a faux file so that the target path will know to be part of the instructions on what to create in the FileCopier class).

    Flawed logic on my part - live and learn.

    ******

    I'll have this ready here in a bit.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 5:23 PM
  • no hurry - also glad I tried the right combination, have been testing this thoroughly last couple of days, so has been good

    Darren Rose

    Monday, February 27, 2017 5:28 PM
  • no hurry - also glad I tried the right combination, have been testing this thoroughly last couple of days, so has been good

    Darren Rose

    I hope this is ready for prime time:

    http://www.fls-online.com/VBNet_Forum/02-27-17/LocalFileCopier_v3.zip

    The last part that I added was that I left the backup intact and went back to the source - added an empty directory - and reran it.

    Even though there's no file that's ever put there, the "fake path" should be created on the target.

    Test that also if you will please.


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 5:31 PM
  • A quick initial test seems to work fine - and also for creating an empty directory and re-running backup

    Will continue to test more thoroughly later and let you know

    Thanks Frank


    Darren Rose

    Monday, February 27, 2017 5:37 PM
  • A quick initial test seems to work fine - and also for creating an empty directory and re-running backup

    Will continue to test more thoroughly later and let you know

    Thanks Frank


    Darren Rose


    Let me know if you find something else. Hopefully not, but ...

    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 5:39 PM
  • Let me know if you find something else. Hopefully not, but ...


    "One who has no vices also has no virtues..."

    Just spent last two hours copying using every combination of setting I could think of, including/excluding filetypes, excluding files and folders and copying to various different locations with empty folders, full folders, 134000 items in one go and everything worked perfectly. Then tried restoring back to a different place using the find/replace for the paths and that also worked like a dream

    So I think I can confidently say this is all working

    Thank you so much Frank

    Perhaps you can as always put link to last test_filecopier project as well for me and then I will wrap this up hopefully

    :)


    Darren Rose

    Monday, February 27, 2017 7:52 PM
  • Darren,

    Here ya go then:

    http://www.fls-online.com/VBNet_Forum/02-27-17/Test_LocalFileCopier_v3.zip


    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 8:02 PM
  • cheers

    Darren Rose

    Monday, February 27, 2017 8:16 PM
  • cheers

    Darren Rose


    :)

    "One who has no vices also has no virtues..."

    Monday, February 27, 2017 8:20 PM
  • A related challenge if you fancy it.... ;)

    Possibly something that most of this class could be used or adapted for

    "The ability to sync files e.g. have got one server with a folder containing lots of installers, drivers etc and need to be able to sync this folder with same location on other servers - so basically same idea really as file copier, but as well as copying files would need to be able to delete anything from the target so that all the targets (6 of them) always keep same files/folder as the source folder does"

    Just a thought as currently use some useless shareware program to do it, but as LocalFileCopier works so well for backups/copies thought perhaps another use for it


    Darren Rose

    Friday, March 3, 2017 12:09 AM
  • ... if you fancy it.... ;)

    "If you can dream—and not make dreams your master;

    If you can think—and not make thoughts your aim;

    If you can meet with Triumph and Disaster;

    And treat those two impostors just the same"

    *****

    If - a powerful means of conditional branching but for this ............ I'll pass. ;-)


    "One who has no vices also has no virtues..."

    Friday, March 3, 2017 1:37 AM
  • ok I will just use tool to copy files and delete manually

    Darren Rose

    Friday, March 3, 2017 10:17 AM
  • ok I will just use tool to copy files and delete manually

    Darren Rose


    Even if I had time (and desire to do this again), I have no way to test it.

    "One who has no vices also has no virtues..."

    Friday, March 3, 2017 4:34 PM
  • ok I will just use tool to copy files and delete manually


    Darren Rose


    Even if I had time (and desire to do this again), I have no way to test it.

    "One who has no vices also has no virtues..."

    That's fine I completely understand

    As for testing you can just test between two local folders i.e. c:\test and d:\test - so instead of just copying from a to b, we are syncing to make sure b same as a.  The networking side I would handle by just mapping a drive to a drive letter so would be no different on network than it is locally - just sync rather than copy

    But as I say I only asked - I didn't expect you to want to, as has been an trial with this one anyway, but I do appreciate all your help with it


    Darren Rose

    Friday, March 3, 2017 4:38 PM