none
VB.NET 2015 - How to count files having today's date

    Question

  • Hi all,

    I'm a novice with VB.NET, but as a former Access MVP, I'm very familiar with classic VB and VBA. I have a need I hope you can help with.

    Using VB.NET 2015, I need to be able to count the number of files in a specific folder, but only those of a specific filetype and only those that are added today. I need to do this for 3 separate folders where the filetypes vary from folder-to-folder. This needs to be extremely fast because the number of files in each folder is quite large (tens of thousands and growing fast each day). Thus cycling through every file to ascertain it's filetype and create date is prohibitively slow. I tried that first.

    In other parts of the little utility I'm writing, I use FileSystemWatchers, which work well in those scenarios, but that approach doesn't help in this case because, as far as I know, you can't filter the FSW by create date.

    I need to get an accurate count (in as close to realtime as possible) as the utility starts up, as well as when files are added or deleted.

    I hope you can help.

    Regards and many thanks,


    Regards, Graham R Seach Sydney, Australia


    • Edited by Graham R Seach Tuesday, April 18, 2017 3:15 AM Added sig line
    Tuesday, April 18, 2017 3:14 AM

Answers

  • Graham,

    You might want to use something like this as the basis:

    Option Strict On Option Explicit On Option Infer Off Imports System.IO Public Class Form1 Private Sub Form1_Load(ByVal sender As System.Object, _ ByVal e As System.EventArgs) _ Handles MyBase.Load Const directoryPath As String = _ "K:\Personal Folders" Dim sw As New Stopwatch sw.Start() Dim files As IEnumerable(Of String) = _ GetFilePaths(directoryPath, SearchOption.AllDirectories) sw.Stop() If files IsNot Nothing Then MessageBox.Show(String.Format("Files Found (Count): {0:n0}{1}Elapsed Time: {2:f2} Seconds", _ files.Count, vbCrLf, sw.Elapsed.TotalSeconds)) Else MessageBox.Show(String.Format("No Files Match Criteria{0}Elapsed Time: {1:f2} Seconds", _ vbCrLf, sw.Elapsed.TotalSeconds)) End If Stop End Sub Private Function GetFilePaths(ByVal directoryPath As String, _ ByVal searchType As SearchOption) As IEnumerable(Of String) Dim retVal As IEnumerable(Of String) = Nothing Try If String.IsNullOrWhiteSpace(directoryPath) Then MessageBox.Show("The directory path cannot be null or empty.") Else Dim di As New DirectoryInfo(directoryPath) If Not di.Exists Then MessageBox.Show("The directory path could not be located.") Else Dim tempList As New List(Of String) Dim files As IEnumerable(Of FileInfo) = _ di.EnumerateFiles("*", searchType) If files IsNot Nothing Then Dim result As System.Collections.Generic.IEnumerable(Of FileInfo) = _ From fi As FileInfo In files _ Where fi.LastWriteTime.Date = Today.Date If result.Count > 0 Then For Each fi As FileInfo In result tempList.Add(fi.FullName) Next retVal = tempList.ToArray End If End If End If End If Catch ex As Exception MessageBox.Show(String.Format("An error occurred:{0}{0}{1}", _ vbCrLf, ex.Message), _ "Exception Thrown", _ MessageBoxButtons.OK, _ MessageBoxIcon.Warning) End Try Return retVal End Function End Class


    The reason that I'm using DirectoryInfo is because using its EnumerateFiles method will return a collection of FileInfo instances - and that's perfect for this job.

    FileInfo has properties that you can use (like LastWriteTime and Extension) and I show that in my LINQ query to find the files (none in my case).

    For my test, I chose a directory that has 12,562 files in 445 directories:

    Even though it found none that match the criteria, it looked through all of those directories and files in about a quarter of a second. I'm sure that yours will be different but it's worth a try, so try that and let's see how you fare with it.


    "A problem well stated is a problem half solved.” - Charles F. Kettering


    Tuesday, April 18, 2017 1:34 PM

All replies

  • Hi,

    I guess you will make a Windows Forms application, and that users (you?) can select/input a target folder(s) and file-extension(s).
    If this is correct, could you provide an image / layout of a Windows Form which you will make?
    ____________
    Ashidacchi
    Tuesday, April 18, 2017 4:16 AM
  • It's just a little utility for myself, so the folder names are hard-coded. Here's the form. I want to populate the [Today] column. The numbers are put into textboxes.

    


    Regards, Graham R Seach Sydney, Australia


    Tuesday, April 18, 2017 4:21 AM
  • Hi,

    I'd like to confirm:
    (1) "Database", such as "Outlivine AU", is a folder name?
    (2) What is "Ready", "Errors"?  I suppose "Today" is the number of files created today.  Is it correct?
    (3) What is "filetype"?  I was thinking it was file-extension.
    (4) Does a hard-coded folder has its subfolder(s)?
    ___________
    Ashidacchi 
    Tuesday, April 18, 2017 4:30 AM
  • It's just a little utility for myself, so the folder names are hard-coded. Here's the form. I want to populate the [Today] column. The numbers are put into textboxes.

    


    Regards, Graham R Seach Sydney, Australia


    Hi

    Here is some code that seems to do what you want - for one Path - would need another 2 FileSystemWatchers for the other 2 Paths you want to watch. In this example, I use a Dictionary to store  Todays matches. A Count of those would appear to be what you want. The MessageBox is just for testing.

    Option Strict On
    Option Explicit On
    Public Class Form1
        ' set path
        Dim Path As String = "C:\Users\Les\Desktop\Plans"
        Dim ExtToCheck As String = ".txt.doc"
        Dim Store As New Dictionary(Of String, DateTime)
        Private Sub Form1_Load(sender As System.Object, e As EventArgs) Handles MyBase.Load
            Dim FW As New IO.FileSystemWatcher()
            FW.Path = Path
            FW.NotifyFilter = IO.NotifyFilters.LastWrite
            FW.Filter = "*.*"
            AddHandler FW.Changed, AddressOf OnChanged
            FW.EnableRaisingEvents = True
        End Sub
        Private Sub OnChanged(ByVal source As Object, ByVal e As IO.FileSystemEventArgs)
            If ExtToCheck.Contains(IO.Path.GetExtension(e.FullPath)) Then
                If IO.File.GetCreationTime(e.FullPath).Date = Today.Date Then
                    If Not Store.Keys.Contains(e.FullPath) Then
                        Store.Add(e.FullPath, Now)
                        MessageBox.Show("Got it")
                    End If
                End If
            End If
        End Sub


    Regards Les, Livingston, Scotland


    • Edited by leshay Friday, May 05, 2017 11:38 PM
    Tuesday, April 18, 2017 4:35 AM
  • (1) "Database", such as "Outlivine AU", is a folder name?
    Just think of "Database" as a category.

    (2) What is "Ready", "Errors"?  I suppose "Today" is the number of files created today.  Is it correct?
    "Ready" refers to files that are ready to send to our warehouse. "Errors" refers to files that could not be uploaded to our warehouse (due to error). "Today" refers to the number of files that have been uploaded to our warehouse today.

    (3) What is "filetype"?  I was thinking it was file-extension.
    Yes, this means file extension.

    (4) Does a hard-coded folder has its subfolder(s)?
    No, these are all discrete folders.


    Regards, Graham R Seach Sydney, Australia


    Tuesday, April 18, 2017 4:42 AM
  • I'll try your code and report back.

    Thanks.  :-)


    Regards, Graham R Seach Sydney, Australia

    Tuesday, April 18, 2017 4:54 AM
  • Hi Les,

    Your code appears to only capture changed files, not files that exist prior to utility startup.

    Forgive me if I'm wrong; I haven't tested it, but it also appears that the code will never remove a file from the target folder, even if one is deleted.

    What I need is to be able to count todays files as soon as the utility starts, increment the count when a file is added, and decrement the count when one is deleted.

    Thanks,


    Regards, Graham R Seach Sydney, Australia

    Tuesday, April 18, 2017 5:13 AM
  • Hi,

    I've tried:
    Public Class Form1
      Dim Path As String = "C:\temp\Les\Desktop\Plans\"
      Dim aryExt() As String = {"*.txt", "*.doc"}
      ' -- when Loaded, count the number of files (Created on Today)
      Private Sub Form1_Load(sender As System.Object, e As EventArgs) Handles MyBase.Load
        Dim cntToday As Integer = 0
        For i As Integer = 0 To aryExt.Length - 1
          For Each f As String In IO.Directory.GetFiles(Path, aryExt(i))
            'Debug.WriteLine("fileName = " & f)
            If (IsToday(f)) Then
              cntToday += 1
            End If
          Next
        Next
        Me.lbl_Today_1.Text = cntToday.ToString
      End Sub
      ' --- check if Creation Date is today
      Private Function IsToday(ByVal fileName As String) As Boolean
        Debug.WriteLine("Creation = " & IO.File.GetCreationTime(fileName).Date)
        Debug.WriteLine("Write    = " & IO.File.GetLastWriteTime(fileName).Date)
        If (IO.File.GetCreationTime(fileName).Date = DateTime.Now.Date) Then
          Return True
        Else
          Return False
        End If
      End Function
    End Class



    How about this?
    _____________
    Ashidacchi

    • Edited by Ashidacchi Tuesday, April 18, 2017 6:19 AM modified code
    Tuesday, April 18, 2017 5:49 AM
  • Your code appears to only capture changed files, not files that exist prior to utility startup.

    You can't avoid scanning all files in the folder unless you maintain an index of some sort based on a file system watcher, and that watcher would have to be running all the time.  But you would have to run an update procedure at least once to seed the index, and whenever the watcher failed.  It begins to get very complex.

    Is there any reason you would use an app to do this?  Windows search, accessible from Explorer, has a 'Search by Date Modified = Today' built in.  If that won't quite work for you, then you can create a custom search and run that.  The indexer is running constantly, so it's always up to date, it operates on the current folder by default, it gives you a file list and a count and it is extremely fast.  If you really have to use an app, then it should be possible to execute a saved search, or even crate one on the fly.

    Tuesday, April 18, 2017 5:56 AM
  • Hi

    OK.

    From what you say,  files may have been added/removed during the time your app wasn't running. If that is the case, then if the originating machine(s) don't explicitly send information regarding any additions/removals to a central database which your machine could read/update, there is no way to detect those changes without scanning the whole folder incurring the necessary time overhead.

    Another consideration is that from what I have read, FileSystemWatchers are of dubious reliability.


    Regards Les, Livingston, Scotland

    Tuesday, April 18, 2017 6:10 AM
  • Hi Acmar and Les,

    I thought to use an app because, as shown on the screen capture, I need to see counts from 9 separate folders simultaneously. It's kinda hard to do that using Windows Explorer.

    Since the  folders for the third column each contain tens of thousands of files (and growing), performing a file scan of those three folders is very expensive. That's why I originally thought to use FileSystemWatcher. FSW is working OK for the first two columns (Ready and Error), because the files they contain are transient, however, it won't work for the third column (Today) because there are so many of them and they all have the same file extension. For example, all files in the Today folder for 'Outliving AU' are *.zip files, whereas all the files in the Today folder for 'Sunnylife UK' are *.csv. The ones in 'Sunnylife US' are *.txt.

    Of course, the next best thing would be to archive files older than today, but I'm not sure I'll be allowed to do that because the third column is the archive folder. I'll ask the question, but in the meantime, since Windows Explorer uses Windows Search, is there a way I can do that using APIs (or some other method)?

    Thanks,


    Regards, Graham R Seach Sydney, Australia


    • Edited by Graham R Seach Tuesday, April 18, 2017 6:43 AM Amended the 2nd paragraph
    Tuesday, April 18, 2017 6:32 AM
  • I thought to use an app because, as shown on the screen capture, I need to see counts from 9 separate folders simultaneously. It's kinda hard to do that using Windows Explorer.

    You can execute the saved search from code.  For example:

    Process.Start("C:\Users\<username>\Searches\datemodifiedtoday.search-ms")
    
    Arranging the explorer windows might be a bit tricky unless you can encourage explorer to remember its window locations.

    Tuesday, April 18, 2017 7:08 AM
  • Graham,

    What is fast is often a point of discussion. Mostly is taken by a desktop program that if a user recognizes wait time, it is to slow. Test has shown that persons do not aware of wait time shorter than 3 seconds. You think that is long. If you are focusing on it, yes then it seems long, otherwise like other humans you don't even recognize it. 

    You can test it yourself, use windows explorer (file explorer) and click on the properties of your largest drive. You probably do not recognize the time it takes to get the information. 

    Therefore I would say, test this code, and see how long it takes. 

    Public Class Form1
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            Label1.Text = "0"
            Dim files = IO.Directory.GetFiles("d:\test\", "*.txt")
            For Each file In files
                Dim info As New IO.DirectoryInfo(file)
                If info.CreationTime.Date = Now.Date Then Label1.Text = CStr(CInt(Label1.Text) + 1)
            Next
        End Sub
    End Class


    Success
    Cor


    Tuesday, April 18, 2017 10:31 AM
  • Hi all,

    My code took 80 seconds for counting the number of files. The total was 38,244 and 23 files satisfied a condition like Graham's.

    Here's code:
    Public Class Form1
      Private Path As String = "X:\Visual Studio 2017\"
      ' -- when Loaded, count the number of files (Created on Today)
      Private Sub Form1_Load(sender As System.Object, e As EventArgs) Handles MyBase.Load
        Dim StartDT, EndDT As Date
        StartDT = DateTime.Now
        Dim cntHit As Long = 0
        Dim cntAll As Long = 0
        Dim ext As String
        ' ---
        For Each f As String In IO.Directory.GetFiles(Path, "*.*", IO.SearchOption.AllDirectories)
          cntAll += 1
          ext = IO.Path.GetExtension(f)
          If (ext = ".vb") OrElse (ext = ".sln") Then
            If (IsToday(f)) Then
              cntHit += 1
            End If
          End If
        Next
        EndDT = DateTime.Now
        ' ---
        Me.lbl_TargetPath.Text = Me.Path
        Me.lbl_Extension.Text = "*.vb,  *.sln"
        ' ---
        Me.lbl_Count.Text = Format(cntHit, "#,0") & " / " & Format(cntAll, "#,0") & " files"
        Me.lbl_StartEnd.Text = StartDT.ToString & " ~ " & vbCrLf & EndDT
        Me.lbl_Elapse.Text = DateDiff("s", StartDT, EndDT) & " sec."
      End Sub
      ' --- check if Creation Date is today
      Private Function IsToday(ByVal fileName As String) As Boolean
        If (IO.File.GetCreationTime(fileName).Date = DateTime.Now.Date) Then
          Return True
        Else
          Return False
        End If
      End Function
    End Class
    If Acamar or Cor wrote/modified this, it would take less time.

    As Cor mentioned, elapsed time is always in discussion.  I think that 80 sec. would be a long time, if we wait for its ending. I don't know how many times Graham execute a software in a day.
    Suppose it would be a TSR software and have a trigger for counting the number at some regular interval, without human intervention. If Graham make such software, time would not matter, I think.
    ________________________________
    Ashidacchi (sorry my poor English

    P.S. in the above code, target folder is in NAS (home-use, low spec) and includes sub-folders (I don't make over 1,000 files in a single discrete folder).
    Tuesday, April 18, 2017 11:53 AM
  • Hi Acmar and Les,

    I thought to use an app because, as shown on the screen capture, I need to see counts from 9 separate folders simultaneously. It's kinda hard to do that using Windows Explorer.

    Since the  folders for the third column each contain tens of thousands of files (and growing), performing a file scan of those three folders is very expensive. That's why I originally thought to use FileSystemWatcher. FSW is working OK for the first two columns (Ready and Error), because the files they contain are transient, however, it won't work for the third column (Today) because there are so many of them and they all have the same file extension. For example, all files in the Today folder for 'Outliving AU' are *.zip files, whereas all the files in the Today folder for 'Sunnylife UK' are *.csv. The ones in 'Sunnylife US' are *.txt.

    Of course, the next best thing would be to archive files older than today, but I'm not sure I'll be allowed to do that because the third column is the archive folder. I'll ask the question, but in the meantime, since Windows Explorer uses Windows Search, is there a way I can do that using APIs (or some other method)?

    Thanks,


    Regards, Graham R Seach Sydney, Australia


    I am confused.  First it was three folders then nine????  Is it safe to say that generically you are watching folders and want only a count based on some criteria?  Is the criteria a specific date (today) and / or a specific extension?  If you could be more specific.


    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Tuesday, April 18, 2017 12:08 PM
  • Graham,

    You might want to use something like this as the basis:

    Option Strict On Option Explicit On Option Infer Off Imports System.IO Public Class Form1 Private Sub Form1_Load(ByVal sender As System.Object, _ ByVal e As System.EventArgs) _ Handles MyBase.Load Const directoryPath As String = _ "K:\Personal Folders" Dim sw As New Stopwatch sw.Start() Dim files As IEnumerable(Of String) = _ GetFilePaths(directoryPath, SearchOption.AllDirectories) sw.Stop() If files IsNot Nothing Then MessageBox.Show(String.Format("Files Found (Count): {0:n0}{1}Elapsed Time: {2:f2} Seconds", _ files.Count, vbCrLf, sw.Elapsed.TotalSeconds)) Else MessageBox.Show(String.Format("No Files Match Criteria{0}Elapsed Time: {1:f2} Seconds", _ vbCrLf, sw.Elapsed.TotalSeconds)) End If Stop End Sub Private Function GetFilePaths(ByVal directoryPath As String, _ ByVal searchType As SearchOption) As IEnumerable(Of String) Dim retVal As IEnumerable(Of String) = Nothing Try If String.IsNullOrWhiteSpace(directoryPath) Then MessageBox.Show("The directory path cannot be null or empty.") Else Dim di As New DirectoryInfo(directoryPath) If Not di.Exists Then MessageBox.Show("The directory path could not be located.") Else Dim tempList As New List(Of String) Dim files As IEnumerable(Of FileInfo) = _ di.EnumerateFiles("*", searchType) If files IsNot Nothing Then Dim result As System.Collections.Generic.IEnumerable(Of FileInfo) = _ From fi As FileInfo In files _ Where fi.LastWriteTime.Date = Today.Date If result.Count > 0 Then For Each fi As FileInfo In result tempList.Add(fi.FullName) Next retVal = tempList.ToArray End If End If End If End If Catch ex As Exception MessageBox.Show(String.Format("An error occurred:{0}{0}{1}", _ vbCrLf, ex.Message), _ "Exception Thrown", _ MessageBoxButtons.OK, _ MessageBoxIcon.Warning) End Try Return retVal End Function End Class


    The reason that I'm using DirectoryInfo is because using its EnumerateFiles method will return a collection of FileInfo instances - and that's perfect for this job.

    FileInfo has properties that you can use (like LastWriteTime and Extension) and I show that in my LINQ query to find the files (none in my case).

    For my test, I chose a directory that has 12,562 files in 445 directories:

    Even though it found none that match the criteria, it looked through all of those directories and files in about a quarter of a second. I'm sure that yours will be different but it's worth a try, so try that and let's see how you fare with it.


    "A problem well stated is a problem half solved.” - Charles F. Kettering


    Tuesday, April 18, 2017 1:34 PM
  • Hi Frank,

    I've tried your code, 100 % same except target folder.
    This is the result. It was slower than my code.

    # As I already mentioned, my target folder is in NAS (Network Attached Storage). It is for home-use (without high spec), and LAN is with gigabit-ether.
    What I want say is that turn-around time varies by each environment.
    _______________
    Ashidacchi
    Tuesday, April 18, 2017 9:48 PM
  • Hi Frank,

    I've tried your code, 100 % same except target folder.
    This is the result. It was slower than my code.

    # As I already mentioned, my target folder is in NAS (Network Attached Storage). It is for home-use (without high spec), and LAN is with gigabit-ether.
    What I want say is that turn-around time varies by each environment.
    _______________
    Ashidacchi

    Like I said earlier, mine isn't representative- it will vary as you've shown.

    The only way to be "fast" is that it's aleady indexed and that then lends the question about what happens with the file structure changes.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, April 18, 2017 9:53 PM
  • Hi Frank,

    Didn't you execute the code you'd shown and get the result (0.26 seconds)?
    ______________
    Ashidacchi
    Tuesday, April 18, 2017 9:57 PM
  • Hi Frank,

    Didn't you execute the code you'd shown and get the result (0.26 seconds)?
    ______________
    Ashidacchi

    Yes,

    I just ran it again and it will be faster the second (and subsequent) times:

    John Wein used to say that it's because of caching; I'm not sure why so I'll just say that too. Without testing it in detail, it's hard to know where the bottleneck is.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, April 18, 2017 10:03 PM
  • I guess you did run in Windows 7 with large RAM and K-drive is in local SSD. (in my case, Windows 10 with 8 GB RAM, and mapped network drive: RAID-5, HDD)
    _________________
    Ashidacchi
    Tuesday, April 18, 2017 10:11 PM
  • I guess you did run in Windows 7 with large RAM and K-drive is in local SSD. (in my case, Windows 10 with 8 GB RAM, and mapped network drive: RAID-5, HDD)
    _________________
    Ashidacchi

    Every use will be what it is; nothing shown here is representative of what anyone else can expect.

    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, April 18, 2017 10:13 PM
  • You said nothing about your environment that might have a cause of bottleneck.
    Tuesday, April 18, 2017 10:20 PM
  • You said nothing about your environment that might have a cause of bottleneck.

    I assume that you're addressing me? We don't all see this forum the same way, so please make it obvious.

    *****

    It doesn't matter.

    You saw what mine were for my environment and what yours is for your environment. When Graham replies back we'll know what his is for his setup.

    ;-)


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Tuesday, April 18, 2017 10:23 PM
  • Sorry everyone. I haven't read your latest posts and because I've just been given a very tight deadline, I won't be able to respond to read them until tomorrow (Sydney time). I hope you understand. I just wanted to let you know that I haven't abandoned this thread.

    Regards, Graham R Seach Sydney, Australia

    Wednesday, April 19, 2017 1:09 AM
  • Hi Graham,

    If "abandon" means "delete", please do not abandon this thread.
    They provided helpful codes (even if it could not help you). I hope it will remain still.
    _______________
    Ashidacchi
    Wednesday, April 19, 2017 1:19 AM
  • @Ashidacchi

    Only threads with not any reply can be removed by an OP otherwise this is only possible by a Moderator. 

    A moderator can do that but is only allowed to do that if it consists mainly from illegal information or words which are at least not suitable for kids if those reads them.



    Success
    Cor



    Wednesday, April 19, 2017 8:48 AM
  • Hi all,

    I'm sorry for the delay in replying. As I mentioned, I was handed a task that had a very short deadline, and as we all know, you meet deadlines or die trying.

    @Cor: I'm aware of the 3 second rule. I've already tested my code, and it takes about 3.5 minutes to count the files in all 3 of the [Today] folders. Far too long.

    @Ashidacchi: 80 seconds for one folder is far too long, especially when you consider that I'd have to do the same thing for 3 folders. 80 seconds will very quickly increase as more files are added to each folder. This was originally intended as a TSR utility, but I may end up launching it several times during a normal day.

    @JohnWein: It's nine folders in total; one for each "cell" in the matrix. Six of them (the [Ready] and [Errors] rows) populate quickly because they each use separate FileSystemWatchers. The code I'm looking for is only the three textboxes in the [Today] column.

    >>Is it safe to say that generically you are watching folders and want only a count based on some criteria?
    Yes, that's true.

    >>Is the criteria a specific date (today) and / or a specific extension?
    I'm looking for a specific date (today) AND a specific extension. The extension will differ from row-to-row.

    @Frank: I will definitely try your code and report back. Thank you.


    Regards, Graham R Seach Sydney, Australia

    Thursday, April 20, 2017 6:32 AM

  • @Cor: I'm aware of the 3 second rule. I've already tested my code, and it takes about 3.5 minutes to count the files in all 3 of the [Today] folders. Far too long.


    Did you also test my code, it takes probably less than writing this message to open a new windows forms project, drag a label on the surface, copy and past my code, change the path and file extention and than to run it. I don't have folders which contain so much files like you wrote about. 

    I'm curious how long that takes. 


    Success
    Cor

    Thursday, April 20, 2017 7:33 AM
  • @op - I conducted a test to enumerate all of the files in a folder on a NAS.  The folder has 5993 files and I am attached via WIFI.  It took 17 seconds just to enumerate them all.  Would you please run the same test and post the results?

            Dim path As String = "path to folder with 10000 files here"
            Dim di As New IO.DirectoryInfo(path)
            Dim stpw As Stopwatch = Stopwatch.StartNew
            Dim ie As IEnumerable(Of IO.FileInfo) = di.EnumerateFiles("*.*", IO.SearchOption.AllDirectories)
            Debug.WriteLine(ie.Count)
            stpw.Stop()
            Debug.WriteLine(stpw.Elapsed)
    

    This will give us an idea of best performance possible I think.

    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Thursday, April 20, 2017 12:44 PM
  • db,

    How many did yours find?

    I suspect that's where the slow down might be.

    If I go back to it, I think that I'll set it up with a Select clause and use .ToArray to return it directly rather than with the For Each.

    *****

    I suspect though that to get the speed he really wants, he won't find it in any high level managed program.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, April 20, 2017 12:55 PM
  • db,

    How many did yours find?

    I suspect that's where the slow down might be.

    If I go back to it, I think that I'll set it up with a Select clause and use .ToArray to return it directly rather than with the For Each.

    *****

    I suspect though that to get the speed he really wants, he won't find it in any high level managed program.


    "A problem well stated is a problem half solved.” - Charles F. Kettering


    It found all of the files (5993) in 17 seconds.  The point I was trying to make is that this problem is IO limited because the folder is on the network.  I meant to run this at work where I have a folder that has 30K files in it.  Maybe tomorrow I will remember.

    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Thursday, April 20, 2017 10:55 PM


  • It found all of the files (5993) in 17 seconds.  The point I was trying to make is that this problem is IO limited because the folder is on the network.  I meant to run this at work where I have a folder that has 30K files in it.  Maybe tomorrow I will remember.

    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Great then - post the results when you can.

    It's interesting if nothing else.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Thursday, April 20, 2017 10:57 PM
  • Hi everyone,

    I ended up using Frank's code. It's simple and fast enough for my purposes (about 34 seconds). Thanks Frank.

    Here's what I'm using.

     
        Private Sub frmMain_Load(sender As Object, e As EventArgs) Handles MyBase.Load
            'Other code...
            '- - - - - 
            '- - - - - 
            CountTodaysPickFiles("AU")
            CountTodaysPickFiles("UK")
            CountTodaysPickFiles("US")
        End Sub
    
        Private Sub CountTodaysPickFiles(strWarehouse As String)
            Dim files As IEnumerable(Of String)
    
            If strWarehouse = "AU" Then
                files = GetFilePaths(FSW_OL_WH_Today.Path.ToString, SearchOption.AllDirectories, "*.zip")
                lblOutlivingPicksToday.Text = files.Count.ToString
            ElseIf strWarehouse = "UK" Then
                files = GetFilePaths(FSW_SUK_WH_Today.Path.ToString, SearchOption.AllDirectories, "*.csv")
                lblSunnylifeUKPicksToday.Text = files.Count.ToString
            Else 'If strWarehouse = "US" Then
                files = GetFilePaths(FSW_SUS_WH_Today.Path.ToString, SearchOption.AllDirectories, "*.txt")
                lblSunnylifeUSPicksToday.Text = files.Count.ToString
            End If
        End Sub
    
        Private Function GetFilePaths(ByVal directoryPath As String, ByVal searchType As SearchOption, ByRef Filter As String) As IEnumerable(Of String)

    @Cor: I did test your code. It took a touch over 6 minutes. Thanks very much for suggesting it, but as you can see, Frank's suggestion is more performant.


    Regards, Graham R Seach Sydney, Australia

    Friday, April 21, 2017 1:39 AM
  • Hi everyone,

    I ended up using Frank's code. It's simple and fast enough for my purposes (about 34 seconds). Thanks Frank.

    I'm glad that it helped. :)


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Friday, April 21, 2017 11:18 AM


  • It found all of the files (5993) in 17 seconds.  The point I was trying to make is that this problem is IO limited because the folder is on the network.  I meant to run this at work where I have a folder that has 30K files in it.  Maybe tomorrow I will remember.

    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Great then - post the results when you can.

    It's interesting if nothing else.


    "A problem well stated is a problem half solved.” - Charles F. Kettering

    I ran this at work against the folder that has 29, 539 files.  It took approximately 3 seconds to load all fileinfos.  I am attached to the server via gigabit ethernet.  My suspicion is that the performance is directly tied to how you are attached to the remote device. 

    I wish the OP had ran the test so we could compare results.


    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    Friday, April 21, 2017 12:41 PM

  • I wish the OP had ran the test so we could compare results.


    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    I'd be interested to know too.

    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Friday, April 21, 2017 1:03 PM

  • @Cor: I did test your code. It took a touch over 6 minutes. Thanks very much for suggesting it, but as you can see, Frank's suggestion is more performant.


    Regards, Graham R Seach Sydney, Australia

    Graham, 

    This is a test with 100000 files, (I think the creation takes some overhead on the OS, if I do it outside the program the time is less than 5 seconds while I use a very old computer)


    Success
    Cor

    Friday, April 21, 2017 3:59 PM
  • Cor,

    That's interesting but like we were talking about the other day, I don't think it means much.

    I say that because the only really valid comparison has to be with the identically same hardware setup.

    What's the OS? What's the type of the hard drive? Is it local or network? What file system? What's the seek time ... there's no end to variables here - see my point?

    *****

    I changed the query some so that it *would* find files:

    Private Function GetFilePaths(ByVal directoryPath As String, _ ByVal searchType As SearchOption) As IEnumerable(Of String) Dim retVal As IEnumerable(Of String) = Nothing Try If String.IsNullOrWhiteSpace(directoryPath) Then MessageBox.Show("The directory path cannot be null or empty.") Else Dim di As New DirectoryInfo(directoryPath) If Not di.Exists Then MessageBox.Show("The directory path could not be located.") Else Dim files As IEnumerable(Of FileInfo) = _ di.EnumerateFiles("*", searchType) If files IsNot Nothing Then retVal = _ (From fi As FileInfo In files _ Where fi.LastWriteTime.Date < Today.Date _ Select fi.FullName).ToArray End If End If End If Catch ex As Exception MessageBox.Show(String.Format("An error occurred:{0}{0}{1}", _ vbCrLf, ex.Message), _ "Exception Thrown", _ MessageBoxButtons.OK, _ MessageBoxIcon.Warning) End Try Return retVal End Function


    I ran the same test as the other day, twice:

    I then restarted my computer and this next time, I chose an older (non SSD) drive with more files in it:

    So what does that mean then?

    In my opinion, not a whole lot - there are too many variables to really draw any sort of conclusion.

    ...for what it's worth


    "A problem well stated is a problem half solved.” - Charles F. Kettering


    Friday, April 21, 2017 4:52 PM

  • I wish the OP had ran the test so we could compare results.


    "Those who use Application.DoEvents() have no idea what it does and those who know what it does never use it." - MSDN User JohnWein    Multics - An OS ahead of its time.

    I'd be interested to know too.

    "A problem well stated is a problem half solved.” - Charles F. Kettering

    Frank, 

    I wished it once happened. Dbasnett and I seems to agree. 

    I get the idea the OP is adding constantly a lot which should not be done. (to say it simple) Or something which he did not tell to us. 


    Success
    Cor

    Friday, April 21, 2017 7:21 PM