none
Process the data and create .txt files. RRS feed

  • Question

  • Hi Team,

    My requirement is to process the data and create 15 .txt files. I have a master table where all the .txt file names are configured and 15 tables where each table contains each .txt file data.

    I have implemented a windows service which will run every 30 mins. I have written logic to process and then create each .txt files and place the files into the folder path (see below).  In the method "ProcessFileData", i am calling 15 procedures as each procedure will process the data from the table pertaining to each .txt file. 

     please let me know if there is any better code then what i thought of.?  Need inputs and help.     Thank you. 

    Data.cs file

    using System;
    using System.Collections.Generic;
    using System.Configuration;
    using System.Data;
    using System.IO;
    using System.Net;
    using System.Linq;
    using System.Runtime.Serialization.Json;
    using System.Text;
    using System.Xml;
    using DataFetch.DAL;
    
    namespace DataFetch
    {
        public class Data
        {            
            public void ProcessData()
            {
               string errorMessage = string.Empty;
               try
                {
                   DataAccess baseDataAccess = new DataAccess();
                   IList <ConfigurationModel> configurationModel = null;
                   configurationModel = baseDataAccess.GetFileConfiguration();                                                          
                   if (configurationModel != null && configurationModel.Count > 0)
                   {                   
                        foreach (var fileConfiguration in configurationModel)
                        {
                            if (fileConfiguration.FileName.Trim() != "")
                            {                           
                                ProcessFileData(fileConfiguration.FileName, fileConfiguration.FilePath, fileConfiguration.ArchiveFilePath);                           
                            }
                            else
                            {
                                errorMessage = string.Format("File Configuration data is null or empty.");
                                LogManager.ApplicationLogger.Error(errorMessage);
                                return;
                            }
                        }
                    }
                    else
                    {                  
                        errorMessage = string.Format("File Configuration data is null.");
                        LogManager.ApplicationLogger.Error(errorMessage);
                    }
                }
                catch (Exception ex)
                {
                    LogManager.ApplicationLogger.Error(ex);
                    errorMessage = "Exception occured while processing base Data : " + ex.Message;
                    throw ex;
                }
                finally
                {
                    LogManager.ApplicationLogger.Debug(Constants.LOG_SEPARATOR + Constants.ENDED);
                }
            }     
          
            private void ProcessFileData(string fileName, string fileFTPPath, string archiveFilePath)
            {
                string errorMessage = string.Empty;
                DataAccess baseDataAccess = new DataAccess();
                try
                {
                    StringBuilder baseDataExport = new StringBuilder();
                    var filePath = fileFTPPath;
                    if (filePath == "" || !Directory.Exists(Path.GetFullPath(filePath)))
                    {
                        errorMessage = string.Format("File path is invalid.");
                        LogManager.ApplicationLogger.Error(errorMessage);
                        return;
                    }
                    var fileNameWithExtension = fileName;
                    var fileNameWithoutExtension = Path.GetFileNameWithoutExtension(fileNameWithExtension);
                    var extension = Path.GetExtension(fileNameWithExtension);
                    if (extension == "" || extension == null || extension != Constants.fileExtension)
                    {
                        extension = Constants.fileExtension;
                        fileNameWithExtension = fileName + extension;
                    }
                    var archivePath = archiveFilePath;
                    if (archivePath == "" || !Directory.Exists(Path.GetFullPath(archivePath)))
                    {
                        errorMessage = string.Format("Archive File path is invalid.");
                        LogManager.ApplicationLogger.Error(errorMessage);
                        return;
                    }
                    // Convert the DateTime to EST TimeZone
                    var timeToConvert = DateTime.Now;
                    var currentDateTime = CurrentDateTime(timeToConvert);
                    var formatFileName = fileName + "_" + currentDateTime.ToString(Constants.dateTimeFormat) + extension.ToLower();
                    var fullFilePath = Path.Combine(filePath + @"\" + fileNameWithExtension);
                    var fullArchiveFilePath = Path.Combine(archiveFilePath + @"\" + formatFileName);
                    //delete the previous generated file
                    if (System.IO.File.Exists(fullFilePath))
                    {
                        System.IO.File.Delete(fullFilePath);
                        System.Threading.Thread.Sleep(20);
                    }
    
                    string procName = string.Empty;
                    string tableName = "FileOutput";
    
                    if (fileName == "File1.txt")
                    {
                        procName = "[dbo].[GetFile1Data]";                    
                    }
                    else if (fileName == "File2.txt")
                    {
                        procName = "[dbo].[GetFile2Data]";                   
                    }
                    else if(fileName == "File3.txt")
                    {
                        procName = "[dbo].[GetFile3Data]";                   
                    }
                    else if (fileName == "File4.txt")
                    {
                        procName = "[dbo].[GetFile4Data]";
                    }
                    else if (fileName == "File5.txt")
                    {
                        procName = "[dbo].[GetFile5Data]";
                    }
     	        else if (fileName == "File6.txt")
                    {
                        procName = "[dbo].[GetFile6Data]";
                    }
    		 else if (fileName == "File7.txt")
                    {
                        procName = "[dbo].[GetFile7Data]";
                    }
    		else if (fileName == "File8.txt")
                    {
                        procName = "[dbo].[GetFile8Data]";
                    }
    		else if (fileName == "File9.txt")
                    {
                        procName = "[dbo].[GetFile9Data]";
                    }
    		 else if (fileName == "File10.txt")
                    {
                        procName = "[dbo].[GetFile10Data]";
                    }
    
                    else if (fileName == "File11.txt")
                    {
                        procName = "[dbo].[GetFile11Data]";
                    }
    		 else if (fileName == "File12.txt")
                    {
                        procName = "[dbo].[GetFile12Data]";
                    }
    		else if (fileName == "File13.txt")
                    {
                        procName = "[dbo].[GetFile13Data]";
                    }
    		else if (fileName == "File14.txt")
                    {
                        procName = "[dbo].[GetFile14Data]";
                    }
    		 else if (fileName == "File15.txt")
                    {
                        procName = "[dbo].[GetFile15Data]";
                    }
    
    
                    IList<DataModel> formattedData = baseDataAccess.GetData(procName, tableName);
    
    
                    if (formattedData == null || formattedData.Count == 0)
                    {
                        var emptyFilePath = fullFilePath;
                        File.AppendAllText(emptyFilePath, "");
                        var emptyArchiveFilePath = fullArchiveFilePath;
                        File.AppendAllText(emptyArchiveFilePath, "");
                        return;
                    }
                    foreach (var baseDataRow in formattedbaseData)
                    {
                        baseDataExport.AppendLine(baseDataRow.finalData);
                    }
                    var upstreamFilePath = fullFilePath;
                    var upstreamFileData = baseDataExport.ToString();
                    File.AppendAllText(upstreamFilePath, upstreamFileData);
                    var downstreamFilePath = fullArchiveFilePath;
                    var downstreamFileData = baseDataExport.ToString();
                    File.AppendAllText(downstreamFilePath, downstreamFileData);
                 
                }
                catch (Exception ex)
                {
                    LogManager.ApplicationLogger.Error(ex);
                    errorMessage = "Exception occured while processing File Data : " + ex.Message;
                    throw ex;
                }
                finally
                {
                    LogManager.ApplicationLogger.Debug(Constants.LOG_SEPARATOR + Constants.ENDED);
                }
            }
    
    
            
            public DateTime CurrentDateTime(DateTime adjustedDateTime)
            {
                var est = TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time");
                var targetTime = TimeZoneInfo.ConvertTime(adjustedDateTime, est).ToString("HH:mm");
                return adjustedDateTime = DateTime.ParseExact(targetTime,"HH:mm", null, System.Globalization.DateTimeStyles.None);
            }
              
    	
        }
    }
    

    DataAcess.cs   file 

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    using System.Data;
    using System.Data.SqlClient;
    
    namespace DataFetch.DAL
    {
        public class DataAccess
        {
            string _connString = null;
            public DataAccess()
            {
                _connString = System.Configuration.ConfigurationManager.AppSettings.Get("DataFetch_DBConnection");
            }                     
             
            public DateTime GetDBDateValue(object dbValueOfTypeDateTime)
            {
                return dbValueOfTypeDateTime != DBNull.Value ? Convert.ToDateTime(dbValueOfTypeDateTime) : DateTime.MinValue;
            }        
    
            private IList<DataModel> MapDataToProcess(DataTable baseDataTable)
            {
                try { 
                if (baseDataTable == null || baseDataTable.Rows.Count == 0)
                { return null; }
                IList<DataModel> baseDataList = new List<DataModel>();
                DataModel baseDataTableValue = null;
                foreach (DataRow dr in baseDataTable.Rows)
                {
                        baseDataTableValue = new DataModel();
                        baseDataTableValue.finalData = dr["FormattedOutput"].ToString();
                        baseDataList.Add(baseDataTableValue);
                }
                  return baseDataList;
                }
                catch (SqlException ex)
                {
                    LogManager.ApplicationLogger.Error(Constants.LOG_SEPARATOR + Constants.EXCEPTIONRAISED + ex.Message + Constants.LOG_SEPARATOR + ex.Source + Constants.LOG_SEPARATOR + ex.InnerException + Constants.LOG_SEPARATOR + ex.StackTrace, ex);
                    throw ex;
                }
                finally
                {
                    LogManager.ApplicationLogger.Debug(Constants.LOG_SEPARATOR + Constants.ENDED);
                }
            }
    
            public IList<DataModel> GetData(string procedureName, string formattedOutput)
            {
                IList<DataModel> formattedData = null;
                try
                {
                    LogManager.ApplicationLogger.Debug(Constants.LOG_SEPARATOR + Constants.STARTED);
                    if (string.IsNullOrEmpty(_connString))
                    { return null; }
                    using (SqlConnection objSQLConnection = new SqlConnection(_connString))
                    {
                        using (SqlCommand objCommand = new SqlCommand())
                        {
                            objCommand.Connection = objSQLConnection;
                            objCommand.CommandType = CommandType.StoredProcedure;                        
                            objCommand.CommandText = procedureName;
                            SqlDataAdapter objAdapter = new SqlDataAdapter();
                            objSQLConnection.Open();
                            objAdapter = new SqlDataAdapter();
                            DataSet objDataset = new DataSet();
                            objAdapter.SelectCommand = objCommand;
                            objAdapter.Fill(objDataset, formattedOutput);
                            if (objDataset != null && objDataset.Tables.Count > 0)
                            {
                                formattedData = MapDataToProcess(objDataset.Tables[formattedOutput]);
                            }
                        }
                    }
                }
                catch (SqlException ex)
                {
                    LogManager.ApplicationLogger.Error(Constants.LOG_SEPARATOR + Constants.EXCEPTIONRAISED + ex.Message + Constants.LOG_SEPARATOR + ex.Source + Constants.LOG_SEPARATOR + ex.InnerException + Constants.LOG_SEPARATOR + ex.StackTrace, ex);
                    throw ex;
                }
                return formattedData;
            }
           
                  
            public IList<ConfigurationModel> GetFileConfiguration()
            {
                try
                {
                    IList<ConfigurationModel> configurationModelValue = null;
                    DBHelper dbClass = new DBHelper();
                    string sql = "SELECT [FileName],[FileFTPPath],[FileArchivePath] FROM [dbo].[DataConfiguration]";
                    DataSet ds = dbClass.GetDataSet(sql, "DataConfiguration");
                    if (ds != null && ds.Tables.Count > 0)
                    {
                        configurationModelValue = MapConfiguration(ds.Tables["DataConfiguration"]);
                        return configurationModelValue;
                    }
                    return null;
                }
                catch (SqlException ex)
                {
                    LogManager.ApplicationLogger.Error(Constants.LOG_SEPARATOR + Constants.EXCEPTIONRAISED + ex.Message + Constants.LOG_SEPARATOR + ex.Source + Constants.LOG_SEPARATOR + ex.InnerException + Constants.LOG_SEPARATOR + ex.StackTrace, ex);
                    throw ex;
                }
                finally
                {
                    LogManager.ApplicationLogger.Debug(Constants.LOG_SEPARATOR + Constants.ENDED);
                }
            }
    
            private IList<ConfigurationModel> MapConfiguration(DataTable fileList)
            {
                if (fileList == null || fileList.Rows.Count == 0)
                { return null; }
    
                IList<ConfigurationModel> configList = new List<ConfigurationModel>();
                ConfigurationModel configurationModelValue = null;
                foreach (DataRow dr in fileList.Rows)
                {
                    configurationModelValue = new ConfigurationModel();
                    configurationModelValue.FileName = Convert.ToString(dr["FileName"]);
                    configurationModelValue.FilePath = Convert.ToString(dr["FileFTPPath"]);
                    configurationModelValue.ArchiveFilePath = Convert.ToString(dr["FileArchivePath"]);
                    configList.Add(configurationModelValue);
                }
                return configList;
            }
                            
        }
    }
    


    Vamshi Janagama

    Thursday, February 20, 2020 9:56 PM

Answers

  • Hi Vamshi K J,

    Thank you for posting here.

    I did not run your project because some custom classes are still missing.But I looked at the code you provided.

    In Data.cs, you use a very long if statement, which seems unnecessary.

    Try replacing them with the following code.

    procName = "[dbo].[Get" + fileName.Split(new char[] { '.' })[0] + "Data]";

    This line can replace many lines. And if you need the16th txt file in the future, you don't need to modify the code here.

    Hope this could be helpful.

    Best Regards,

    Timon


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Vamshi K J Monday, February 24, 2020 2:29 PM
    Friday, February 21, 2020 5:36 AM
  • Hi,

    According to my understanding of your code, the ProcessFileData method only involves one file each time it is called, right?

    If so, the question now is how to call this method asynchronously.

    After .net framework 4.5, we can implement this feature using Task.Run ().

    Here is a code example.

          static void Main(string[] args)
            {
                for (int i = 0; i < 10; i++)
                {
                    new Task(() => { DoWork(i); }).Start();
                }
                Console.ReadKey();
            }
    
    
    
            static void DoWork(int n)
            {
                for (int i = 0; i < 10; i++)
                {
                    Console.WriteLine("Test" + n);
                }
            }

    Hope this could be helpful.

    Best Regards,

    Timon


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Vamshi K J Tuesday, February 25, 2020 3:24 PM
    Monday, February 24, 2020 9:13 AM
  • Thank you Timon Yang for the sample code inputs.  So i changed my code using Task.Run () as below, 

    if (configurationModel != null && configurationModel.Count > 0)
                   {                   
                        foreach (var fileConfiguration in configurationModel)
                        {
                            if (fileConfiguration.FileName.Trim() != "")
                            {                           
                              new Task(() => { ProcessFileData(fileConfiguration.FileName, fileConfiguration.FilePath, fileConfiguration.ArchiveFilePath); }).Start();       
                            }
                            else
                            {
                                errorMessage = string.Format("File Configuration data is null or empty.");
                                LogManager.ApplicationLogger.Error(errorMessage);
                                return;
                            }
                        }
                    }

    Based on your sample code, what i understand is each file is processed as a separate individual task which helps to process all the files data without any interruptions  before the windows service Timer refreshes to process the new data and create new files again.   

    please correct me if i miss anything.


    Vamshi 




    • Edited by Vamshi K J Monday, February 24, 2020 5:24 PM
    • Marked as answer by Vamshi K J Tuesday, February 25, 2020 3:24 PM
    Monday, February 24, 2020 2:39 PM

All replies

  • Hi Vamshi K J,

    Thank you for posting here.

    I did not run your project because some custom classes are still missing.But I looked at the code you provided.

    In Data.cs, you use a very long if statement, which seems unnecessary.

    Try replacing them with the following code.

    procName = "[dbo].[Get" + fileName.Split(new char[] { '.' })[0] + "Data]";

    This line can replace many lines. And if you need the16th txt file in the future, you don't need to modify the code here.

    Hope this could be helpful.

    Best Regards,

    Timon


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Vamshi K J Monday, February 24, 2020 2:29 PM
    Friday, February 21, 2020 5:36 AM
  • Thank you Timon Yang


    Currently the requirement is to process 15 files BUT there is a possibility of more files being added for data processing. i.e. 25 files etc
    so since there are 25 files, the time to process data and create each file is based on the data volume. 

    Need inputs and help  to convert method to async task method so that each and every file is created with data before the windows service timer refreshes to process the new data and create files again. 


    Vamshi Janagama

    Monday, February 24, 2020 5:37 AM
  • Hi,

    According to my understanding of your code, the ProcessFileData method only involves one file each time it is called, right?

    If so, the question now is how to call this method asynchronously.

    After .net framework 4.5, we can implement this feature using Task.Run ().

    Here is a code example.

          static void Main(string[] args)
            {
                for (int i = 0; i < 10; i++)
                {
                    new Task(() => { DoWork(i); }).Start();
                }
                Console.ReadKey();
            }
    
    
    
            static void DoWork(int n)
            {
                for (int i = 0; i < 10; i++)
                {
                    Console.WriteLine("Test" + n);
                }
            }

    Hope this could be helpful.

    Best Regards,

    Timon


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Vamshi K J Tuesday, February 25, 2020 3:24 PM
    Monday, February 24, 2020 9:13 AM
  • Thank you Timon Yang for the sample code inputs.  So i changed my code using Task.Run () as below, 

    if (configurationModel != null && configurationModel.Count > 0)
                   {                   
                        foreach (var fileConfiguration in configurationModel)
                        {
                            if (fileConfiguration.FileName.Trim() != "")
                            {                           
                              new Task(() => { ProcessFileData(fileConfiguration.FileName, fileConfiguration.FilePath, fileConfiguration.ArchiveFilePath); }).Start();       
                            }
                            else
                            {
                                errorMessage = string.Format("File Configuration data is null or empty.");
                                LogManager.ApplicationLogger.Error(errorMessage);
                                return;
                            }
                        }
                    }

    Based on your sample code, what i understand is each file is processed as a separate individual task which helps to process all the files data without any interruptions  before the windows service Timer refreshes to process the new data and create new files again.   

    please correct me if i miss anything.


    Vamshi 




    • Edited by Vamshi K J Monday, February 24, 2020 5:24 PM
    • Marked as answer by Vamshi K J Tuesday, February 25, 2020 3:24 PM
    Monday, February 24, 2020 2:39 PM