none
Excel ADODB Sql Query Execution taking hours when manipulate excel tables RRS feed

  • General discussion

  • Hello All 

    I have 28000 records with 8 column in an sheet. When I convert the sheet into ADODB database and copy to new excel using below code it is executing in less than a min

    Set Tables_conn_obj = New ADODB.Connection Tables_conn_str = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & Table_Filename & ";Extended Properties=""Excel 12.0;ReadOnly=False;HDR = Yes;IMEX=1""" Tables_conn_obj.Open Tables_conn_str First_Temp_sqlqry = "Select * INTO [Excel 12.0;DATABASE=C:\Prod Validation\Database\Second Acat Table.xlsb].[Sheet1] Table [first - Table$];" Tables_conn_obj.Execute First_Temp_sqlqry




    But when I change the query to manipulate one column in current table based on another table in the same excel and try to copy the results in another excel, it is taking more than one hour.. why it is taking this much time when both the query results returns the same number of rows and column. I almost spend one week and still not able to resolve this issue.

    Even I tried copyfromrecordset, getrows(), getstring(), Looping each recordset fields options all of them taking same amount of time. Why there is huge difference in execution time.

    Important note: Without into statement even below query is executing in few seconds.

    
    
    select ( ''''''manipulating first column based on other table data''''''''''''''
    
    iif( 
    
    [Second - Table$].[Policy Agent] = (select max([ACAT$].[new_Agent_number]) from [ACAT$] where [ACAT$].[new_Agent_number] = [Second - Table$].[Policy Agent] and [ACAT$].[ACAT_EffectiveDate] = ( select MAX([ACAT$].[ACAT_EffectiveDate] ) from [ACAT$] where [ACAT$].[new_Agent_number] = [Second - Table$].[Policy Agent]and [ACAT$].[ACAT_EffectiveDate] > '2014-10-01') ) , (select max([ACAT$].[Old_Agent_number]) from [ACAT$] where [ACAT$].[new_Agent_number] = [Second - Table$].[Policy Agent] and [ACAT$].[ACAT_EffectiveDate] = ( select MAX([ACAT$].[ACAT_EffectiveDate] ) from [ACAT$] where [ACAT$].[new_Agent_number] = [Second - Table$].[Policy Agent]and [ACAT$].[ACAT_EffectiveDate] > '2014-10-01')) ,
    
    iif( [Second - Table$].[Policy Agent] = (select max([ACAT$].[Old_Agent_number]) from [ACAT$] where [ACAT$].[Old_Agent_number] = [Second - Table$].[Policy Agent] and [ACAT$].[ACAT_EffectiveDate] = ( select MAX([ACAT$].[ACAT_EffectiveDate] ) from [ACAT$]where [ACA T$].[Old_Agent_number] = [Second - Table$].[Policy Agent]and [ACAT$].[ACAT_EffectiveDate] <= '2014-10-01') ), (select max([ACAT$].[new_Agent_number]) from [ACAT$] where [ACAT$].[Old_Agent_number] = [Second - Table$].[Policy Agent] and [ACAT$].[ACAT_EffectiveDate] = ( select MAX([ACAT$].[ACAT_EffectiveDate] ) from [ACAT$] where [ACAT$].[Old_Agent_number] = [Second - Table$].[Policy Agent]and [ACAT$].[ACAT_EffectiveDate] <= '2014-10-01')) ,
    
    [Second - Table$].[Policy Agent] ))) as [Policy Agent],
    
    
    ''''''summing up all other columns''''''''''''''
    
    (iif(isnull(sum([Second - Table$].[Auto BW-Line Of Business Detail])),0,sum([Second - Table$].[Auto BW-Line Of Business Detail]))) as [Auto BW-Line Of Business Detail],(iif(isnull(sum([Second - Table$].[Auto Farmers])),0,sum([Second - Table$].[Auto Farmers]))) as [Auto Farmers],(iif(isnull(sum([Second - Table$].[MCA])),0,sum([Second - Table$].[MCA]))) as [MCA],(iif(isnull(sum([Second - Table$].[CEA])),0,sum([Second - Table$].[CEA]))) as [CEA],(iif(isnull(sum([Second - Table$].[Commercial P&C])),0,sum([Second - Table$].[Commercial P&C]))) as [Commercial P&C],(iif(isnull(sum([Second - Table$].[Comm WC])),0,sum([Second - Table$].[Comm WC]))) as [Comm WC],(iif(isnull(sum([Second - Table$].[Fire Farmers])),0,sum([Second - Table$].[Fire Farmers]))) as [Fire Farmers],(iif(isnull(sum([Second - Table$].[Flood])),0,sum([Second - Table$].[Flood]))) as [Flood],(iif(isnull(sum([Second - Table$].[Kraft Lake])),0,sum([Second - Table$].[Kraft Lake]))) as [Kraft Lake],(iif(isnull(sum([Second - Table$].[Life])),0,sum([Second - Table$].[Life]))) as [Life],(iif(isnull(sum([Second - Table$].[Foremost])),0,sum([Second - Table$].[Foremost]))) as [Foremost],(iif(isnull(sum([Second - Table$].[Umbrella])),0,sum([Second - Table$].[Umbrella]))) as [Umbrella],(iif(isnull(sum([Second - Table$].[MCNA])),0,sum([Second - Table$].[MCNA]))) as [MCNA]
    
    INTO [Excel 12.0;DATABASE=C:\Prod Validation\Database\Second Acat Table.xlsb].[Sheet1]
    
    from [Second - Table$] group by [Second - Table$].[Policy Agent] ;









    Friday, January 9, 2015 9:56 PM

All replies

  • Hi brajesheee,

    Thanks for posting in MSDN forum.

    Based on the description, the time cost on importing two tables with same num rows and columns is much different.

    According to the second SQL query, it is a little complex. Could ths issue be reproduced use a simple SQL query like the first one?

    If not, the time may cost by the calulation of complex SQL query. Otherwise, it may relative to the data in second table.

    Hope it is helpful.

    Regards & Fei


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, January 12, 2015 5:35 AM
    Moderator
  • Hi Fei,

      Thank you so much for the reply post. I just executed the same above SQL without INTO Statement and assigned the SQL result to ADODB recordset as below. If the time difference is due to the SQL query then below statements also should execute for hours right, but it gets executed in seconds. But to copy the recordset to excel again it is taking hours. I tried copyfromrecordset, getrows(), getstring(), Looping each recordset fields options and all of them taking same amount of time. Please let me know there is delay in time for this much small data

    Even I tried to typecast all columns to double, string in SQL and still the execution time  is not reduced. 

    First_Temp_Recordset.Open sql_qry, Tables_conn_obj, adOpenStatic, adLockOptimistic ''' OR

    SET First_Temp_Recordset = Tables_conn_obj.Execute sql_qry



    Monday, January 12, 2015 3:30 PM