none
Flat File Source - overflowed the disk I/O buffer

    Question

  • I've a problem with reading the CSV source file, see below error.  This only happens to some files, and I noticed that the files I've problem with are over 200K rows.  All files have the same format, 8 columns total.  For the files that work, I can review and see column 0 to column 7 under flat file source editor.  For the files that fail, I can only see column 0; and when I remove the rows in the file I can see all columns.  I wonder if this has to do with the limited number of rows ssis can read in csv.  I'd appreciate any comments.  Thanks!

    Error: 0xC020209C at Download Keyword Search Data, Flat File Source [1]: The column data for column "Column 7" overflowed the disk I/O buffer.

    Error: 0xC0202092 at Download Keyword Search Data, Flat File Source [1]: An error occurred while processing file "\\server_abc\temp\test.csv" on data row 1.


    Ash_
    Monday, April 12, 2010 11:08 PM

Answers

  • It sounds to me like your large files have data corruption in them - that would explain why columns look "shifted" when you use LF only as a delimiter.  The reason that is likely is how the flat file source reads and interprets column and line delimiters.

    You may want to try the Delimited File Source instead - I hear it's better at dealing with problematic sources.

    I would wager that you'll find a specific line in your large source file that has one less column than you expect, and/or has some binary characters in it.


    Todd McDermid's Blog
    • Marked as answer by Ash_ Thursday, April 15, 2010 12:02 AM
    Wednesday, April 14, 2010 4:26 PM

All replies

  • SSIS doesn't have a limit on the number of rows it can read in for a CSV file.  It's much more likely that your files are malformed somehow - and perhaps only in your larger files.

    If you configure the Flat File Source to redirect rows in error, and pull the error flow to go into a Row Count, do the files load?  If so, do any rows go to the Row Count?


    Todd McDermid's Blog
    Tuesday, April 13, 2010 1:12 AM
  • The files do not load even when I configure the Flat File Source to ignore failure on error.  So I don't think redirect rows would work.  I've delimited format, CR/LF for header row delimiter, CR/LF for row delimiter, and comma for column delimiter; this works for small files.  When I change row delimiter to LF, it works for the large files too but not loading the data correctly; some of the fields are combined into one column and shifted.

    I'm still troubleshooting the problem.  Thanks for the reply.


    Ash_
    Wednesday, April 14, 2010 4:20 PM
  • It sounds to me like your large files have data corruption in them - that would explain why columns look "shifted" when you use LF only as a delimiter.  The reason that is likely is how the flat file source reads and interprets column and line delimiters.

    You may want to try the Delimited File Source instead - I hear it's better at dealing with problematic sources.

    I would wager that you'll find a specific line in your large source file that has one less column than you expect, and/or has some binary characters in it.


    Todd McDermid's Blog
    Wednesday, April 14, 2010 4:26 PM
  • It sounds to me like your large files have data corruption in them - that would explain why columns look "shifted" when you use LF only as a delimiter.  The reason that is likely is how the flat file source reads and interprets column and line delimiters.

    You may want to try the Delimited File Source instead - I hear it's better at dealing with problematic sources.

    I would wager that you'll find a specific line in your large source file that has one less column than you expect, and/or has some binary characters in it.


    Todd McDermid's Blog
    • Marked as answer by Ash_ Thursday, April 15, 2010 12:02 AM
    Wednesday, April 14, 2010 4:26 PM
  • All of our files have 8 columns total, but not all rows have the same number of columns.  For example, the first 5 rows have only 2 columns, after the 5th rows all have 8 columns, the 8th column always has data, some rows have data in the 7th column and some not.  All files have exact same format.

    You're right with the less column than expected.  It still does not make sense why it works for small file but not larger file.  I'm unable to find any binary characters in the files due to large volume of data.  I'll try the Delimited File Source you suggested, as I've 30+ files to process on a regular basis.  Thanks for all your help.


    Ash_
    Thursday, April 15, 2010 12:02 AM