locked
Flat File Debatching with Recoverable Interchange RRS feed

  • Question

  • I have a simple CSV flat file , that can have multiple records (rows)

    Field1,Field2 ... {CR\LF}

    Field1,Field2 ... {CR\LF}

    We want that , if any of the record contains error, that particular record should be filtered out. For that I have set Recoverable Interchange to True.

    What happens in , if the flat file as 10 rows,  and 5th row contains error , then all rows from 5th to 10th are not processed (suspended). Only  1st to 4th row are successfully debatched and sent to the output.

    Is this the Flat file disassembler behavior ? What is the way to fix this ?

    Regards. Thanks in advance for any help.

    Thursday, November 26, 2015 2:39 PM

Answers

  • Yes, this is the expected behavior though it may not be the desired behavior.

    Whether or not the Flat File Disassembler can recover depends on the type of problem and to a lesser extent, how the Schema is composed (you don't have a log of flexibility there).

    What exactly is the error?

    Thursday, November 26, 2015 3:09 PM
    Moderator
  • Hi,

    Welcome on MSDN forum.

    Just re-iterating the steps considered for debatching a flat file with Recoverable interchange processing:

    1. Created an Envelope Schema and a document schema (the input message) for debatching the incoming message.
    2. Created a custom pipeline and placed the Flat File Disassembler component on Disassemble stage.
    3. Set the Envelope Schema & Document Schema properly as per step 1.
    4. Set the Recoverable interchange processing property to TRUE.
    5. Deploy & test the solution.

    I look forward to receiving your test results.


    Thanks,

    If my reply is helpful please mark as Answer or vote as Helpful.

    My blog | Twitter | LinkedIn

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    Thursday, November 26, 2015 7:18 PM
    Moderator

All replies

  • Yes, this is the expected behavior though it may not be the desired behavior.

    Whether or not the Flat File Disassembler can recover depends on the type of problem and to a lesser extent, how the Schema is composed (you don't have a log of flexibility there).

    What exactly is the error?

    Thursday, November 26, 2015 3:09 PM
    Moderator
  • Hi,

    Welcome on MSDN forum.

    Just re-iterating the steps considered for debatching a flat file with Recoverable interchange processing:

    1. Created an Envelope Schema and a document schema (the input message) for debatching the incoming message.
    2. Created a custom pipeline and placed the Flat File Disassembler component on Disassemble stage.
    3. Set the Envelope Schema & Document Schema properly as per step 1.
    4. Set the Recoverable interchange processing property to TRUE.
    5. Deploy & test the solution.

    I look forward to receiving your test results.


    Thanks,

    If my reply is helpful please mark as Answer or vote as Helpful.

    My blog | Twitter | LinkedIn

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    Thursday, November 26, 2015 7:18 PM
    Moderator
  • Hi,

    Can you please share the exception message which is being logged in BizTalk Console or the event log ?

    Without proper information it will be difficult to give you any better solution approach.

    Thanks

    Abhishek


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply

    Friday, November 27, 2015 6:43 AM
  • The error is the right one pointing to the fault :

    The document failed to validate because of the following error:"The 'Remarks' element has an invalid value according to its data type." . .
     The sequence number of the suspended message is 6. 

    The Schema is this :

    <?xml version="1.0" encoding="utf-16"?>
    <xs:schema xmlns="http://Name.Schemas.ItemStatusesFFCanonical" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://Name.ItemStatusesFFCanonical" xmlns:xs="http://www.w3.org/2001/XMLSchema">
      <xs:annotation>
        <xs:appinfo>
          <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
          <b:schemaInfo standard="Flat File" codepage="65001" default_pad_char=" " pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="true" compile_parse_tables="false" root_reference="ItemStatuses" />
        </xs:appinfo>
      </xs:annotation>
      <xs:element name="ItemStatuses">
        <xs:annotation>
          <xs:appinfo>
            <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" child_order="postfix" />
          </xs:appinfo>
        </xs:annotation>
        <xs:complexType>
          <xs:sequence>
            <xs:annotation>
              <xs:appinfo>
                <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
              </xs:appinfo>
            </xs:annotation>
            <xs:element maxOccurs="1" name="ItemLine">
              <xs:annotation>
                <xs:appinfo>
                  <b:recordInfo structure="delimited" child_delimiter_type="char" child_delimiter="," child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
                </xs:appinfo>
              </xs:annotation>
              <xs:complexType>
                <xs:sequence>
                  <xs:annotation>
                    <xs:appinfo>
                      <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
                    </xs:appinfo>
                  </xs:annotation>
                  <xs:element name="Application" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="1" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="TrackingNo" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="2" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="StageCode" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="3" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="StageUpdatedDate" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="4" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="UpdatedLocation" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="5" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="Remarks">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="6" />
                      </xs:appinfo>
                    </xs:annotation>
                    <xs:simpleType>
                      <xs:restriction base="xs:string">
                        <xs:maxLength value="20" />
                      </xs:restriction>
                    </xs:simpleType>
                  </xs:element>
                  <xs:element name="Discontinued" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="7" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="LastUpdatedBy" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="8" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                  <xs:element name="LastUpdatedDate" type="xs:string">
                    <xs:annotation>
                      <xs:appinfo>
                        <b:fieldInfo justification="left" sequence_number="9" />
                      </xs:appinfo>
                    </xs:annotation>
                  </xs:element>
                </xs:sequence>
              </xs:complexType>
            </xs:element>
          </xs:sequence>
        </xs:complexType>
      </xs:element>
    </xs:schema>


    • Edited by NdBot Friday, November 27, 2015 7:19 AM Edited namespaces
    Friday, November 27, 2015 7:17 AM
  • The document failed to validate because of the following error:"The 'Remarks' element has an invalid value according to its data type." . .
     The sequence number of the suspended message is 6.

    This means that the instance of XML you are trying to load has a value in a field that is different to what the XSD expects. In the above example the XSD schema had the following definition

    <xs:element name="Remarks">
    <xs:annotation>
      <xs:appinfo>
    	<b:fieldInfo justification="left" sequence_number="6" />
      </xs:appinfo>
    </xs:annotation>
    <xs:simpleType>
      <xs:restriction base="xs:string">
    	<xs:maxLength value="20" />
      </xs:restriction>
    </xs:simpleType>
    </xs:element>
    
    What value you are passing in Remarks ?

    Thanks,

    If my reply is helpful please mark as Answer or vote as Helpful.

    My blog | Twitter | LinkedIn

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    Friday, November 27, 2015 7:30 AM
    Moderator
  • @ Kamlesh,

    I know what the error is & by the way it is a Flat file , I am purposely passing wrong (exceeding length) value in Remarks node.

    The whole point of this question is , one error record in Flat file , is preventing any subsequent records in the file to be processed. Recoverable Interchange is not working as it should normally.  I would expect (or our requirement) is the filter out error record , but process the correct ones.

    As such the schema is fairly simple , so not sure why the Disassembler cannot continue with further correct records if it finds a error record.

    Friday, November 27, 2015 7:47 AM
  • Thank you more clarification, I would suggest to insert the records in SQL database and put the validation. If records are correct then only process it with BizTalk and for rest records throw the error notification. As this is simple and doable approach.

    Thanks,

    If my reply is helpful please mark as Answer or vote as Helpful.

    My blog | Twitter | LinkedIn

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    Friday, November 27, 2015 9:47 AM
    Moderator