locked
Error Debatching Flat File RRS feed

  • Question

  • HI,
    I am trying to debatch a flat file data into multiple XML mesasges. I have created a schema and I am able to succefully debatch it into one XML message with repeating records but as I try to change the Maxoccur property of element Orders, I get a error message 'The Messaging Engine encountered an error during the processing of one or more inbound messages. '

    Data  is
    ORDER100|2009-04-01
    IT-1000Fuse Bulbs  5
    IT-1000Fuse Bulbs  2
    ORDER101|2009-06-01
    IT-1000Fuse Bulbs  5
    IT-1000Fuse Bulbs  2

    XML Mesasge Which I get after debatching is

      <?xml version="1.0" encoding="utf-8" ?>
    - <OrdersBatch xmlns="http://DebatchFlatFile.OrdersBatch">
    - <Orders xmlns="">
    - <OrderMain>
      <OrderID>100</OrderID>
      <Date>2009-04-01</Date>
      </OrderMain>
    - <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
    - <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders>
    - <Orders xmlns="">
    - <OrderMain>
      <OrderID>101</OrderID>
      <Date>2009-06-01</Date>
      </OrderMain>
    - <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
    - <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders>
      </OrdersBatch>

    I can post the schema if requested..

    Shady
    Tuesday, June 9, 2009 9:33 AM

Answers


  • I was able to debatch this xml in pipeline, only difference i see is prefix
    <ns0:OrdersBatch xmlns:ns0="http://DebatchFlatFile.OrdersBatch">
     <Orders >
     <OrderMain>
      <OrderID>100</OrderID>
      <Date>2009-04-01</Date>
      </OrderMain>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders> 
    <Orders>
     <OrderMain>
      <OrderID>100</OrderID>
      <Date>2009-04-01</Date>
      </OrderMain>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders> 
      </ns0:OrdersBatch>
    KiranMP
    Tuesday, June 9, 2009 10:50 AM
  • Hi

    I think using the flat file disassembler on the way in is much prettier than calling it from an orchestration, which I think should be avoided if possible.

    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 10:52 AM
  • Hi,

    I agree with Ealisen. You can debatch the Flat File at receive Pipeline level And also the xml you posted in the 1st post is not debatched. Correct me if I am wrong in your context, I guess by debatching you mean  seperating each Order Record. And after that you have a map where you are mapping(Order Header and Items info) one Order at a time.

    You can create Header and Detail, in your case the Detail would be whole Order record, and then debatch the Flat File in receive pipeline.

    Ajeet Kumar
    Tuesday, June 9, 2009 11:02 AM
  • I am sure you are Right Abdul...that why I try to avoid long posts :)

    Also I am sure we both are saying the same thing. I have explained why he was getting the error and in what scenario.
    With the current FlatFile it will never work debatch at Pipeline level. And I am trying to put stress on if there is a standard way then we should follow that until its completely out of our control to go for some work around.

    Debatching FlatFile at Pipeline level will be best rather than at Orchestration level using xpath and this is the last option you should keep. So if anyway we can change the File generation process (if its in our control and not involve heavy process) so that we can debatch the FlatFile as soon we get at ReceiveLocation.

    If changing the file is not possible then we need to have some work around.We can have several solutions but we should proceed with the best and feasible one.
    Ajeet Kumar
    Tuesday, June 9, 2009 2:18 PM

All replies

  • For Debatching you can use xpath function and assign it in a message of a new type that will hold the single record. Make a new schema that will hold the single order record.

    ordersinglemessage = xpath(ordermessage,"xpath of the Orders node");

    ordersinglemessage will hold single order record and you can loop through all the records in the orchestration. Just count the no of records and loop through. You can use xpath to get the no. of orders.

    count = xpath(ordersmessage, "count(xpath of the Orders node)");


    Thanks,
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this answer if it helps
    Tuesday, June 9, 2009 9:39 AM
  • Changing the maxOccurs is the right way to go. I am surprised that you get an error. Doesn't the error have more information than what you have posted?


    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 10:06 AM
  • have you set child order? if possible try yo post schema


    KiranMP
    Tuesday, June 9, 2009 10:10 AM
  • <?xml version="1.0" encoding="utf-16" ?>
    - <xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://DebatchFlatFile.OrdersBatch" targetNamespace="http://DebatchFlatFile.OrdersBatch" xmlns:xs="http://www.w3.org/2001/XMLSchema">
    - <xs:annotation>
    - <xs:appinfo>
      <schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
      <b:schemaInfo standard="Flat File" codepage="65001" default_pad_char="" pad_char_type="char" count_positions_by_byte="false" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" early_terminate_optional_fields="false" allow_message_breakup_of_infix_root="false" compile_parse_tables="false" root_reference="OrdersBatch" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:element name="OrdersBatch">
    - <xs:annotation>
    - <xs:appinfo>
      <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:complexType>
    - <xs:sequence>
    - <xs:annotation>
    - <xs:appinfo>
      <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:element maxOccurs="1" name="Orders">
    - <xs:annotation>
    - <xs:appinfo>
      <b:recordInfo structure="delimited" child_delimiter_type="hex" child_delimiter="0xD 0xA" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:complexType>
    - <xs:sequence>
    - <xs:annotation>
    - <xs:appinfo>
      <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:element name="OrderMain">
    - <xs:annotation>
    - <xs:appinfo>
      <b:recordInfo tag_name="ORDER" structure="delimited" child_delimiter_type="char" child_delimiter="|" child_order="infix" sequence_number="1" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:complexType>
    - <xs:sequence>
    - <xs:annotation>
    - <xs:appinfo>
      <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:element name="OrderID" type="xs:string">
    - <xs:annotation>
    - <xs:appinfo>
      <b:fieldInfo justification="left" sequence_number="1" />
      </xs:appinfo>
      </xs:annotation>
      </xs:element>
    - <xs:element name="Date" type="xs:string">
    - <xs:annotation>
    - <xs:appinfo>
      <b:fieldInfo justification="left" sequence_number="2" />
      </xs:appinfo>
      </xs:annotation>
      </xs:element>
      </xs:sequence>
      </xs:complexType>
      </xs:element>
    - <xs:element maxOccurs="unbounded" name="Item">
    - <xs:annotation>
    - <xs:appinfo>
      <b:recordInfo structure="positional" sequence_number="2" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:complexType>
    - <xs:sequence>
    - <xs:annotation>
    - <xs:appinfo>
      <groupInfo sequence_number="0" xmlns="http://schemas.microsoft.com/BizTalk/2003" />
      </xs:appinfo>
      </xs:annotation>
    - <xs:element name="ID" type="xs:string">
    - <xs:annotation>
    - <xs:appinfo>
      <b:fieldInfo justification="left" pos_offset="0" pos_length="7" sequence_number="1" />
      </xs:appinfo>
      </xs:annotation>
      </xs:element>
    - <xs:element name="Descr" type="xs:string">
    - <xs:annotation>
    - <xs:appinfo>
      <b:fieldInfo justification="left" pos_offset="0" pos_length="12" sequence_number="2" />
      </xs:appinfo>
      </xs:annotation>
      </xs:element>
    - <xs:element name="Qty" type="xs:string">
    - <xs:annotation>
    - <xs:appinfo>
      <b:fieldInfo justification="left" pos_offset="0" pos_length="1" sequence_number="3" />
      </xs:appinfo>
      </xs:annotation>
      </xs:element>
      </xs:sequence>
      </xs:complexType>
      </xs:element>
      </xs:sequence>
      </xs:complexType>
      </xs:element>
      </xs:sequence>
      </xs:complexType>
      </xs:element>
      </xs:schema>


    Shady
    Tuesday, June 9, 2009 10:15 AM
  • I am afraid, This what the description about error I was able to find out using event log.
    Shady
    Tuesday, June 9, 2009 10:16 AM
  • Flat file disassembler can be used. You can call the pipeline in your orchestration as you already have your flat file message. You will get an enumerator and then you can iterate through the collection of your messages. Here is a post on my blog which explains it may be it will help.

    http://abdulrafaysbiztalk.wordpress.com/2008/08/10/calling-send-and-receive-pipelines-from-the-orchestration-expression-shapes/
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this answer if it helps
    Tuesday, June 9, 2009 10:16 AM

  • I was able to debatch this xml in pipeline, only difference i see is prefix
    <ns0:OrdersBatch xmlns:ns0="http://DebatchFlatFile.OrdersBatch">
     <Orders >
     <OrderMain>
      <OrderID>100</OrderID>
      <Date>2009-04-01</Date>
      </OrderMain>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders> 
    <Orders>
     <OrderMain>
      <OrderID>100</OrderID>
      <Date>2009-04-01</Date>
      </OrderMain>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>5</Qty>
      </Item>
     <Item>
      <ID>IT-1000</ID>
      <Descr>Fuse Bulbs</Descr>
      <Qty>2</Qty>
      </Item>
      </Orders> 
      </ns0:OrdersBatch>
    KiranMP
    Tuesday, June 9, 2009 10:50 AM
  • Hi

    I think using the flat file disassembler on the way in is much prettier than calling it from an orchestration, which I think should be avoided if possible.

    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 10:52 AM
  • Hi,

    I agree with Ealisen. You can debatch the Flat File at receive Pipeline level And also the xml you posted in the 1st post is not debatched. Correct me if I am wrong in your context, I guess by debatching you mean  seperating each Order Record. And after that you have a map where you are mapping(Order Header and Items info) one Order at a time.

    You can create Header and Detail, in your case the Detail would be whole Order record, and then debatch the Flat File in receive pipeline.

    Ajeet Kumar
    Tuesday, June 9, 2009 11:02 AM
  • Hi,
    XML Message specified is what I receive using my Flat File disassembler receive pipeline. Now, I want this message to be split into separate messages on the bases of orders element which si the real task.
    In order to do that, when I try to put MAx occurance for the element Orders, it gives me the error.

    Shady
    Tuesday, June 9, 2009 11:06 AM
  • well schema u have given doesnt match with the input you are trying to send(debatch) try to  change input , if you cant then follow what abdul suggest debatch the file in orchestration

    KiranMP
    Tuesday, June 9, 2009 12:00 PM
  • Just out of curiousity; What makes you think the pipeline called form an orchestration can parse the input, if the same pipeline in a receive location can't? Is there some feature I am not aware of?
    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 12:02 PM
  • Can you send me your flat File thru email (ajk86in@gmail.com). I will try debatching and let you know the details.
    Ajeet Kumar
    Tuesday, June 9, 2009 12:02 PM
  • Hi Ajeet. His instance is posted in the top post.

    It would be nice to know, though, if the instance has a 0x0d 0x0a at the end of the file or not...

    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 12:04 PM
  • hi ajeet he gave input as

    ORDER100|2009-04-01
    IT-1000Fuse Bulbs  5
    IT-1000Fuse Bulbs  2
    ORDER101|2009-06-01
    IT-1000Fuse Bulbs  5
    IT-1000Fuse Bulbs  2

    KiranMP
    Tuesday, June 9, 2009 12:04 PM
  • Thats why i suspect there is some problem with FlatFile or FlatFile Schema..:)
    Try validating this sample with the schema(first update schema for Order Maxoccur=*, just to see if the whole instance gets validated or not)
    Ajeet Kumar
    Tuesday, June 9, 2009 12:16 PM
  • Hi,

    I think we can specify Maxoccurance to a repeating type of record. So, Orders has to be a record type, but if that is the case then Order starts appearing under <Item> element because Item record is a repeating record.
    Shady
    Tuesday, June 9, 2009 12:16 PM
  • I will, but flat file is just the data I have pasted. But anyhow It's on your way.....
    Thanx for the effort..
    Shady
    Tuesday, June 9, 2009 12:17 PM
  • Just out of curiousity; What makes you think the pipeline called form an orchestration can parse the input, if the same pipeline in a receive location can't? Is there some feature I am not aware of?
    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk

    Eliasen,

    I think that he is using the custom pipeline to disassemble the flat file that converts the flat file into the XML. Can he use the same pipeline for debatching?
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this answer if it helps
    Tuesday, June 9, 2009 12:29 PM
  • All flat file handling must be done in a custom pipeline.

    And the pipelines are the same - whether you call them from within an orchestration or from a receive location.

    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    • Proposed as answer by Abdul Rafay Tuesday, June 9, 2009 1:12 PM
    Tuesday, June 9, 2009 12:43 PM
  • Hi,

    The problem is with the FlatFile and the schema. There are two scenario
    1) When maxoccur for Orders is 1 and use custom pipeline is being used to debatch the FlatFile.
    Then in that case, you will get Error with Discription "The Messaging Engine encountered an error during the processing of one or more inbound messages" the one posted by Raminder with Reason as "The remaining stream has unrecognizable data.
    ".
    I will explain this error.If you validate the input with Order MaxOccur=1 then the validated instance will look like below.Check the last Item node, here its having problem and because of the same reason disassembler will throw error as it will not be able to Identify the remaining stream. And hence can not debatch at Receive Port.

    <OrdersBatch xmlns="http://DebatchFlatFile.OrdersBatch">
        <Orders xmlns="">
            <OrderMain>
                <OrderID>100</OrderID>
                <Date>2009-04-01</Date>
            </OrderMain>
            <Item>
                <ID>IT-1000</ID>
                <Descr>Fuse Bulbs</Descr>
                <Qty>5</Qty>
            </Item>
            <Item>
                <ID>IT-1000</ID>
                <Descr>Fuse Bulbs</Descr>
                <Qty>2</Qty>
            </Item>
            <Item>
                <ID>ORDER10</ID>
                <Descr>1|2009-06-01</Descr>
                <Qty>
                </Qty>
            </Item>

        </Orders>
    </OrdersBatch>


    2) Now update the Orders MaxOccur=unbounded . Doing so, dissassembler won't have any problem, but it will not bebatch the message, just convert to XML.In that case you can debatch the XML message inside the Orchestration and seperate out each Orders.But you can't use any pipeline here as it will not work. You have to use xpath and seperate out each Orders record.
    But rather than going with this, I would suggest to verify the input FlatFile(the system generating the flat file) and if it is possible to use different delimeters for each record then it would be best.

    Ideally the debatching should happen at the Receive Port level untill its totally impossible to do so else it will be overhead for BizTalk...:)

    Ajeet Kumar
    Tuesday, June 9, 2009 1:01 PM
  • All flat file handling must be done in a custom pipeline.

    And the pipelines are the same - whether you call them from within an orchestration or from a receive location.

    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk

    Sorry Accidentaly clicked on Mark as answer button.

    I agree the pipeline won't change from where ever you call and will give the same result. But the pipeline he created was for the purpose of disassembling the flat file and not for debatching his XML. He made the pipeline he got his XML. Now he wants to debatch his multiple occuring records.
    Now there should be another way of doing that like using an xpath function.
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this answer if it helps
    Tuesday, June 9, 2009 1:15 PM
  • Right, sorry. If he is receiving the XML inside his orchestration and wants to debatch that, then he can use the builtin XMLReceiv epipeline - he just needs to mark his schema as an envelope and have anoter schema that matches one "Orders" element.
    eliasen, representing himself and not the company he works for.
    MVP and three times MCTS in BizTalk.
    Blog: http://blog.eliasen.dk
    Tuesday, June 9, 2009 1:19 PM
  • Ajeet,

    I would disagree that he has used a flat file dissasembler in the disassemble stage of his custom pipeline. So the only functionality that the pipeline will give is converting his flat file into XML. What are you expecting from the pipeline to give multiple XML's as single records to his receive message?

    That is possible if he has MaxOccur set to greater than 1 then lets say if he has 10 records in the flat file and he has set it to 2, he will have 5 instances of the orhcestration running at a time concurrently. I dont think thats what he wants. And if he wants that he has to create a new flat file schema, with no multiple orders but single order and then he can have multiple orchestrations running concurrently equal to the records in the flat file.

    I had a scenario once like this.
    Abdul Rafay http://abdulrafaysbiztalk.wordpress.com/ Please mark this answer if it helps
    Tuesday, June 9, 2009 1:20 PM
  • I am sure you are Right Abdul...that why I try to avoid long posts :)

    Also I am sure we both are saying the same thing. I have explained why he was getting the error and in what scenario.
    With the current FlatFile it will never work debatch at Pipeline level. And I am trying to put stress on if there is a standard way then we should follow that until its completely out of our control to go for some work around.

    Debatching FlatFile at Pipeline level will be best rather than at Orchestration level using xpath and this is the last option you should keep. So if anyway we can change the File generation process (if its in our control and not involve heavy process) so that we can debatch the FlatFile as soon we get at ReceiveLocation.

    If changing the file is not possible then we need to have some work around.We can have several solutions but we should proceed with the best and feasible one.
    Ajeet Kumar
    Tuesday, June 9, 2009 2:18 PM