Virtual Stream in Disassembler Pipeline RRS feed

  • Question

  • Hi All,

    I am trying to loop through in Custom Pipeline using serialization and break the records at INS level and read the messages using GetNext. But I have tough time with Virtual Stream. I am able to do with Memory stream but Virtual Stream is giving issues. Can any one let me know on this? I have mentioned below the Error and Code. Am I doing something wrong with Virtual Stream?


    xlang/s engine event log entry: Uncaught exception (see the 'inner exception' below) has suspended an instance of service 
    The service instance will remain suspended until administratively resumed or terminated. 
    If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
    InstanceId: f502a2c8-59c6-4c48-8206-e43e15f17827
    Shape name: XXXXXXX
    ShapeId: 7ae7ac8e-ee82-4378-b297-3f64c97eb617
    Exception thrown from: segment 2, progress 18
    Inner exception: Data at the root level is invalid. Line 102, position 20.
    Exception type: XmlException
    Source: System.Xml
    Target Site: Void Throw(System.Exception)
    The following is a stack trace that identifies the location where the exception occured

       at System.Xml.XmlTextReaderImpl.Throw(Exception e)
       at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
       at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
       at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
       at System.Xml.XmlTextReaderImpl.Read()
       at System.Xml.XmlTextReader.Read()
       at Microsoft.XLANGs.Core.Service.CompositeStreamReader.Read()
       at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
       at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
       at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
       at System.Xml.XmlDocument.Load(XmlReader reader)
       at Microsoft.XLANGs.Core.Service.ApplyInMemoryTransform(Type mapRef, TransformMetaData trfMetaData, Object[] outParams, Stream[] inStreams)
       at Microsoft.XLANGs.Core.Service.ApplyTransform(Type mapRef, Object[] outParams, Object[] inParams)
       at Input_EDI_Process._834_EDI_Process.segment2(StopConditions stopOn)
       at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)


            private Queue<IBaseMessage> _MessageQueue = new Queue<IBaseMessage>();

            public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
                Stream originalDataStream = pInMsg.BodyPart.GetOriginalDataStream();
                 Create834Stream(pContext, pInMsg.Context, originalDataStream, pInMsg);   

                if (!originalDataStream.CanSeek)
                     ReadOnlySeekableStream seekableStream = new ReadOnlySeekableStream(originalDataStream);
                   seekableStream.Position = 0;
                    seekableStream.Seek(0, SeekOrigin.Begin);
                 originalDataStream = seekableStream;

                Create834Stream(pContext, pInMsg.Context, originalDataStream , pInMsg);


            //Virtual Stream

            private void Create834Stream(IPipelineContext pContext, IBaseMessageContext context, Stream originalDataStream, IBaseMessage pInMsg)
                var outputStream = new VirtualStream();

                IBaseMessage Msg = null;
                string messageType = "";
                string systemNamespace = "";

                using (var msgReader = XmlReader.Create(originalDataStream))
                    while (msgReader.Read())
                        XmlSerializer deser = new XmlSerializer(typeof(X12_00501_834));
                        X12_00501_834 input = (X12_00501_834)deser.Deserialize(msgReader);
                        int i = 0;

                        foreach (TS834_2000_Loop TS834_2000 in input.TS834_2000_Loop)
                            i = i + 1;
                            X12_00501_834 Split834Record = new X12_00501_834();
                            Split834Record.ST = input.ST;
                            Split834Record.BGN_BeginningSegment = input.BGN_BeginningSegment;
                            Split834Record.REF_TransactionSetPolicyNumber = input.REF_TransactionSetPolicyNumber;
                            Split834Record.DTP_FileEffectiveDate = input.DTP_FileEffectiveDate;
                            Split834Record.QTY_TransactionSetControlTotals = input.QTY_TransactionSetControlTotals;
                            Split834Record.N1_SubLoop = input.N1_SubLoop;
                            Split834Record.SE = input.SE;
                            Split834Record.TS834_2000_Loop = new TS834_2000_Loop[1];
                            Split834Record.TS834_2000_Loop[0] = TS834_2000;

                            //Create our own namespaces for the output
                            XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
                            //Add an empty namespace and empty value
                            ns.Add("", "");

                            var streamWriter = new StreamWriter(outputStream, System.Text.Encoding.UTF8);

                            XmlSerializer serializer = new XmlSerializer(typeof(X12_00501_834), "");
                            serializer.Serialize(streamWriter, Split834Record, ns);
                            streamWriter.AutoFlush = true;
                            outputStream.Seek(0, SeekOrigin.Begin);

                            Msg = pContext.GetMessageFactory().CreateMessage();
                            Msg.AddPart("Body", pContext.GetMessageFactory().CreateMessagePart(), true);
                            Msg.BodyPart.Data = new MemoryStream();
                            outputStream.Position = 0;
                            Msg.BodyPart.Data = outputStream;

                            // Copy properties from the original     
                            Msg.Context = context;
                            Msg.Context.Promote("MessageType", systemNamespace, messageType);

                            // queue the split message for processing    

                System.Diagnostics.EventLog.WriteEntry("Count Messages", _MessageQueue.Count().ToString());


            public IBaseMessage GetNext(IPipelineContext pContext)
                IBaseMessage msg = null;

                if ((_MessageQueue.Count > 0))
                    msg = ((Microsoft.BizTalk.Message.Interop.IBaseMessage)(_MessageQueue.Dequeue()));
                return msg;

    • Edited by HariKotha Tuesday, January 2, 2018 9:00 PM
    Tuesday, January 2, 2018 8:59 PM

All replies


    You do not need to do any of this.

    The 834 Schema already splits on the 2000 Loop (which is the Member level).

    Just make sure you're using the .xsd from the multiple folder and have not set Preserve Interchange.

    Wednesday, January 3, 2018 2:02 PM
  • Hi John,

    The Multiple Schema provided by Biztalk doesnt do this. I have compared with Ultraedit and compared both Single and Multiple Schema and there is no different. However I am able to modify the Multiple Schema as per Microsoft to break at INS level. The problem I have with that that is after I drop the file in a particular location, it is taking lot of time to get picked up by the Biztalk when there are 80K members to get loaded. The Biztalk starts processing as soon I dropped it but it takes an hour to get the file disappeared from the folder when the records are more in volume. It runs for about an 1hr 30 mins to get it processed if Split using Multiple Schema for 80K records.

    That is the reason I am going with this approach.

    Wednesday, January 3, 2018 2:12 PM
  • HOLD ON!

    The multiple Schema 100% debatches at Loop 2000/INS.  I've done it, many times.  You have to look more closely for the differences, it's only 2 schema attributes.

    So, with 80k members, what you're seeing is not unusual.  Is 90 minutes to process an actual problem?  Meaning you are not meeting a specific, documented SLA because of this?  If not, maybe just let it run, with the proper tuning of course.

    The problem is that debatching is transnational and there is no way to disable this behavior, which I will concede has caused problems in the past.

    Don't bother with deserializing the Xml or Schema types, it doesn't get you anything and is actually making this way more complicated than it needs to be.

    What I have done is wrap the Edi Disassembler, then call GetNext() internally until debatching is complete while writing each message to either the file system or MSMQ.  The individual message can them be picked up and processed while the rest of the batch is being disassembled.

    Wednesday, January 3, 2018 2:40 PM
  • Thanks John. 

    When you say "Wrapping the EDI Disassembler and Calling GetNext()", Did you write separate C# or was this done in Pipeline?

    I have to load the ISA and GS segment information in Master table and get the Key back which need to stored in Detail table (which has Key and and all the member related data along with ST information). If I pass to MSMQ, how can I pass the Key also along with message? 

    The Application which inserts into MSMQ after debatching and the one which reads from MSMQ need to be different? Sorry I am new to Biztalk hence asking these questions. 

    • Edited by HariKotha Wednesday, January 3, 2018 5:10 PM
    Wednesday, January 3, 2018 5:02 PM
  • Basically, inherit from Microsoft.BizTalk.Edi.Pipelines.EdiDisassmebler, then override Disassemble() and GetNext().  It's just regular C# programming, nothing really BizTalk specific about that.

    Well, do you really have to get a table key back first?  I've seen this before too and, well, that also provides almost no benefit and just makes things harder than they need to be.

    Also, all the ISA and GS information is automatically stored and searchable by BizTalk.  Anything else you do is just duplication.  So, make sure EDI Status Reporting doesn't already address your needs.

    You can still use the InterchangeID or even generate your own ID, then set the MSMQ Label property to correlate (not necessarily in the BizTalk way, but maybe) all the messages.

    You can read the individual message by BizTalk and process them as you would any other message.

    Wednesday, January 3, 2018 5:48 PM
  • Thank you John for your suggestion. I will look in to this.

    • Edited by HariKotha Wednesday, January 3, 2018 6:41 PM
    Wednesday, January 3, 2018 6:38 PM