none
HL7 - MLLP Adapter Custom Decoder BodyPart stream length=0 RRS feed

  • Question

  • OK, had a good one here. I don't know if anybody can help me with this. This seems to be a bug in the BizTalk HL7 Accelerator's MLLP Adapter. 

    We are using a custom decoder to save raw HL7 messages, with some extra info, to an archiving database. This custom decoder seemed to work fine, until we were receiving larger messages. Larger means 10000 Kb, yes, large is relative :) When receiving larger messages, the BodyPart stream length became 0 bytes. If you process the message (ignoring it in the custom decoder), it will just process correctly, the message is there, also in the BizTalk MsgBox.

    I have been investigating it became even weirder. The body stream length of the larger messages sometimes was correct, say 14000 Kb, and the second time you try it, it's 0, the third time it's 0, the fourth time it's 14000 Kb again. This behavior occurs totally out of the blue. And since I don't have the MLLP adapter code, it's impossible to see where this is coming from.



     

    You can understand I am pretty much lost over here and I am not far away from filing a bug report here. I've read another post, also coping with the same problem. The poster claims the length may not exceed 32 KB, but that's weird, because I get the behavior at around 10 KB. 

    http://social.msdn.microsoft.com/Forums/eu/biztalkgeneral/thread/6ab58f7a-30e2-48dd-a58c-dc44a80ac862

    Ok, another thing. This behavior does NOT occur of you are using a different adapter, e.g. the FILE adapter. Every message gets processed by the custom decoder correctly. The difference between the two is that the MLLP adapter uses a different message type and stream than the FILE adapter. The first being the MLLP adapter, the second the FILE adapter. I guess that’s maybe where it can go wrong.


    I don’t know if somebody can help me with this one, or encountered something similar? Or does it sound like a buggy MLLP adapter?

     


    Monday, February 6, 2012 1:20 PM

Answers

  • OK Solution found! :)

    Important to understand is that the MLLP adapter delivers a somewhat different stream than e.g. a FILE adapter. So straightforward streamreading is not an option. Well, actually it is, after a coversion.

    I had to convert the message's OriginalDataStream to a ReadOnlyNonSeekable-stream and then the properties of the stream became available. After this conversion, you can just read the stream like a every seekable stream.

    Stream seekStream = new ReadOnlySeekableStream(inmsg.BodyPart.GetOriginalDataStream());

    For more info:
    http://msdn.microsoft.com/en-us/library/ee377071(v=bts.10).aspx

    Regards Rob

    • Marked as answer by Roborop Tuesday, February 14, 2012 2:29 PM
    Tuesday, February 14, 2012 2:29 PM

All replies

  • Is this your custom decoder code?  If it is, many of the decoder streams are actually created in memory because encryption, decryption, compression, decompression, signing, ... cannot be processed in a streaming fashion.  In some cases depending on a path through the logic, the stream is not reset to the beginning after the decoder/encoder processing... this means the downstream processing is pulling from a stream that has already been processed, leading to unpredictable results in the data returned downstream.

    Another issue with streaming is when reading a stream, the read loop must not be terminated if the amount of bytes returned is less then the size of the allocated buffer (see below).

    bytesRead = myStream.Read(buffer, offset, count);

    The stream contract does not require the amount returned to be the size of count... even if there is at least that much data remaining in the steam.  Make sure the termination check is for (bytesRead == 0) not (bytesRead < count).

    Another thing to keep in mind... not all streams present the length (take a video feed stream for instance).

    If it is your custom decoder, you can throw an exception before the return from the Execute() method if the length is 0 (assuming your implementation shouldn't return 0).  Internally, BizTalk processes messages in pipelines differently, depending the the operations available on the stream... file streams are seekable, but streams from wire transports are usually not.


    David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.


    Friday, February 10, 2012 5:11 AM
  • Hi David,

    first of all, thanks for your extensive reply. I had issues with a custom decoder (archiving component and the MLLP adapter). I built the decoder displayed above, just for testing purposes and it turned out the stream length was 0 about half of the time, the other half the stream length was OK. I was also able to read the stream in case of the length being OK, not when the length was 0, then it threw an error (the stream is read buffered).

    So you are saying there is not always a stream in the custom decoder? And this is arbitrary? That would be quite a bug then, because, isn't that what the custom decoder is all about? Handling the stream. I have tested it with a different adapter (FILE) and I don't run into these issues with that adapter.

    Should I file a bugreport about this or not? Is this a known issue?


    regards Rob


    • Edited by Roborop Monday, February 13, 2012 7:22 AM thx
    Monday, February 13, 2012 7:21 AM
  • In this case I need a bit more information; the receive pipeline from what I understand is something like the following:

    <MLLP Adapter>   <ReceivePipeline> -->  <custom decoder> --> <HL7 disassembler> ...

    1. Within your custom decoder, you are seeing a length value of "0" for large messages, is that correct?
    2. Have you checked the seekable, and position properties of the stream from within your component?
    3. Have you attempted a simple read operation and stepped through it in the debugger when the length is "0"?
    4. Have you tried sending the data from the MLLP adapter through BizTalk without going through your decoder to see what is being published into the Message Box?

    Let me elaborate a bit on my previous post:

    • When processing a stream, it's important to check what the stream capabilities are when you're processing data from the stream.  If the stream is not seekable, and your processing requires seekability, you will need to wrap it in a stream that is capable of buffering to allow the seek operation (BizTalk provides a VirtualStream for this purpose).
    • Because not all streams have a known length, when you're processing a stream try not to depend on a length.
    • Rather than using a MemoryStream which has a maximum size, try using the VirtualStream instead.

    That said, there may be a bug in the MLLP adapter, but it's also possible the bug is in your custom decoder code.  Please try my suggestions above and if you would like to send me the source code for your custom decoder, you can zip it up and send it to my MSDN email at: ddownin@hotmail.com

    I'll take a look and see if I can see anything.


    David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.


    Monday, February 13, 2012 4:20 PM
  • Hi David,

    thx again! Indeed everything is fine after it passes the decoder. I have outlined it below. It's the weirdest thing if you ask me. I have only tested it in 2009 configuration at the hospital itself. I will also do a test run through 2010 later on this week.


    (Click to enlarge)

    This is what happens:

    1. Decoder: "Large" message passes through the decoder. Sometimes the stream is there, sometimes it isn't. If the stream isn't there, trying to read it will generate an error. I think it was a "stream is unseekable"-error. I can try this tomorrow at the location.
      I have tested this with our original archiving component and with an "empty" pipeline component in a new project. Both with the same problem.
      Also, GetOriginalDataStream() doesn't return a stream.
    2. Disassembler: OK. The stream is there, message gets parsed.
    3. Validate: OK. The stream is there. Gets through correctly.
    4. Resolve Party: Dunno, didn't need it, haven't tested it.
    5. MessageBox: OK. Message is there, togerther with the generated acknowledgement.

    We are archiving the message in the decoder stage to preserve the originally submitted message. We have skipped the archiving component for the time being and everything is running OK. When we switch it on, the error occurs (duh). It happens on the development, test and production machines, so it's not incidental.

    I always use the Pipeline Component Wizard to generate the pipeline code. 

    The following things are remarkable:

    • The stream is sometimes there, sometimes it isn't with "larger" messages (10+ Kb, hmz, indeed "large" ;) ).
    • The stream is always there if I use the FILE adapter, also with larger messages.
    • The type of the stream coming from a FILE adapter differs from the MLLP adapter:
      - FILE: Microsoft.BizTalk.Streaming.CEventingReadStream
      - MLLP: Microsoft.BizTalk.Streaming.BasicStreamWrapper

    I have sent you the custom component code from my first post. It's not much code, I just return the message it has received :)

    Regards Rob


    Monday, February 13, 2012 8:46 PM
  • I'll take a look when I get home... a couple more questions;

    1. Is this always reproducable with the same message?
    2. If it is, did you try a Passthru receive pipeline -> Passthru send pipeline with the same message?   (BTS.ReceivePortName = "your receive port" filter on a new file send port

    David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

    Monday, February 13, 2012 9:05 PM
  • OK Solution found! :)

    Important to understand is that the MLLP adapter delivers a somewhat different stream than e.g. a FILE adapter. So straightforward streamreading is not an option. Well, actually it is, after a coversion.

    I had to convert the message's OriginalDataStream to a ReadOnlyNonSeekable-stream and then the properties of the stream became available. After this conversion, you can just read the stream like a every seekable stream.

    Stream seekStream = new ReadOnlySeekableStream(inmsg.BodyPart.GetOriginalDataStream());

    For more info:
    http://msdn.microsoft.com/en-us/library/ee377071(v=bts.10).aspx

    Regards Rob

    • Marked as answer by Roborop Tuesday, February 14, 2012 2:29 PM
    Tuesday, February 14, 2012 2:29 PM
  • I sent the following proposed solution to his issue:

            public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
            {
                Stream bodyStream = inmsg.BodyPart.GetOriginalDataStream();
     
                // If this is an unseekable stream, wrap it in a VirtualStream to allow seekablility
                if (!bodyStream.CanSeek)
                {
                    // Wrap the stream
                    bodyStream = new VirtualStream(bodyStream);
     
                    // Replace the body stream on the IBaseMessage
                    inmsg.BodyPart.Data = bodyStream;
                }
               
                // ----------------------------------------------------------------
                // *** This is where you can log your data using the bodyStream ***
                // ...
                // *** Done Logging data ***
     
                // Reset the stream to the beginning for downstream components!!!!!!
                bodyStream.Seek(0, SeekOrigin.Begin);
     
                // Return the updated data stream
                return inmsg;
            }

    One thing to keep in mind with wrapping a stream to make the properties available; if the stream is non-seekable and the length is not available, the stream will read the entire contents into a buffer (both memory and possibly file overflow) just to obtain the length and then seek back to the beginning.  In the case where logging is done after obtaining the length, two full stream reads have occurred before BizTalk starts it's processing.  If possible, do your logging in a streaming fashion... this can eliminate an entire stream pass.  If the ReadOnlySeekableStream is wrapped one more time with a custom stream, both BizTalk's read and the logging read can be used simultaneously, this eliminates both unnecessary full stream reads.


    David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

    Tuesday, February 14, 2012 3:25 PM