none
biztalk send port to MQ server: segmentation and reasembly process RRS feed

  • Question

  • Hello,

    I have an orchesration that generate a file and send it trough  a send port  to MQ server.

    everything is  ok: if  the file  > 4MB, i can send it because i activate the segmentation process.

    it's just that the queue/chanel  that get the message reassebled and not  N segments.

    the probleme is that  the channel can support only  4mb files, and i would like that  in MQ Server side receive my file segmented (N segments of  4MB  each)  and not reasembled.

    exemple :  a file of 6mb. if the chanel is 10 mb size, i can send it. but once  it 's  4mb,  they can not receive it.

    even if i have my segmentation  process activated.  they receive my file reassembled

    is there any  property to  do  in biztalk or in mq server  to avoid the reassebling of a message  after being segmented?

    Thanks


    Thursday, July 6, 2017 2:46 PM

All replies

  • So....do you have any alternative to MSMQ?  While we all love MSMQ, the 4MB limit is frequently treated as a showstopper if it comes up.

    Thursday, July 6, 2017 2:56 PM
  • it's client   strategy.

    it's mandatory  to  send it to mq server.

    the idea  is that mq server   get the segments of  my file  and not the file  reassembled.

    Thursday, July 6, 2017 2:58 PM
  • Ah ha!

    Nothing's ever as 'mandatory' as it sometimes sounds.  In this case, the first thing you need to do is tell them, very correctly, that MSMQ is not a viable solution for this application's runtime characteristics.  It's really their responsibility to work with you to identify a workable solution.

    But, if they are unable or unwilling to advance their internal services, the next think you have to do is inform them, very correctly, that it will cost a lot more time and money to accommodate the MSMQ restriction.

    Can explain what the 'segmentation' process is?  That's not something that exists in BizTalk or MSMQ.  Is that a custom program?

    Thursday, July 6, 2017 5:34 PM
  • Hi john,

    i know  that MQS is not a great solution for big files

    the idea is just to send files  morr than 4mb and less than 12mb.

    it  work  when the MQ channel is up to 12mb.

    the segmentation process  it's a metada    of  MQHeader   (biztalk request put to mq server).

    when it's   activated, i can send my file  to this  channel. it's just that mq server get the hole file reassembled  and not just all segmentes

    Friday, July 7, 2017 8:09 AM
  • Check this thread
    https://social.msdn.microsoft.com/Forums/en-US/f779a6a2-2b2b-471e-816f-11632970c877/max-file-size-that-msmq-support?forum=biztalkgeneral

    There was an extension (a dll called Mqrtlarge.dll) created by Microsoft to get around this limitation

    /Peter

    Friday, July 7, 2017 9:34 AM
  • Sorry, it's still not clear what you're trying to do.

    Why don't you want the receiver to get the entire file?  If you're sending a 12MB message, the receiver should get a 12MB message.  It's not really a good pattern to expect the protocol to chunk in any way.

    Friday, July 7, 2017 11:17 AM
  • It's a standard pattern to dissamble/assemble messages in chunks
    In most cases it's transparent to the implementation

    eg Chunked transfer encoding in the HTTP protocol

    I've been using MSMQ as part of a 3rd party middleware solution for 9 years without issues
    We had a datalayer to send/receive messages (IDOCs)
    I assume the datalayer was implemented similar to this
    https://support.microsoft.com/en-us/help/198686/how-to-send-files-larger-than-4-mb-by-using-microsoft-message-queuing

    /Peter


    Friday, July 7, 2017 11:32 AM
  • John,

    i want the receiver to get all segments of  4mb  and  not the  entire file.

    i think that there is some metadata   to  force this .....

    when my  metadata  segmentation is activated, i can  send the file from biztalk to the receiver  .

    my biztalk mqsc adapter with segmentation works fine...it's just that the receiver  get the entire file  reassembled  and not all segments.

    the limitation of transfert  by default  is 4mb.   the mq team   want that  the  channel remain on 4mb.

    i can do  the segmentation to 4mb each segment.

    i can send the file from biztalk to mq server.

    the probleme  is that mq server  get the entire  file reassembled  and  not   by segments

    Friday, July 7, 2017 1:43 PM
  • thank you Peter   for this answer.

    in fact, this is what i am using and  i can  send  the file when it's more than 4mb.

    what i want to do is that the receiver get the segments of  my file  and not my entire file reassembled.

    why?  the channel  limitation is 4mb,  and each segment my adapter can produce is 4mb  too.  but  once the file is  6mb, my segmentation process  will generate 2 segments,  but my receiver get the entire  file and not  just 2 segments.....there is some reassebling process that  just happend but i want to know if i can disabled   in biztalk or  does  the receiver ( mq server)   to deal  with this.

    i think  there is  some metadadata  in header to use  bu i have no idea  

    Friday, July 7, 2017 1:49 PM
  • You cannot vote on your own post                            
    0
                    

    thank you Peter   for this answer.

    in fact, this is what i am using and  i can  send  the file when it's more than 4mb.

    what i want to do is that the receiver get the segments of  my file  and not my entire file reassembled.

    why?  the channel  limitation is 4mb,  and each segment my adapter can produce is 4mb  too.  but  once the file is  6mb, my segmentation process  will generate 2 segments,  but my receiver get the entire  file and not  just 2 segments.....there is some reassebling process that  just happend but i want to know if i can disabled   in biztalk or  does  the receiver ( mq server)   to deal  with this.

    i think  there is  some metadadata  in header to use  bu i have no idea  

    Friday, July 7, 2017 1:49 PM
  • Check with the MSMQ forum
    https://social.msdn.microsoft.com/Forums/windowsdesktop/en-US/home?forum=msmq

    Imo you must have a wrapper to change the "protocol" and have msmq as a part of the process

    Who is responsible of the MSMQ server you or your partner?

    /Peter

    Friday, July 7, 2017 2:33 PM
  • the partner  who is responsible for mq server.

    in biztalk side, i have juste  the connction settings (channel, queue and queue manager )

    Friday, July 7, 2017 2:42 PM
  • You need an API/interface to the queue from your partner otherwise you'd probably spend a lot of trustrating evenings to get it to work properly

    The alternative is a new "MSMQ" adapter where the protocol rules are implemented
    Imo I'd rather have an API

    /Peter

    Friday, July 7, 2017 2:59 PM
  • What we're asking is why you want to send 12MB but have the receiver get 3 x 4MB.  That is highly, highly unusual.

    Chunking is not an uncommon feature, but as a protocol feature, the expectation is that what is chunked will be un-chunked so that the receiver gets exactly what was sent, meaning not modifications at all.


    Friday, July 7, 2017 3:13 PM
  • exactely man...

    but we have a limitaion in the receiver side of 4mb  tough  the channel.

    for now the receiver  get the entire file, because we change  the  mq channel size  in dev. 

    in go live env,  the receiver  is limited to 4mb....that's  why  i am  asking if it's possible to  force  the receiver  to not get the entire file  but  only  the segments ....

    Friday, July 7, 2017 3:26 PM
  • So, what exactly do you mean by 'receiver side'?  The app, MSMQ?  Have you tried, is there an error?

    Next, why can't they change it?  4MB is really small.

    Finally, it's highly unlikely what you asking will work.  Protocol chunking is done in an arbitrary way, without any consideration to the actual content.  So, if you send a 12MB message, what you get will be 3 invalid 4MB messages.  This is why chunking/unchunking happens automatically, to preserve message integrity.

    For this to be a reliable process, either they have to increase the receive limit to whatever the sender may produce, or  you will have to manage the message size yourself (meaning lots more time and money) to stay under the 4MB limit.

    This is not a problem with you, your app, BizTalk Server or even MSMQ.  This is a limitation created by either the receiving app or policy.

    Friday, July 7, 2017 4:44 PM
  • Yes a new layer is needed on top of the protocol stack to make it work

    /Peter

    Friday, July 7, 2017 5:06 PM
  • Thank you  John for those advices
    Monday, July 10, 2017 7:50 AM
  • Thnak you Peter  for your intervention  on my post!
    Monday, July 10, 2017 7:50 AM