none
BizTalk - multiple flat Files in a folder to be zipped and sent RRS feed

  • Question

  • I have a requirement where 5 flat files will be dropped into a folder, BizTalk needs to pick them, zip all those 5 files into 1 zip file and send out. How can I achieve this ?

    MBH

    Thursday, July 23, 2015 1:37 PM

Answers

  • If you need to receive multiple files from a folder (2, 3, 5 or more) and zip them into a single zip file... then there isn't a nice and easy way to accomplished that in BizTalk... because each file that is being pick up by the adapter is a different thread.

    You have two options:

    Option 1:

    • Create a convoy orchestrations, which is never a good choice

    Option 2 (The most simple way):

    • create a PowerShell that is listen that folder and that zip the files
    • Create a receive port\location with the mask "*.zip"
    • Create a send port with a filter for that receive port

    Sandro Pereira
    DevScope | MVP & MCTS BizTalk Server 2010
    http://sandroaspbiztalkblog.wordpress.com/ | @sandro_asp
    Oporto BizTalk Innovation Day | 14th March 2013 – Oporto, Portugal
    Please mark as answered if this answers your question.

    Thursday, October 22, 2015 9:26 PM
    Moderator

All replies

  • I have a requirement where I will be receiving 10 files, I have to pick them , zip those files in 1 folder and send to sftp . How can I achieve this task ?

    MBH

    Thursday, July 16, 2015 1:58 PM
  • There are a number of samples and Pipeline Components out there you can use.

    Start with the list of available Pipeline Components: http://social.technet.microsoft.com/wiki/contents/articles/11679.biztalk-list-of-custom-pipeline-components.aspx

    Thursday, July 16, 2015 2:03 PM
    Moderator
  • I hope your expectation is to have individual xml files in a Zipped container.

    BizTalk zip multiple large files in a pipeline component will help you to accomplish this.


    Please mark as answer or vote as helpful if my reply does

    Thursday, July 16, 2015 5:28 PM
    Moderator
  • the links sent doesnot support the framework, can the links be sent which supports VS2013 ?

    Is writing custom pipeline component only way to achieve this ?


    MBH

    Thursday, July 16, 2015 5:49 PM
  • Yes you need to develop a Pipeline to Zip the file ,there is no other way for file  compression in Biz Talk and it is good case to implement in custom Pipeline only .

    There are two ways you can achieve it either using System.IO.Compression.FileSystem assembly for Zip Class or using Ionic.Zip.dll(for BizTalk 2010 and prior version ).

     Ionic.Zip.dll which is being mentioned in link share above cannot be used as VS 2013 does not support Ionic version . More information is available at http://www.dllme.com/dll/files/ionic_zip_dll.html 

    For System.IO.Compression.FileSystem sample code you can take reference from below MSDN article

    https://msdn.microsoft.com/en-us/library/system.io.compression.ziparchive%28v=vs.110%29.aspx

    https://library.octopusdeploy.com/#!/step-template/actiontemplate-file-system-zip-directory-contents

    Thanks

    Abhishek

    Thursday, July 16, 2015 7:03 PM
  • the links sent doesnot support the framework, can the links be sent which supports VS2013 ?

    Is writing custom pipeline component only way to achieve this ?


    MBH


    Did you look at the existing Pipeline Components???
    Thursday, July 16, 2015 7:12 PM
    Moderator
  • Hi Abhishek, the html link sent doesn't work ? Can you please send a valid link

    MBH

    Thursday, July 16, 2015 7:33 PM
  • I did John, there are  IpropertyBag and other properties mentioned separately. I am following the wizard Pipeline Component project and that creates everything to me. I just need the exact code to be implemented in either execute, decode or encode stage.

    For my current requirement I get five different positional flat files, which I have to zip and sent to sftp folder.

    So is there directly code available where I can put in my wizard generated custom pipeline component in a particular stage using VS2013 ?


    MBH

    Thursday, July 16, 2015 7:42 PM
  • Well, my question is still the same.  Did you look at any of the existing Pipeline Components?

    This is very important.  Either they already accomplish what you need or provide sample code on how to do it.

    Thursday, July 16, 2015 8:40 PM
    Moderator
  • this is what error I am getting on taking the code from the project. I don't know where can I get Microsoft.BizTalk.Pipeline.dll and Ionic.Zip.dll ???

    the type or namespace name 'ZipOutputStream' could not be found


    MBH


    • Edited by JaguarsJag Friday, July 17, 2015 3:22 PM info
    Friday, July 17, 2015 2:54 PM
  • Hi Abhishek, the html link sent doesn't work ? Can you please send a valid link

    MBH

    Hi Sorry for the late reply ,

     Ionic.Zip.dll is being used for for Zipping files .It can be found athttp://dotnetzip.codeplex.com/ 

    This is the dll which is being used in most of the earlier zipper pipeline component . You can also find the Ionic dll at http://www.dllme.com/dll/files/ionic_zip_dll.html

    Microsoft.BizTalk.Pipeline.dll can be found at the BIzTalk Installtion directory at c:\Program Files (x86)\Microsoft BizTalk Server 2013\.


    Thanks

    Abhishek


    Friday, July 17, 2015 4:09 PM
  • So, none of the existing Pipeline Components meet your requirement?
    Friday, July 17, 2015 5:29 PM
    Moderator
  • this below code is not zipping any files ? I have flat files coming in, I am using Passthrough in my Receive Pipeline and setting RP on send port filter and using created a custom pipeline with below code. It is not zipping anything ? Is there a way to update this code  to zip all received files into 1 zipped file ?

    public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
            {
                IBaseMessagePart bodyPart = pInMsg.BodyPart;
                if (bodyPart != null)
                {
                    Stream originalStream = bodyPart.GetOriginalDataStream();
                    if (originalStream != null)
                    {
                        MemoryStream memStream = new MemoryStream();
                        using(ZipOutputStream zipOutputStream = new ZipOutputStream(memStream))
                        {
                            byte[] buffer = new Byte[1024];
                            zipOutputStream.CompressionLevel = Ionic.Zlib.CompressionLevel.BestCompression;
    
                            if (_password != null)
                                zipOutputStream.Password = _password;
                            if (_entryExtension != null)
                                if (_entryExtension.Length > 0)
                                    _entryExtension = "." + _entryExtension;
    
                            zipOutputStream.PutNextEntry(pInMsg.MessageID + _entryExtension);
                            int bytesRead = 1024;
                            while (bytesRead != 0)
                            {                            
                                bytesRead = originalStream.Read(buffer, 0, buffer.Length);                            
                                zipOutputStream.Write(buffer, 0, bytesRead);  
                            }
                        }                    
                        byte[] buff = memStream.GetBuffer();
                        MemoryStream ms = new MemoryStream(buff);
                        ms.Position = 0;
                        pInMsg.BodyPart.Data = ms;
                    }
                }
                return pInMsg;


    MBH

    Tuesday, July 21, 2015 8:03 PM
  • Not in a practical way because .Execute takes only a single message.

    You would have to either:

    1. Aggregate them into a Multi-Part Message and loop through those

    2. Pass a Collection of Messages to a Send Pipeline in an Orchestration and use the .Assemble Method.  In there, you would also loop.

    So, none of the existing components work for you?

    Tuesday, July 21, 2015 8:18 PM
    Moderator
  • In that case what is the use of using custom pipeline ? If practically it can zip 1 file at a time, then i would have given .zip file extension in the Send port itself. My requirement is I get on daily basis 5 flat files, have to send this into 1 zip file. How can I achieve this ?

    MBH

    Tuesday, July 21, 2015 8:23 PM
  • You can achieve this by following one of these patterns.

    1. Aggregate them into a Multi-Part Message and loop through those

    2. Pass a Collection of Messages to a Send Pipeline in an Orchestration and use the .Assemble Method.  In there, you would also loop.


    Thursday, July 23, 2015 1:47 PM
    Moderator
  • I don't want to aggregate 5 files into 1 large files, I want 5 files stay the same but zip them into 1 zip file. Does my question make sense ?

    MBH

    Thursday, July 23, 2015 2:01 PM
  • Yes, I understand perfectly what you're trying to do, add the 5 files to a single .zip.

    A third option would be zip code in Expression Shapes.

    Thursday, July 23, 2015 2:05 PM
    Moderator
  • what is zip code? If you are answering, please describe in detail.

    MBH


    • Edited by JaguarsJag Thursday, July 23, 2015 2:09 PM info
    Thursday, July 23, 2015 2:08 PM
  • Sorry, I'm not understanding.  You've already posted code to create .zip files.  You just have to loop and add each message.
    Thursday, July 23, 2015 3:43 PM
    Moderator
  • In earlier thread you have created custom Pipeline component as been described by BoatSeller and myself .

    You can call the pipeline inside your Orchestration and aggregate the incoming message and than you can zip and send it .What it requires is a loop shape and the counter to the number of files coming into BizTalk.

    Thanks

    Abhishek

    Friday, July 24, 2015 4:53 AM
  • I don't want the message to be aggregated into 1 large message, i.e ...I don't want 5 files to be accumulated into 1 big file.....I want all 5 files uniquely in a folder and create a zip file of that folder on daily basis.

    MBH

    Monday, July 27, 2015 1:50 PM
  • Yes, we understand that, that's never been in question.

    You have to gather all 5 messages and zip them into one archive.  You choices are still the same:

    1. Create a multi-part message for a Send Port.

    2. In an Orchestration, create a SendPipelineInputMessages collection and call the Send Pipeline there.

    Either way, you will have to loop through them and add each to the archive.

    Monday, July 27, 2015 2:13 PM
    Moderator
  • will this change the file names ? because file names should be same as received in source and while sending out .

    MBH

    Monday, July 27, 2015 2:31 PM
  • That's entirely up to you.  .zip will retain the relative paths so as long as you add the file correctly.

    Please just try.  You will find this is pretty simple thing to do.

    Monday, July 27, 2015 3:34 PM
    Moderator
  • so can you please give me detail step by step procedure right from step 1 to achieve this ?

    MBH

    Monday, July 27, 2015 3:38 PM
  • ... I don't know where can I get Microsoft.BizTalk.Pipeline.dll and Ionic.Zip.dll ???

    I still think that you are missing the point that John is making, that your solution either is implemented by a component or by sample code available at the location he provided.

    To answer you specific question above:

    Get Ionic.Zip.dll here: https://dotnetzip.codeplex.com/

    Get Microsoft.BizTalk.Pipeline.dll on your BizTalk installation root folder (i.e.: C:\Program Files (x86)\Microsoft BizTalk Server 2013)

    Tuesday, October 20, 2015 12:59 AM
  • If you want to see a step by step, try looking here: Zip Files in a Custom Pipeline Component

    http://www.codeproject.com/Articles/152405/Zip-Files-in-a-Custom-Pipeline-Component

    Tuesday, October 20, 2015 1:05 AM
  • If you need to receive multiple files from a folder (2, 3, 5 or more) and zip them into a single zip file... then there isn't a nice and easy way to accomplished that in BizTalk... because each file that is being pick up by the adapter is a different thread.

    You have two options:

    Option 1:

    • Create a convoy orchestrations, which is never a good choice

    Option 2 (The most simple way):

    • create a PowerShell that is listen that folder and that zip the files
    • Create a receive port\location with the mask "*.zip"
    • Create a send port with a filter for that receive port

    Sandro Pereira
    DevScope | MVP & MCTS BizTalk Server 2010
    http://sandroaspbiztalkblog.wordpress.com/ | @sandro_asp
    Oporto BizTalk Innovation Day | 14th March 2013 – Oporto, Portugal
    Please mark as answered if this answers your question.

    Thursday, October 22, 2015 9:26 PM
    Moderator