none
Issue with custom receive Pipeline component RRS feed

  • Question

  • I have been facing issue with creating a custom receive pipeline component. The Pipeline is to receive large file, if the file size is large it has to read the incoming stream to a folder and pass only some meta data through the MessageBox. The Execute method I am using is,

            #region IComponent Members
            public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
            {
                if (_largeFileLocation == null || _largeFileLocation.Length == 0)
                    _largeFileLocation = Path.GetTempPath();
    
                if (_thresholdSize == null || _thresholdSize == 0)
                    _thresholdSize = 4096;
    
                if (pInMsg.BodyPart.GetOriginalDataStream().Length > _thresholdSize)
                {
                    Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
                    string largeFilePath = _largeFileLocation + "\\" + pInMsg.MessageID.ToString() + ".zip";
    
                    FileStream fs = new FileStream(largeFilePath, FileMode.Create);
    
                    // Write message to disk
                    byte[] buffer = new byte[1];
                    int bytesRead = originalStream.Read(buffer, 0, buffer.Length);
                    while (bytesRead != 0)
                    {
                        fs.Flush();
                        fs.Write(buffer, 0, buffer.Length);
                        bytesRead = originalStream.Read(buffer, 0, buffer.Length);
                    }
                    fs.Flush();
                    fs.Close();
    
                    // Create a small xml file
                    string xmlInfo = "<ns0:MsgInfo xmlns:ns0='http://SampleTestPL.SchemaLocation'><LargeFilePath>" + largeFilePath + "</LargeFilePath></ns0:MsgInfo>";
                    byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(xmlInfo);
                    MemoryStream ms = new MemoryStream(byteArray);
                    pInMsg.BodyPart.Data = ms;
                }
                return pInMsg;
            }
    
            #endregion

    Here I want the xml to be dropped in to the File share Eg: E:\Dropbox\PL\send and and the entire message to be dropped in the folder Eg: E:\Dropbox\sendLarge. so in the ReceivePipeline properties i set like

    And in the send port the destination i give is E:\Dropbox\PL\send.

    The issue is both the xml and the message are getting dropped in to the same folder E:\Dropbox\PL\send and the message is not getting dropped in E:\Dropbox\SendLarge. Any help is greatly appreciated.

    Monday, March 16, 2015 2:26 PM

Answers

  • 1. I wouldn't because with the SFTP Adapter, .Length will A) throw a NotSupportedException B) force a read of the entire stream get the actual Length or C) not be correct (such as returning only the current buffer).

    There a slight possibility that the SFTP Adapter does some internal gyrations to learn the length but I would genuinely be surprised if so.  You can always test that, but that would be a tedious exercise.

    2. As for the file appearing in the same place, that's..well..you have something configured wrong. You just need to keep looking for wherever you have the same path set.

    You can also set a Breakpoint at like fs.Close() to check the value of _LargeFilePath. If you Stop the Send Port, does one or both still appear?

    • Marked as answer by vdha Tuesday, March 17, 2015 1:13 PM
    Monday, March 16, 2015 5:16 PM
    Moderator
  • Again for your latest issue, its not just the + "\\" +. But when you read the “ReceivedFileNamefrom the context, it will have value with FullPath+FileName. So for accesing the file name while formatting you shall use code as follows:

    string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
    string largeFilePath = _largeFileLocation + "\\" + System.IO.Path.GetFileName(srcFileName) + ".zip";
    
    

    YOu need to use System.IO.Path.GetFileName

    Regards,

    M.R.Ashwin Prabhu


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

    • Marked as answer by vdha Tuesday, March 17, 2015 1:13 PM
    Tuesday, March 17, 2015 12:13 PM

All replies

  • Hi, good afternoon.

    After reading your code, I think that everything is correct.

    The only place that something can be wrong is while reading/setting the "_largeFileLocation" Property. Maybe the property value is not setting properly and it has the value "E:\Dropbox\PL\send" by default.

    Can you share the full component code please?

    I'm so sorry if I cannot be much more helpful.

    Many thanks.

    Monday, March 16, 2015 3:31 PM
  • using System;
    using System.Collections.Generic;
    using System.Text;
    using Microsoft.BizTalk.Message.Interop;
    using Microsoft.BizTalk.Component.Interop;
    using System.IO;
    
    namespace Sample.ReceivePipelineLargeFile
    {
        [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
        [ComponentCategory(CategoryTypes.CATID_Decoder)]
        [System.Runtime.InteropServices.Guid("53fd04d5-8337-42c2-99eb-32ac96d1105a")]
        public class ReceivePipelineLargeFile : IBaseComponent,
                                                IComponentUI,
                                                IComponent,
                                                IPersistPropertyBag
        { 
            #region IBaseComponent Members
            public string Description
            {
                get
                { 
                    return "Pipeline component used to receive large file and save it ina disk";
                }
            }
            public string Name
            {
                get 
                { 
                    return "ReceivePipelineLargeFile";
                }
            }
            public string Version
            {
                get
                { return "1.0.0.0";
                }
            }
            #endregion
    
            #region IComponentUI Members
            public IntPtr Icon
            {
                get 
                {
                    return new System.IntPtr();
                }
            }
            public System.Collections.IEnumerator Validate(object projectSystem)
            {
                return null;
            }
            #endregion
    
            #region IPersistPropertyBag Members
            private string _largeFileLocation;
            private int _thresholdSize;
            public string LargeFileLocation
            {
                get { return _largeFileLocation; }
                set { _largeFileLocation = value; }
            }
            public int ThresholdSize
            {
                get { return _thresholdSize; }
                set { _thresholdSize = value; }
            }
            public void GetClassID(out Guid classID)
            {
                classID = new Guid("B261C9C2-4143-42A7-95E2-0B5C0D1F9228");
            }
            public void InitNew()
            {
            }
            public void Load(IPropertyBag propertyBag, int errorLog)
            {
                object val1 = null;
                object val2 = null;
                try
                {
                    propertyBag.Read("LargeFileLocation", out val1, 0);
                    propertyBag.Read("ThresholdSize", out val2, 0);
                }
                catch (ArgumentException)
                {
                }
                catch (Exception ex)
                {
                    throw new ApplicationException("Error reading PropertyBag: " + ex.Message);
                }
                if (val1 != null)
                    _largeFileLocation = (string)val1;
    
                if (val2 != null)
                    _thresholdSize = (int)val2;
    
            }
            public void Save(IPropertyBag propertyBag, bool clearDirty, bool saveAllProperties)
            {
                object val1 = (object)_largeFileLocation;
                propertyBag.Write("LargeFileLocation", ref val1);
    
                object val2 = (object)_thresholdSize;
                propertyBag.Write("ThresholdSize", ref val2);
            }
            #endregion
    
            #region IComponent Members
            public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
            {
                if (_largeFileLocation == null || _largeFileLocation.Length == 0)
                    _largeFileLocation = Path.GetTempPath();
    
                if (_thresholdSize == null || _thresholdSize == 0)
                    _thresholdSize = 4096;
    
                if (pInMsg.BodyPart.GetOriginalDataStream().Length > _thresholdSize)
                {
                    Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
                    string largeFilePath = _largeFileLocation + "\\" + pInMsg.MessageID.ToString() + ".zip";
    
                    FileStream fs = new FileStream(largeFilePath, FileMode.Create);
    
                    // Write message to disk
                    byte[] buffer = new byte[1];
                    int bytesRead = originalStream.Read(buffer, 0, buffer.Length);
                    while (bytesRead != 0)
                    {
                        fs.Flush();
                        fs.Write(buffer, 0, buffer.Length);
                        bytesRead = originalStream.Read(buffer, 0, buffer.Length);
                    }
                    fs.Flush();
                    fs.Close();
    
                    // Create a small xml file
                    string xmlInfo = "<ns0:MsgInfo xmlns:ns0='http://SampleTestPL.SchemaLocation'><LargeFilePath>" + largeFilePath + "</LargeFilePath></ns0:MsgInfo>";
                    byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(xmlInfo);
                    MemoryStream ms = new MemoryStream(byteArray);
                    pInMsg.BodyPart.Data = ms;
                }
                return pInMsg;
            }
    
            #endregion
        }
    }
    
    Thanks Osman Hawari, for trying to help me out.
    Monday, March 16, 2015 3:34 PM
  • Hi again,

    I'll try copying your component and test it in my local environment to reproduce your problem.

    Have you tried testing your pipeline with visual studio unit testing feature?

    Regards.

    Monday, March 16, 2015 3:46 PM
  • Sorry, I didnt test using unit test feature. How can I do that??

    Monday, March 16, 2015 3:47 PM
  • Please, take a look at the following article. It explains how to do unit testing in BizTalk pipelines.

    https://msdn.microsoft.com/en-us/library/dd792682.aspx

    Remember that you have to set the Enable Unit Testing flag to True in the Deployment page in the BizTalk Project containing the pipeline to test.

    Regards.

    Monday, March 16, 2015 4:11 PM
  • This must be a follow on to your previous thread.

    My advice: Don't test for the message size, always stream to disk and publish the Metadata.

    Why?  Because testing forces a read of the entire stream just to test the length and reading the very large stream is what you're trying to avoid.  Also, .Length is not always supported or unreliable if the actual length cannot be determined such as a NetworkStream.

    Looping on 1 byte is something you don't want to do.  Since I presume this is BizTalk 2013, you can use the Stream.CopyTo() method copy the incoming data stream to the FileStream.

    originalStream.CopyTo(fs);
    fs.Flush();
    fs.Close();

    Finally, you need to reset the Position on the new metadata message MemoryStream before returning the message.

    ms.Position = 0;

    Monday, March 16, 2015 4:15 PM
    Moderator
  • Hi Vdha,

    what you're trying to do is already available online why don't just use it or customize it to fit yours (use it as reference may be).

    Receice pipeline with code:http://www.codeproject.com/Articles/180216/Transfer-Large-Files-using-BizTalk-Receive-Side

    Send Pipeline with code:http://www.codeproject.com/Articles/180333/Transfer-Large-Files-using-BizTalk-Send-Side

    Regards,

    M.R.Ashwin Prabhu


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

    Monday, March 16, 2015 4:22 PM
  • Yes Ashwin I am trying to implement the Receive Side in codeproject, but I have the issue of files getting dropped in the same folder though I specify two different location.
    Monday, March 16, 2015 4:26 PM
  • Have you referred to the codeproject article's code for this requirement?



    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

    Monday, March 16, 2015 4:27 PM
  • Yes!!
    Monday, March 16, 2015 4:28 PM
  • It's almost the exact code from CP article but it still has problems so you shouldn't use it as-is.

    This sample assumes the File Adapter which uses an underlying FileStream which has special knowledge of the Length since it can get it from NTFS.

    Are you absolutely the value for LargeFileLocation and the Send Port path aren't the same?  That's the only way both would appear in the same place.

    Monday, March 16, 2015 4:37 PM
    Moderator
  • I am just trying to work with File Adapter now to check how it works, but I have to use this in SFTP adapter for my requirement. So, can't I use this with SFTP adapter too??

    Yes I am giving the LargeFileLocation and Send Port as different because I dont want the .xml to be sent with the LargeFile. But here both the files are sent to the Send Port.

    Sorry Johns, if I am missing something or not understanding the basics.

    Monday, March 16, 2015 4:44 PM
  • 1. I wouldn't because with the SFTP Adapter, .Length will A) throw a NotSupportedException B) force a read of the entire stream get the actual Length or C) not be correct (such as returning only the current buffer).

    There a slight possibility that the SFTP Adapter does some internal gyrations to learn the length but I would genuinely be surprised if so.  You can always test that, but that would be a tedious exercise.

    2. As for the file appearing in the same place, that's..well..you have something configured wrong. You just need to keep looking for wherever you have the same path set.

    You can also set a Breakpoint at like fs.Close() to check the value of _LargeFilePath. If you Stop the Send Port, does one or both still appear?

    • Marked as answer by vdha Tuesday, March 17, 2015 1:13 PM
    Monday, March 16, 2015 5:16 PM
    Moderator
  • Thanks Johns. I found the issue for the files being dropped in to the same location.I thought that I can give the LargeFileLocation while adding the component in to pipeline, on properties window

    But I had to explicitly specify the _largeFileLocation under IPersistPropertyBag like,

        if (val1 != null)
                    _largeFileLocation = (string)val1;
                else
                    _largeFileLocation = "E:\\Dropbox\\sendLarge\\";
    Now I cannot change the MessageID to SourceFileName

    string largeFilePath = _largeFileLocation + "\\" + pInMsg.MessageID.ToString() + ".zip";

    tried to use

    pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
    shows me error!!!!!!!!


    • Edited by vdha Monday, March 16, 2015 7:22 PM
    Monday, March 16, 2015 7:18 PM
  • You need to post the actual error message.
    Monday, March 16, 2015 7:25 PM
    Moderator
  • string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
    string largeFilePath = _largeFileLocation + srcFileName + ".zip";
    
    I used the above code and I am getting the error as "Reason: The given path's format is not supported."
    Monday, March 16, 2015 7:40 PM
  • Well, that has to be because you aren't constructing the full path properly.  Most likely, you're mixing escaped and unescaped '\' characters somewhere.

    Set a breakpoint and check the value of largeFilePath.

    Monday, March 16, 2015 7:58 PM
    Moderator
  • You are getting the error message because i believe there is extra "\\" in the path.

    Try to debug the code and see the final name constructed or try writing to the eventlog.

     _largeFileLocation = "E:\\Dropbox\\sendLarge\\"; 

     string largeFilePath = _largeFileLocation +"\\" + pInMsg.MessageID.ToString() + ".zip";

    If you see there are "\\" twice. So remove it from at least one place.


    Thanks,
    Prashant
    ----------------------------------------
    Please mark this post accordingly if it answers your query or is helpful.

    Tuesday, March 17, 2015 9:22 AM
  • I have tried your requirement.

    The code from codeproject works like a charm. No issues.

    I have compared the code you have shown here in the MSDN, I can see only difference is the archive file extension has been changed from *.msg to *. Zip (FYI, just changing the file name to .zip will not compress the file, but will simply rename the file extension). So no issue even with your code.

    If “_largeFileLocation” is not set, then the file will be archived to the user’s local temp drive. For me: C:\Users\<<YourUseProfileName>>\AppData\Local\Temp. So again it will simply createboth the large file and metadata file in the same location.

    I believe your send port’s file is based on ReceivePortName. i.e.

    BTS.ReceivePortName == YourReceivePortNameWhereThisRecivePipelineIsConfigured


    Regards,

    M.R.Ashwin Prabhu


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

    Tuesday, March 17, 2015 12:11 PM
  • Again for your latest issue, its not just the + "\\" +. But when you read the “ReceivedFileNamefrom the context, it will have value with FullPath+FileName. So for accesing the file name while formatting you shall use code as follows:

    string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
    string largeFilePath = _largeFileLocation + "\\" + System.IO.Path.GetFileName(srcFileName) + ".zip";
    
    

    YOu need to use System.IO.Path.GetFileName

    Regards,

    M.R.Ashwin Prabhu


    If this answers your question please mark it accordingly. If this post is helpful, please vote as helpful by clicking the upward arrow mark next to my reply.

    • Marked as answer by vdha Tuesday, March 17, 2015 1:13 PM
    Tuesday, March 17, 2015 12:13 PM
  • Thanks Ashwin, I have used ".zip" because the file I receive in the receive port is ".zip" format.
    Tuesday, March 17, 2015 12:25 PM