none
Custom Pipeline Error RRS feed

  • Question

  • I am new to BizTalk and I have the following problem. I have written my own custom C# pipeline disassembler component to strip a complex flat file down to the lines I want. This works and I have tested the custom component using pipeline.exe, a sample flat file to confirm it works. My setup is as follows:

    Custom C# Disassembler

    • Strips lines out of flat file
    • Puts message on queue once done

    Visual Studio Dev Environment
    • VS 2008
    • Orchestration with Receive Port with Message Type set to schema of flat file after the custom pipeline component has run ie with all the excess lines stripped out
    • Receive box with Message same set to Message based on same schema receive port message
    • Next is a transform step that  maps inbound to outbound using Map
    • Send step with Message set to outbound format
    • Send port with Message Type set to outbound schema format
    • Receive Pipeline component with my custom dll as the first step in Disassemble stage then a Flat file disassembler as the next step (tried with and without flat file Disassembler - made no difference)

    Server Setup
    • BizTalk server 2009
    • Using FILE receive and file send
    • Receive Location with the Receive Pipeline set to the pipeline with my custom C# component on it.
    • The Send Port has a mapping on it (the same as the mapping in the orchestration that doesnt seem to do anything)
    • Filter set up correctly
    Errors
    There was a failure executing the send pipeline: "JcTransformer.SendPipelineMsPbIntraDay, 
    JcTransformer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=35907271f8e51d66"
    Source: "Flat file assembler" Send Port: "SendPortMsPbFile"
    URI: "C:\Documents and Settings\adm-ken.stewart\BizTalk\out\%MessageID%.xml"
    Reason: Data at the root level is invalid. Line 1, position 1.
    I have tried setting the outbound port to PassThruTransmit and the file saved is indeed my flat file.

    My question is why is the file not being transformed to XML and what am I doing wrong?
    Monday, August 3, 2009 10:24 AM

Answers

  • Actually I finally got this to work using a custom Decode step and then passed the output to the Flat file disassembler in a Recei9ve Pipeline. So now I have complex file in -> custom file stripper -> Flat file disassembler -> XML. Works fine.

    Anyway Ajeet thanks for the advice - much appreciated
    Thursday, August 6, 2009 2:16 PM

All replies

  • First thing, If you use your pipeline component in Disassemble stage then you need to disassemble the Flat File inside your component only using FlatFile Disassembler class. This is required, as only one disassembler per message will be executed, so if the First Disassembler get the message then the rest of the disassembler will be ignored.
    Here you can read about custom disassembler http://www.codeproject.com/KB/biztalk/DebatchingFlatfile.aspx?display=Print

    Also Test the map in VS, and make sure its working without any error.

    Let me know if it helps.

    Ajeet Kumar
    Monday, August 3, 2009 10:49 AM
  • Hi Ajeet,

    The map works fine  - tested. Do you mean that I need to manually convert the flat file to the XML structure that my schema will recognise? Or do you mean that if I put the standard FlatFile disassembler (in the same pipeline as my custom disassembler) after my custom disassembler in the Disassemble stage that it will be ignored?

    The example is slightly different in that it splits a large file into lots of smaller records. I will take in a large file and output a large file (with some mappings).

    Ken

    Monday, August 3, 2009 1:28 PM
  • "do you mean that if I put the standard FlatFile disassembler (in the same pipeline as my custom disassembler) after my custom disassembler in the Disassemble stage that it will be ignored?" : Yes

    From the article take only the disassembling part (How to extend FlatFile disassembler) and put your logic along with it. Use only this component in the Disassembler stage.
    Ajeet Kumar
    Monday, August 3, 2009 1:40 PM
  • Many thanks but I have tried that already and it didnt work "Receive Pipeline component with my custom dll as the first step in Disassemble stage then a Flat file disassembler as the next step (tried with and without flat file Disassembler - made no difference)"

    Any other ideas?
    Monday, August 3, 2009 2:08 PM
  • Ok, do it in this way. Implement the disassembling logic inside your Pipeline component instead of using the FlatFile Disassembler seperatly and use this component in Disassembler Stage (Now don't use default FlatFile Disassembler). So now you have only one pipeline component in Disassemble stage i.e; your custom pipeline component which will do
    1) Disassemble the FlatFile message
    2) Your custom logic
    Ajeet Kumar
    Monday, August 3, 2009 2:15 PM
  • That is what I mean - I tried both ways and it didnt work. Now the standard Flat file disassembler is not in the pipeline any longer ie my receive pipeline only contains my custom dll in the Disassemble stage. It is obviously being called and the file is transformed to the cleansed format but no transformation is taking place to my expected XML output and I have no idea why. So the process is:

    - custom component gets called and cleanses file in receive pipeline
    - orchestration called
    - send pipeline cannot transform using map as the file is in flat file format not XML ie no mapping seems to have taken place but I have no idea why??
    Monday, August 3, 2009 3:24 PM
  • Can you share custom pipeline component code?
    I am sure something is getting wrong inside the component.
    Ajeet Kumar
    Tuesday, August 4, 2009 6:40 AM
  • Could it be that I need to manually transform the flat file structure to XML in my custom component? I think the hint you gave me regarding only the firstmatch in the Disassemble stage will be run. So my assumption that the Flat file disassembler would be run proved to be false and thats where the translation to XML was happening.

    Code is below:

    using System;
    using System.ComponentModel;
    using System.Collections;
    using System.Reflection;
    using System.Text;
    
    using Microsoft.BizTalk.Message.Interop;
    using Microsoft.BizTalk.Component.Interop;
    using Microsoft.BizTalk.ParsingEngine;
    using Microsoft.BizTalk.Component;
    using System.IO;
    
    namespace MyNamespace.CustomPipeline
    {
    	[ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    	[ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    	[System.Runtime.InteropServices.Guid("3EB8E4F0-AB89-4b11-AD70-C814491D465A")]
    	public class MyDisassembler : IBaseComponent, IPersistPropertyBag, IComponentUI, IDisassemblerComponent
    	{
    
    		#region Private members
    
    		System.Collections.Queue qOutputMsgs = new System.Collections.Queue();
    		IBaseMessagePart msgPart;
    		string textProperty = string.Empty;
    		#endregion
    
    		#region Constructors
    
    		public MyDisassembler()
    		{
    
    		}
    
    		#endregion
    
    		#region IBaseComponent Members
    		
    		/// <summary>
    		/// Description of the component.
    		/// </summary>
    		[Browsable(false)]
    		public string Description
    		{
    			get { return "MyDisassembler strips spurious lines"; }
    		}
    
    		public string Name
    		{
    			get { return "MyDisassembler Disassembler"; }
    		}
    
    		public string Version
    		{
    			get { return "1.0.0.0"; }
    		}
    
    		#endregion
    
    		#region IPersistPropertyBag Members
    
    		public void GetClassID(out Guid classID)
    		{
    			classID = new Guid("3EB8E4F0-AB89-4b11-AD70-C814491D465A");
    		}
    
    		public void InitNew()
    		{
    		}
    
    		public void Load(IPropertyBag propertyBag, int errorLog)
    		{
    			object val = ReadPropertyBag(propertyBag, "TextProperty");
    			if (val != null)
    			{
    				textProperty = (string)val;
    			}
    		}
    
    		public void Save(IPropertyBag propertyBag, bool clearDirty, bool saveAllProperties)
    		{
    			object val = textProperty;
    			propertyBag.Write("Namespace", ref val);
    		}
    
    		#endregion
    
    		#region IComponentUI Members
    
    		public IntPtr Icon
    		{
    			get { return System.IntPtr.Zero; }
    		}
    
    		public System.Collections.IEnumerator Validate(object projectSystem)
    		{
    			return null;
    		}
    
    		#endregion
    
    		#region IDisassemblerComponent Members
    
    		public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
    		{
    			System.Diagnostics.Trace.WriteLine("Pipeline Disassemble Stage Enter");
    			msgPart = pInMsg.BodyPart;
    			Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();
    
    			string bodyOut = "";
    
    			StreamReader streamReader = new StreamReader(originalStream);
    			while (!streamReader.EndOfStream)
    			{
    				string lineInMessage = streamReader.ReadLine();
    				string[] splitLine = lineInMessage.Split('\t','"');
    				if (splitLine.Length > 27)
    				{
    					if (splitLine[27] != "")
    					{
    						bodyOut += lineInMessage + Environment.NewLine;
    					}
    				}
    			}
    
    			streamReader = null;
    			
    			try
    			{
    				if (msgPart != null)
    				{
    					if (originalStream != null)
    					{
    						// do the disassembling
    						byte[] arrByte = ConvertToByteArray(bodyOut);
    						originalStream = (new MemoryStream(arrByte));
    					}
    				}
    			}
    			catch (Exception)
    			{
    				System.Diagnostics.Trace.WriteLine("Pipeline Disassemble Stage Exception");
    				throw;
    			}
    
    			originalStream.Seek(0, SeekOrigin.Begin);
    			msgPart.Data = originalStream;
    
    			IBaseMessage outMsg = pInMsg;
    			outMsg.BodyPart.Data = originalStream;
    
    			qOutputMsgs.Enqueue(outMsg);
    
    			System.Diagnostics.Trace.WriteLine("Pipeline Disassemble Stage Exit");
    		}
    
    		public IBaseMessage GetNext(IPipelineContext pContext)
    		{
    			if (qOutputMsgs.Count > 0)
    			{
    				IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();
    				return msg;
    			}
    			else
    				return null;
    		}
    
    		#endregion
    
    		#region Private Functions
    
    		private byte[] ConvertToByteArray(string strInput)
    		{
    			return Encoding.ASCII.GetBytes(strInput);
    		}
    
    		private string ConvertToString(Stream inStream)
    		{
    			byte[] buffer = new byte[inStream.Length];
    			inStream.Read(buffer, 0, (int)inStream.Length);
    
    			return Encoding.ASCII.GetString(buffer);
    		}
    		private object ReadPropertyBag(IPropertyBag propertyBag, string propName)
    		{
    			object val = null;
    			try
    			{
    				propertyBag.Read(propName, out val, 0);
    			}
    			catch (ArgumentException)
    			{
    				return val;
    			}
    			catch (Exception e)
    			{
    				throw new System.ApplicationException(e.Message);
    			}
    			return val;
    		}
    
    		#endregion
    
    	}
    }
    Tuesday, August 4, 2009 7:14 AM
  • You need to disassemble the flat file explicitly using FlatFile Disassembler class. Use the below code to disassemble inside your custom component

    base.DocumentSpecName = this.DocumentSpecName;

    base.Disassemble(pContext, pInMsg);

    DocumentSpecName is the schema name.

    Refer Step 4 of this link http://www.codeproject.com/KB/biztalk/DebatchingFlatfile.aspx?display=Print .

    Let me know if it helps.


    Ajeet Kumar
    Tuesday, August 4, 2009 8:11 AM
  • Ok I have implemented the code but can no longer run anything. The line call below throws an ArgumentNullException when attempting to set a property

    		void IPersistPropertyBag.Load(IPropertyBag propertyBag, int errorLog)
    		{
    			base.Load(propertyBag, errorLog);
    		}
     Exception

    Key cannot be null.Parameter name: key
     Any ideas?
    Tuesday, August 4, 2009 12:59 PM
  • Can you check if you are setting any value for this property (errorLog) in the Pipeline property page?
    Ajeet Kumar
    Tuesday, August 4, 2009 1:55 PM
  • Only the Document schema
    Tuesday, August 4, 2009 3:08 PM
  • The thing is I implemnted the code pretty much exactly as in the example. I am happy to post it if it helps. Any idea why I would get this error?
    Tuesday, August 4, 2009 8:17 PM
  • Oh...use this code
    public
    
    
    void
     Load(IPropertyBag propertyBag, int
     errorLog)
    {
    object val = ReadPropertyBag(propertyBag, "TextProperty" );
    if (val != null )
    {
    textProperty = (string )val;
    }
    }
    as you had earlier.
    errorLog is Address of an error log object in which the object stores any errors that occur during initialization, so I think you need to pass some value, not sure though.

    Ajeet Kumar
    • Edited by Ajeet.Kumar Wednesday, August 5, 2009 8:34 AM Modified the comment
    Wednesday, August 5, 2009 8:30 AM
  • Actually I finally got this to work using a custom Decode step and then passed the output to the Flat file disassembler in a Recei9ve Pipeline. So now I have complex file in -> custom file stripper -> Flat file disassembler -> XML. Works fine.

    Anyway Ajeet thanks for the advice - much appreciated
    Thursday, August 6, 2009 2:16 PM
  • Can you please explain the steps which you performed to make this work........I am struck with the same issue now.........?

     

    Thanks

    Mallesh B

    Monday, August 2, 2010 6:57 PM