Skipping empty flatfile messages

Question

Hi,

I've got (BTS2010): 

- ReceivePort \ ReceiveLocation pulling flatfile files from SFTP (PassThroughReceive).

- SendPort sending files to MSMQ queue, subscribing messages from above ReceivePort (PassThroughTransmit).

- ReceivePort \ ReceiveLocation taking files from above MSMQ queue and publishing them for orchestration (CustomFlatfilePipeline).

The overview is:

 SFTP   --->    MSMQ   ---> Orchestration

Periodically, on the SFTP server, the empty messages appear (0kb). Then we've got an error on the CustomFlatfilePipeline, because an empty file cannot be parsed (that's logical). 

My question is: how can I skip empty files (skip or route to some dummy place)? I cannot set the filter, because there is no biztalk property with the file size. There is no possibility to set the minimum file size on RL. My aim is: eat, skip, do whatever with empty messages, but do not generate parsing error. Is there any straightforward solution for that?

Regards,

Dawid

Answer 1

In case if FILE Adapter , it will work . in case of empty file , warning will come

The FILE receive adapter deleted the empty file "FileName" without performing any processing.

Answer 2

What I can think of is that have a custom pipeline where u check the file size before processing..

---

Regards
Ritu Raj
When you see answers and helpful posts,
please click Vote As Helpful, Propose As Answer, and/or Mark As Answer

Answer 3

I believe if a custom pipeline component returns a null from the Execute(...) method, no exception will occur... effectively "eating" the message.

If this is not the case, you can do it in the disassembly phase by writing your own flat file disassembler that uses  the BizTalk flat file disassembler to process the messages.  If the message happens to be empty, simply return null in your first GetNext(...) method call.  I recently published a sample flat file disassembler that uses the BizTalk flat file disassembler to parse and disassemble the flat files, but has additional processing as well.  With very minor changes (mostly removing code), this can be modified to accomplish your requirements.

You can browse or download my sample from: http://code.msdn.microsoft.com/BizTalk-Trailer-Preserving-fbe4a87a

Let me know if you have any questions.  My MSDN email is: [e-mail]

---

David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

Answer 4

I do think that you can create a simple dissassemble component and i the GetNext() menthod implementation just check the Length of the data stream attached to message and if it is 0 then return null otherwise return the message. You can use the code below to create it, test this code I have not tested it:

System.Collections.Queue qOutputMsgs= new System.Collections.Queue()

#region IDisassemblerComponent Members
public void Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
{   
    Stream originalStream = pInMsg.BodyPart.GetOriginalDataStream();

    if(originalStream.Length>0)
        qOutputMsgs.Enqueue(outMsg);
}

public IBaseMessage GetNext(IPipelineContext pContext)
{
    IBaseMessage msg=null;
    if (qOutputMsgs.Count > 0)
    {
         msg = (IBaseMessage)qOutputMsgs.Dequeue();        
    }
    
    return msg;
}
#endregion

---

Please mark the post answered your question as answer, and mark other helpful posts as helpful, it'll help other users who are visiting your thread for the similar problem, Regards -Rohit Sharma (http://rohitt-sharma.blogspot.com/)

Answer 5

You need to be careful with the "Length" property of some streams... many do not return the length or will thrown an exception if the length property is accessed.  An example of a stream without a length is the output stream from the flat file disassembler... this is because the XML output length is not known until after the stream has been read... the XML output is generated as the stream is being read, not before.

---

David Downing... If this answers your question, please Mark as the Answer. If this post is helpful, please vote as helpful.

Answer 6

In my opinion the fact that ZERO byte files are being picked up the sftp adapter is what would require probing. It would seem to me that the file is being picked up for transfer which has NOT been WRITTEN yet.

Is this SFTP location external to the organization? even if they are, let the files get SFTP'ed there. Write a windows servive which polls the drop location every once a while, picks the file and in case of a non-ZERO length file, posts it to the MSMQ. This way the ZERO length files will not get picked up (in case of a deffered write by the SFTP server they will show as non-ZERO files in a later pass and then get processed). The service could in case of a ZERO byte file send out an alert or something.

My 2 cents.

Regards