Keeping files together RRS feed

  • Question

  • Hi

    I am working on a project where I have to translate Flat Files to EDIFACT. I get several flat files each day. Each flat file contains one or more Invoices with an invoic list at the end (All invoices have a reference inside to which invoice list they belong). The sender and the receiver of the documents are always the same.
    At the moment the "big" flat files are splitted into single files (one file for each invoic and invoice list). The single files are translated into edifact and batched together at the end that I get one complete EDIFACT file with all Invoices and the Invoice list in it. This works as long as I get only one flat file and the batching process has ended before the next flat file arrives
    If I get more than one big flat file than all the files are mixed up during the batching process. The only information which invoice belongs to which invoic list is the number of the invoic list that is inside every invoice and in the invoice list itself.
    What do I have to do that I get for each big flat file an edifact file with the same Invoices and Invoice list  inside.

    Kind regards
    Klaus Weber
    Thursday, April 9, 2009 11:41 AM

All replies

  • Can you put the filename in to differentiate which file it came in during the batching process?

    Eric Stott [] - Mark as Answer if this reply does.
    Thursday, April 9, 2009 3:25 PM
  • Hi eric

    I thought about that. But how does the batching process knows which edifact files belong together. Perhaps my batching architecture is wrong.
    First I am translating the single files into edifact which are drropped into a directory. from this directory a receive port with a EdiReceivePipeline takes the files an the batching process is initialized. The trigger for the batch release is  time scheduled (every day at 18:00 ) or, at the moment, manually released.

    The following question arise to me now:

    1. how can I use the filename for that
    2. is my batching architecture correct

    Thanks for your help
    Klaus Weber
    Thursday, April 9, 2009 3:44 PM
  • "dropped into a file" - hurts my eyes to see that

    Can I assume you are using BTS 2006 R2?

    If so I would recommend using the batching process that comes installed in the EDI Application.

    I wrote a little of some things you can do to 'enhance' batching in EDI (either X12 or EDIFACT).
    Eric Stott [] - Mark as Answer if this reply does.
    • Proposed as answer by Leonid Ganeline Monday, April 13, 2009 5:06 AM
    • Unproposed as answer by KWeber Wednesday, April 15, 2009 2:03 PM
    Thursday, April 9, 2009 3:49 PM
  • Hi Eric

    I know that my configuration is not only hurtinig your eyes :)

    I am using 2006 R2 and I am also using the batching processes of BizTalk. 
    My Problem is how I can tell the batching process to release all the messages that belong to a certain reference. This reference differs from flatfile to flatfile. The easiest way would be translate the big flatfile into edifact without debatching first. But as far as I understood biztalk this is not possible.

    I read your blog but I did not find the clue how I could tell the biztalk server to keep the files together according to a defined reference. I am looking for the missing link how I can tell the batching process to release the batch for all messages with the referencenumber "xyz" inside.


    Klaus Weber
    Wednesday, April 15, 2009 2:03 PM
  • My solution initiated by a stored proc call that I was able to organize the results into a 'batch' of orders followed by the next 'batch' so it would throw it to batch it up, release the first batch, release the first batch, name it, and send it out. After that send the next batch of transactions to the batching process and then when the next batch is encountered, it releases that batch and names it and so on and so on.

    Are you saying that you don't have the ability to create a convoy orchestration?
    Eric Stott [] - Mark as Answer if this reply does.
    Thursday, April 16, 2009 5:13 PM
  • Hi Eric

    Of course I'm able to create a convoy. Perhaps I didn't explain my problem correctly.
    I get a bunch of flatfiles at the same time. Every file contains several invoices with an invoic list. These invoices and the invoicelist can be referenced to each other by a certain reference. The reference for the invoices is in a record named REFABO and the reference in the invoice list is in a record named BGM (the flat file has a similar structure like EDIFACT). All the files should be translated to EDIFACT, batched and send to the same partner. All files with the same reference number inside should be batched.
    I The difference to your solution is, that I cannot control how many flatfiles I get and how many invoices are inside these flatfiles.
    Creating a convoy correlated according to the reference number is not the problem. The problem is how to tell the batching orchestration that all files of a certain reference number are processed an now the batch for this reference number can be released.

    Klaus Weber
    Friday, April 17, 2009 1:46 PM
  • You can do it by promoting a reference for messages belonging to same group and use this promoted context property in Batch filter.
      Batch filter is powerful way to batch together the messages which have same context properties.

    Bhola Meena []
    Tuesday, April 21, 2009 4:54 PM
  • I know that the promoted context properties can be used for batching. But what do you do if the reference number differs from file to file? I always need a fix value for the batch filter to release the batch.

    Is there a way to change the batch filter value during runtime?

    Klaus Weber
    Wednesday, April 22, 2009 8:06 AM
  • Hi there

    Since a long time I was not working on this issue but now its time to finish this problem. I went through Erics Outbound batching example and it is working perfectly. Eric, thank you very much for sharing your ideas with others.
    Now I have still one unsolved problem. Erics Example works perfectly for one batch per party. How can I modifiy this solution to achieve a batchrelease for each incoming batch.

    I.e. if there are 2 incoming files (let's say one with 4 invoices and the other with 6 invoices) the solution creates one batch with 10 invoices. What I need is 2 files with 4 respectively 6 invoices inside. A batch filter cannot be used because of the varying reference numbers. The reference number is always the number of the invoice list to which the invoiecs belong to. A different party cannot be used either because all invoices belong to one party. Functional grouping (UNG-UNE) is not allowed by the receiver of the messages.

    Thanks for the input

    Klaus Weber
    Monday, July 27, 2009 12:01 PM
  • Well the best option I can see here is to use Correlation on Reference Number. As you are saying you want to batch the files having same Reference Number. And along with this make the Orchestration Singleton (Sequential Convoy). Now promote the Reference Number as promoted property. So in that case BizTalk will create one Orchestration Instance for one Reference Number which will process all the XMLs having same Reference Number. Say you received 3 FlatFile with different reference Number, the BziTalk will create 3 Orchestration instances which are independent of each other.

    With Sequential Convoy, there can be some performance issue (depending upon the FlatFile size). Try to use Attribute inside the Schema instead of Element. this will significanlty reduce the File size and improve the performance.
    Ajeet Kumar
    Monday, July 27, 2009 2:05 PM
  • Hi

    I agree with the sequential convoy. The problem is that at the end there is a batching mechanism and all flatfiles belong to the same party. When the first instance ends with the batch release, all files that are inside the batchingpool are released and batched because all are stamped for this distinct party
    In case there are 3 flatfiles in the receiveport at the same time, only one file is in the outfile directory after processing.

    I think what I need is an idea how to process one file after the other.
    Klaus Weber
    Tuesday, August 4, 2009 2:37 PM
  • "When the first instance ends with the batch release, all files that are inside the batchingpool are released and batched because all are stamped for this distinct party" : Can you explain this process in detail.

    If you have 3 files then as per my post it should create 3 Orchestration (also depend upon Reference Number) and in the out you should get multiple files (>=3).

    Ajeet Kumar
    Tuesday, August 4, 2009 3:06 PM
  • I have three files. each of them has multiple invoices inside. the only difference between the three files is the reference number.
    Because of the later translation to EDIFACT I have to debatch each file first and to re-batch it after the translation. the party number for all three files is the same but it is not allowed to batch all three files together. After the process there have to be three EDIFACT files in the output folder.
    At the moment my orchestration (with sequential convoy-correlated to the reference number) batches all three files together.

    Where is my mistake?

    Klaus Weber
    Tuesday, August 4, 2009 3:35 PM
  • What is the correlation property you have used for your re-batch Orchestration? I believe you have to use combination of Party number and Reference nos, which can filter the messages in three parts.
    Once you are able to do this the re-batch Orchestration will create three Orchestration instance and hence you will get 3 files in output.
    Ajeet Kumar
    Wednesday, August 5, 2009 8:04 AM
  • I can see only one solution for this problem.
    You need to calculate the messages inside batch before or while debatching. You do not have enough data to make batching process without this counter.
    BTW As I can remember all interchanges has the groups and document counters. You create the uppermost correlation/grouping level under interchanges, right?
    I suggest you to double-check this business requirement. Here the file itself works as a correlator for interchanges inside it, right? NOt a good decision. There should be something else to correlate interchanges.
    Leonid Ganeline [BizTalk MVP] My BizTalk blog
    Friday, August 7, 2009 4:12 AM
  • Hi

    My biggest Problem is to control the batch release mechanism. at the moment all files are processed at same time. That means all files that are ready for batch release. The first instance that comes to the batch release mechanism releases the whole batch with all files in it. I don't know how to tell the batch mechanism to release the batch for files and not for the others that belong to another sourcefile.
    Actually biztalk should process one file after the other. When the first one has been batched the process for the next one can be startetd.
    The orhcestration works fine if only one file is processed or the time span between the incoming files is long enough.

    Klaus Weber
    Friday, August 7, 2009 12:34 PM
  • Actualy you can use the singleton orchestration to make a "pseudo sequental" behavior.
    Pick up the input batch and delay till the output batch is created,
    then pick up next input batch...
    Use the C# code to check creating of the output batch, for example.

    Leonid Ganeline [BizTalk MVP] My BizTalk blog
    Friday, August 7, 2009 9:20 PM