none
BizTalk FTP Adapter Wired Behavior when one of subscriber is not reachable in send port group.

    Question

  •  

    Hi Friends,

     

      I would like to share my wired experience with BizTalk 2006 R2 ftp adapter, from last three days I have been working around and going thru all known issues addressed with the ftp adapter, did change many settings but no use.

    Scenario and Configuration:

    Before I go in detail about the issue i would like to explain the scenario and what i would like to achieve. One of client has a scenario where we have one ftp receive location to publish the files to BizTalk 2006 R2 and the same subscribed by three ftp remote locations. Since due to live enterprise constraints we could not able to test the same scenario on live environments and the same replicated on test servers where we have four servers to cater the scenario. We have a setup of 1 receive location and 3 send locations on local area network (LAN) connected ftp servers.
    it is pure (CBR) content based routing based on receive port the messages are sent to 3 ftp locations using 3 send ports and send port group filter, fairly no orchestrations and dynamic resolution of endpoint (DREP) patterns some thing like involved, in simple words its not a complex integration at all in fact very simple . Just used out of box features with out any development efforts, and used default pipeline to pass thru.

    Problem Area:

    once we began testing we dropped few files(less then 100) in receive location and the message delivery worked absolutely fine without any issue .after all we have come across a test scenario where one of my clients location network is very weak and it breaks few times in a day due to the infrastructure issues at remote location, to replicate the issue and test the same we unplugged the network cable to one of destination ftp server within the LAN and dropped 1000 files, the results are very wired BizTalk 2006 R2 delivered more than 600 messages to two active connections and rest of all messages dehydrated and stopped processing. Initially i felt like its performance issue but BizTalk 2006 R2 box we have installed on dell power edge 2950 with 16 GB ram, when i looked in to the system monitor statistics server not even taking 3% processor time 2 GB of memory consumption.

    But i was really surprised when i connected the unplugged server back on to network after few seconds all dehydrated messages are delivered to destination properly (400 messages on each active ftp locations and 1000 to unplugged weak location), the hectic factor is how come our active ports are dependent on the disconnected port? As per pub sub BizTalk basic architecture the rest of 400 messages delivered with out any dependency. am not sure and it is very hard to figure it out and convince the client be cause as per client in real scenario network connection do not turn up for couple of days on some occasions.

    What is the cause? Any one of you guys come across this sort of situation?

    Somehow i am suspecting the network issues!! i would like to carry out netcap.exe diagnosis and few other network, ftp  server setting related tests on all servers .. Also would be doing adapter logging ...please do let me know if you think any diagnosis techniques to figure it out the issue?

    If I get this working,,, surely will let you know the cause. I really appreciate any help and your views on above.

    Thanks,

    Rajesh Gorla. 

    Friday, January 16, 2009 6:08 AM

All replies

  • try to test by removing sendport group filter, if u want to send the same message to 3 different locations, add filters to 3 send ports and subscribe.
    Friday, January 16, 2009 10:18 AM
  • Hi,
    I think you should not use Send port group with a filter, just 3 send Ports with the correct filter.
    Then only the messages that are meant to be sent through the not reachable FTP will get dehydrated.

    You mention also
    "as per client in real scenario network connection do not turn up for couple of days on some occasions."
    If you know one of your destination FTP server isn't going to be reachable for days , I strongly suggest you carefully set the retry count & retry interval on you send port. I would also set "Enable Routing for failed message" on to be able to catch the messages that still would not be sent in case the retry count is exhausted.
    Friday, January 16, 2009 10:27 AM
  • Hi Kiran,

       i did try no use.taken the filter out from group and kept it on each individual port same behavior. no luck!!

    Thanks,

    Rajesh Gorla

     
    Sunday, January 18, 2009 8:33 PM
  • have you tested sending only to 2 ports without configuring the third port?
    Monday, January 19, 2009 11:13 AM
  • Kiran,


        over the weekend i have tested various scenarios in fact  its looks like a real problem, i have replicated the same bug on almost four new environments, very soon updating more info on the same, also logged a support call from microsoft.
     
    -Rajesh Gorla
    Monday, January 19, 2009 4:42 PM
  • Further to my discussion I would like you to update few more points.

    I have carried out few more tests and different scenarios on FTP receive and Send Handlers.

    1.       I would be able to replicate the same issue on three different servers and a brand new installation of BizTalk server. So I conclude myself and I strongly consider it’s not an environmental issue.

     

    2.       I have captured the per protocol statistics on BizTalk box using netstat –s option, when executing the scenario under working , dehydrated and rehydrated taken place at various stages, the netstat results are as follows. Please do let me know if you feel any info is useful to track and debug.

    ---------------------------------------------------------------------------------------------------------------------------
    When all ports are connected and send ports are active no network issue, the all ftp servers are connected
    ---------------------------------------------------------------------------------------------------------------------------

    IPv4 Statistics

      Packets Received                   = 1714848
      Received Header Errors             = 0
      Received Address Errors            = 10822
      Datagrams Forwarded                = 0
      Unknown Protocols Received         = 0
      Received Packets Discarded         = 228
      Received Packets Delivered         = 1703798
      Output Requests                    = 1684343
      Routing Discards                   = 0
      Discarded Output Packets           = 20
      Output Packet No Route             = 0
      Reassembly Required                = 0
      Reassembly Successful              = 0
      Reassembly Failures                = 0
      Datagrams Successfully Fragmented  = 0
      Datagrams Failing Fragmentation    = 0
      Fragments Created                  = 0

    ICMPv4 Statistics

                                Received    Sent
      Messages                  61          105      
      Errors                    0           0        
      Destination Unreachable   5           53       
      Time Exceeded             0           0        
      Parameter Problems        0           0        
      Source Quenches           0           0        
      Redirects                 4           0        
      Echos                     0           52       
      Echo Replies              52          0        
      Timestamps                0           0        
      Timestamp Replies         0           0        
      Address Masks             0           0        
      Address Mask Replies      0           0        

    TCP Statistics for IPv4

      Active Opens                        = 2856
      Passive Opens                       = 23201

      Failed Connection Attempts          = 67
      Reset Connections                   = 59
      Current Connections                 = 81
      Segments Received                   = 1679622
      Segments Sent                       = 1661115
      Segments Retransmitted              = 2121

    UDP Statistics for IPv4

      Datagrams Received    = 24062
      No Ports              = 167
      Receive Errors        = 0
      Datagrams Sent        = 20999


    ---------------------------------------------------------------------------------------------------------------------------
    When one of the ftp servers is pulled off from network.
    ---------------------------------------------------------------------------------------------------------------------------


    IPv4 Statistics

      Packets Received                   = 2507948
      Received Header Errors             = 0
      Received Address Errors            = 15919
      Datagrams Forwarded                = 0
      Unknown Protocols Received         = 0
      Received Packets Discarded         = 228
      Received Packets Delivered         = 2491801
      Output Requests                    = 2455575
      Routing Discards                   = 0
      Discarded Output Packets           = 20
      Output Packet No Route             = 0
      Reassembly Required                = 0
      Reassembly Successful              = 0
      Reassembly Failures                = 0
      Datagrams Successfully Fragmented  = 0
      Datagrams Failing Fragmentation    = 0
      Fragments Created                  = 0

    ICMPv4 Statistics

                                Received    Sent
      Messages                  91          139      
      Errors                    0           0        
      Destination Unreachable   5           57       
      Time Exceeded             0           0        
      Parameter Problems        0           0        
      Source Quenches           0           0        
      Redirects                 4           0        
      Echos                     1           81       
      Echo Replies              81          1        
      Timestamps                0           0        
      Timestamp Replies         0           0        
      Address Masks             0           0        
      Address Mask Replies      0           0        

    TCP Statistics for IPv4

      Active Opens                        = 4121
      Passive Opens                       = 30064

      Failed Connection Attempts          = 166
      Reset Connections                   = 80
      Current Connections                 = 30
      Segments Received                   = 2456255
      Segments Sent                       = 2421351
      Segments Retransmitted              = 3264

    UDP Statistics for IPv4

      Datagrams Received    = 35398
      No Ports              = 204
      Receive Errors        = 0
      Datagrams Sent        = 30817


    ---------------------------------------------------------------------------------------------------------------------------
    When the disconnected ftp server connected back onto network
    --------------------------------------------------------------------------------------------------------------------------


    IPv4 Statistics

      Packets Received                   = 3581973
      Received Header Errors             = 0
      Received Address Errors            = 21777
      Datagrams Forwarded                = 0
      Unknown Protocols Received         = 0
      Received Packets Discarded         = 228
      Received Packets Delivered         = 3559974
      Output Requests                    = 3513948
      Routing Discards                   = 0
      Discarded Output Packets           = 20
      Output Packet No Route             = 0
      Reassembly Required                = 0
      Reassembly Successful              = 0
      Reassembly Failures                = 0
      Datagrams Successfully Fragmented  = 0
      Datagrams Failing Fragmentation    = 0
      Fragments Created                  = 0

    ICMPv4 Statistics

                                Received    Sent
      Messages                  118         163      
      Errors                    0           0        
      Destination Unreachable   9           58       
      Time Exceeded             0           0        
      Parameter Problems        0           0        
      Source Quenches           0           0        
      Redirects                 4           0        
      Echos                     1           104      
      Echo Replies              104         1        
      Timestamps                0           0        
      Timestamp Replies         0           0        
      Address Masks             0           0        
      Address Mask Replies      0           0        

    TCP Statistics for IPv4

      Active Opens                        = 6190
      Passive Opens                       = 50950
      Failed Connection Attempts          = 218
      Reset Connections                   = 85
      Current Connections                 = 92
      Segments Received                   = 3513307
      Segments Sent                       = 3469019
      Segments Retransmitted              = 4541

    UDP Statistics for IPv4

      Datagrams Received    = 46513
      No Ports              = 237
      Receive Errors        = 0
      Datagrams Sent        = 40224


    ----------------------------------------------------------------------------------------------------------------------------------------------------------

    3.
          
    Somehow on other scenario I gotrid of the issue by creating one more host instance. But after the execution it led so many questions.

     

    I have created a one more host instance and hosted the weak port to process and when I disconnected the server the results are quite appropriate and all message processing was successful on other 2 active ports.

     

    Are there any limitations on opening connections to destination on host instance?

    Does disconnected port really creates dependency on ports hosted on same host instance?

     

    My answers are YES. Here I went little further since I was under assumption to take off all weak ports on to one separate host instance but finally I can’t, I have added one independent sub pub integration point. The second send location is on “weak ports host instance” and nothing related first scenario It receives files from different receive location and sends to the different send location.

     

    Again the results are wired due the disconnected port all messages are stopped processing within that host instance and all send ports under that host instance is non functional. As explained as I am having two active ports hosted on one send handler host instance and the third port is hosted on second host send handler host instance along with the new integration point.

     

    The active send are processing still good since all send ports are active within that host instance  but due to inactive one on second instance new integration point also non functional.

     

    I have captured the logs at adapter level, end result nothing is written, I have noticed on logs adapter not even opening a connection to ftp servers to process messages. So certainly I do not want to create “1 to 1” send port to and host instance to host the same. I am just leaving this challenge to adapter development team really suspecting how the adapter manages out going connections on ftp adapters? How the inactive connections are managed?

                    So I can’t take a direction to change my architecture to move weak send ports on different host instance.

    4.       I have changed the configuration of FTP send port as per the few discussion went on various BizTalk sites.

    “QUIT after put” tried this setting, eventually noticed a change in log files adapter terminates the connection after sending a message successfully and noticed the same on FTP server log closing the connection on delivery of every successful message.

    Still no use same old story. But this is quite expensive for BizTalk Adapter to establishing connection and closing per each document delivery. Any how this setting is not help for my scenario and issue.

    Today I will be working on fine-tuning host advanced properties and Microsoft BizTalk Server 2006 R2 Management Pack, will update you any pro and cons on my later discussion notes.

    Tuesday, January 20, 2009 10:24 AM
  • The workaround you have seems to be the simplest way to work around this - put each send port in a separate host / host instance.
    Tuesday, January 20, 2009 6:27 PM
  • Hi Mustansir,

      i think creating one to one host instance is not appropriate since i have more then 1900 send ports. i do not want to use that way anyhow i have workaround to know the issues..its a try but not a proper solution to issue.

    Thanks,

    Rajesh Gorla.

     

     

    Tuesday, January 20, 2009 8:40 PM
  • Hi Rajesh,
     
    I think the problem lies in the Send Port Group.As you have all the send port under the same Send Port Group, so when any dehydrated message is subscribed by the send port(weak n/w) then it will be sent to all the send port of the send port group( http://msdn.microsoft.com/en-us/library/ms943913.aspx). So be careful with send port group. Instead you can have separate send ports with filters(don't create send port group) to subscribe the messages separately.

    And as far as host instances are concerned, its meant for load balancing and there should not have any problem or dependency with two send port running on different host instances.
    Ajeet Kumar
    Tuesday, April 14, 2009 8:43 AM
  • Sorry Folks,

    As I was busy n didn’t get a chance to update the developments on the issue, we have a workaround to resolve the problem but I am not really convinced with that hence just left it open.

    Ajeet,

    I hope you misunderstood the query; did you try to configure the scenario as I explained? If you do so you will come to know the issue. As you said send port and port groups are very clear to us and we have working scenarios over 100s.

    The same query has been escalated to Microsoft and they had suggested us to have a host instance per FTP server wise, practically the solution resolved the issue, the issue reported remains same. “BizTalk FTP adapter is not processing the messages within the host instance when one of the destination servers is down” really nothing to do with the filter on port groups or at port level,

    Resolution:

    We have categorized ports per ftp server and location wise therefore the rest of active destinations will receive messages on the event of non availability of any destination within the send port group.  

    Cheers,

    Rajesh Gorla.

    • Proposed as answer by BizBuilder Monday, June 21, 2010 2:04 PM
    Wednesday, May 06, 2009 7:02 AM