locked
Integrate BizTalk with Amazon MQ (AWS Messaging Broker) RRS feed

  • Question

  • Hi,

    We have a third party application with which BizTalk integrates by using MQSC adapter(part of HIS 2016) that sends/receives messages from their IBM WebSphere MQ queues.

    Now they are planning to use Amazon MQ(AWS Messaging Broker) as they are migrating their application to AWS. Is there any way that BizTalk can communicate with Amazon MQ? I tried searching for any documentation but can't find any pointers.

    Can you please share some information around this?

    Please note: Currently we are using BizTalk server 2016.

    Thank you.

    /Biranchi

    Tuesday, May 26, 2020 1:25 PM

All replies

  • Hi,

    It seems Amazon MQ is a message broker for Apache ActiveMQ.

    What protocol will you be using? OpenWire? STOMP? WebSocket?

    Br,

    Leo


    Did my post help? Please use "Mark as answer" or "Propose as answer". Thank you!

    Tuesday, May 26, 2020 2:43 PM
  • Hi Leo,

    Thanks  a lot for a quick reply.

    At this point I have very limited knowledge on which protocols will be used. But is there a way we can use BizTalk to communicate directly with Amazon MQ?

    /Biranchi

    Tuesday, May 26, 2020 2:59 PM
  • Hi Biranchi

    I would start of be reading up about it.  From their page Amazon MQ for Apache ActiveMQ

    ". Connecting your current applications to Amazon MQ is easy because it uses industry-standard APIs and protocols for messaging, including JMS, NMS, AMQP, STOMP, MQTT, and WebSocket"

    So with keywords "BizTalk stomp" I came up with this blog Integrating Apache Active Message Queue(AMQ) with BizTalk Server – Publishing messages Part 2 (extra linefeed)

    So yes, it is possible to do, but there are a lot of pitfalls and challenges as this article highlights.  As I recall we had a issue using STOMP with another MQ where rather than just doing a GET to get the message, we had to do a DELETE otherwise the messages were still accumulating on the queue even after we picked them up.


    Tuesday, May 26, 2020 10:01 PM
  • Hi Colin,

    Thanks a lot for the email. Looks like there are challenges. Looking at the blog you mentioned, seems HTTP adapter can be used to send messages to Amazon MQ and use Web-HTTP adapter to receive messages.

    Off-course this is a matter of POC and verification.

    Regards

    Biranchi


    Thursday, May 28, 2020 1:16 PM
  • Hi Colin,

    I have been following up your blogs and setup Active MQ locally and able to access the REST API to consume messages from a test Queue.

    Now from BizTalk, I can use the classic HTTP adapter to push messages to the Queue and use WCF-WebHTTP adapter(Static Solicit-Response send port) to receive messages from the Queue.

    Although it has worked like a charm using the REST API, just wanted to know is there any other way to use the other protocols from BizTalk?

    Just wanted to check different available options. Any information will really help.

    Regards

    Biranchi

    Thursday, June 11, 2020 12:44 PM
  • Begin by deploying some consumer applications on AWS. These consumers can be new applications or additional instances of applications already running on-premises. You configure these consumer applications to receive messages from Amazon MQ. At this stage, message producers are still on-premises sending messages to the IBM MQ broker.

    Next, bridge from IBM MQ to Amazon MQ using a proxy pattern. The proxy pattern is technology-agnostic, and you implement the pattern using Apache Camel to build a JMS bridge. Apache Camel is an open source integration framework for implementing Enterprise Integration Patterns. Apache Camel includes JMS components that easily connect with IBM MQ and Amazon MQ.

    In this step, you build an Apache Camel route to consume messages from IBM MQ, and forward to Amazon MQ. Here is an example from the camel-context.xml file, which defines the configuration:

    <route id="ibmMQ-to-amazonMQ">
    <description>Camel Route from IBM MQ to Amazon MQ</description>
    <from uri="ibmMQ:queue:DEV.QUEUE.2?concurrentConsumers=5"/>
    <inOnly uri="amazonMQ:queue:DEV.QUEUE.2?preserveMessageQos=true"/>
    </route>
    This Apache Camel route defines how messages from the producer applications connected to IBM MQ move to Amazon MQ. In this example, there is one sample route but you may have many routes in your production use-case.

    Apache Camel is deployed as a Docker container running on AWS Fargate as compute engine for Amazon Elastic Container Service (ECS). ECS is a container orchestration framework that manages the deployment of the containers, and runs them in a highly scalable manner.

    AWS Fargate eliminates the heavy lifting of scaling the underlying virtual machines for your ECS cluster. By defining the desired capacity of AWS Fargate tasks, it introduces self-healing capabilities to the JMS bridge. AWS Fargate tracks the number of healthy tasks, and creates new tasks automatically if an old one is no longer available.

    Now the JMS bridge and the on-premises consumers are listening on the same queue and waiting for messages. Messages sent to IBM MQ are consumed by on-premises consumers as well as the JMS bridge.  The JMS bridge forwards messages to Amazon MQ to be consumed by the applications already migrated to the cloud. You can now validate that messages are processed by applications on AWS and on-premises.

    Now phase 1 of the migration is validated. You can continue to move more consumers as you get more comfortable with the availability and scalability of the bridge solution. The goal of this phase is to reach a state where messages are still produced on-premises, with some consumers running on AWS.

    Several consumer applications are now migrated to AWS. The goal now is to move the producers to AWS. Start the migration by running a few producer applications on AWS and connect them to Amazon MQ. The Apache Camel bridge is also updated to facilitate bidirectional flow of messages.

    The following configuration code shows the route, which moves messages from Amazon MQ to IBM MQ:

    <route id="amazonMQ-to-ibmMQ">
    <description>Camel Route from Amazon MQ to IBM MQ</description>
    <from uri="amazonMQ:queue:DEV.QUEUE.2?concurrentConsumers=5"/>
    <inOnly uri="ibmMQ:queue:DEV.QUEUE.2?preserveMessageQos=true"/>
    </route>

    Step – 3
    To make this JMS bridge resilient, it must scale in and out automatically, based on your current load. You can configure this by using Amazon CloudWatch metrics and CloudWatch alarms. These alarms can trigger scaling activities to scale in or out, with a fixed number of instances or a percentage-based scaling.

    You can also scale out your AWS system based on the utilization of your on-premises broker by defining custom CloudWatch metrics. For example, by running a CRON script on the on-premises broker machine to periodically report the metrics such as queue depth.

    Thursday, June 11, 2020 1:26 PM
  • Hi,

    While trying to find more information around accessing AWS MQ from BizTalk, I came across this AMQPBizTalk adapter written by AbdelazizElbaz(Thanks a lot for sharing the source code).

    https://github.com/AbdelazizElbaz/AMQPBizTalkAdapter

    Just wanted to check, has someone used this Adapter that uses AMQP protocol to communicate with AWS MQ from BizTalk?

    Please shade some light here which would be really helpful.

    Thank you.

    Regards

    Biranchi

    Thursday, June 18, 2020 9:38 AM