iPaaS is a key component of what Gartner calls a hybrid integration approach1, and as someone accustomed to creating ESB solutions, it’s been eye-opening to see how quickly and easily you can get them up and running. But as I create iPaaS solutions, one thought keeps running through my mind: this would be so much more powerful and elegant if we added an event broker to the mix.
The aim of iPaaS is to simplify integration by creating a managed experience, particularly when deployed in the cloud. iPaaS typically includes:
Although there are adapters for many different information providers such as databases and EDI, the bread and butter of most iPaaSes (and most ESBs) has been synchronous REST calls between software as a service (SaaS) providers like Salesforce and SAP S/4HANA Cloud. As a result, iPaaS providers have focused on making synchronous API integrations as easy as possible.
Architecturally, this generally means the same process that receives an incoming request maps the data using a graphical tool, connects to the information consumer, and sends the transformed message.
Figure 1: Typical iPaaS process flow
Adding an Event Broker to Your iPaaS Solution
Adding an event broker allows you to separate (decouple) that process, such that the iPaaS receives information via one process and writes it to a queue or topic2, then a second process picks up the message and performs additional logic, typically including data transformation, before forwarding it to a consumer.
Figure 2: iPaaS flow with event broker
That decoupling can enhance traditional ESBs, but in some ways it’s even more important for an iPaaS. The architectural shift to decoupling using an event broker can improve an iPaaS’s resilience, agility and real-time capabilities — all without complicating its architecture or operation. The approach is a lot like breaking up a monolithic application into multiple microservices.
Some brief asides:
Boomi has partnered with Solace to extend the value of the industry leading Boomi Platform with enterprise grade event streaming and event management capabilities. Learn more here.
One of the most complex and time-consuming tasks for any integration architect is making sure a system performs when things go wrong – in fact some experts say up to 25 percent of an integration project should be devoted to designing exception handling, and iPaaS solutions are no different. That isn’t surprising given that architects need to consider complex questions including:
Introducing an event broker as part of a decoupled architecture, on the other hand, can answer some of the questions while keeping the iPaaS simple.
As shown in Figure 3, by persisting messages to a queue or topic after arrival at the iPaaS, the event broker takes responsibility for guaranteeing that the message will not be lost in the event of a failure that brings down either your process or the entire iPaaS. Without an event broker, an iPaaS either requires custom persistence logic or accepts that messages will be lost in the event of failure, requiring the information provider to resend as necessary.
Figure 3: Event brokers allow for message persistence in the case of failure
Once the message is persisted by an event broker, the iPaaS receiver (not the end consumer) can acknowledge that the message has been successfully processed and becomes available for the next incoming message.
This ability for the iPaaS receiver to quickly and independently process messages comes in handy if the downstream information consumer isn’t available. Typically, an information consumer being unavailable results in multiple attempts to retry the transaction until it succeeds.
As shown in Figure 4, if the receiver and sender are on the same thread, this can quickly cause the iPaaS to run out of resources, creating a chain reaction that blocks incoming transactions. By buffering incoming transactions, an event broker helps to keep your solution up and running. This buffering is even more valuable as we add in additional consumers, which we discuss in the next section.
Figure 4: Event brokers conserve resources during retry
Lastly, if the error turns out to be one that will take a while to resolve (e.g. someone to add a value to a look up table), the message can be placed into an error queue. When the issue is resolved, the message can be requeued for processing.
Figure 5: Event brokers simplify replay after human intervention
To this point, we’ve focused on the benefits of decoupling a single sender and receiver from one another, and the benefits of that decoupling are worthwhile on their own. But the value really adds up as you begin distributing transactions amongst multiple diverse consumers using a fully event-driven architecture. Making your integrations more agile lets your architects and developers innovate more freely, solving business needs and exploring new frontiers for your business.
Architects need to consider complex questions including:
To aid agility, iPaaS providers typically feature integrated source control and deployment capabilities. Including these components reduces the amount of additional infrastructure needed to start building an iPaaS – no more setting up a Git repository and configuring Jenkins.
Most integration platforms as a service include the ability to call multiple sender processes from a single receiver process, but as Figure 6 shows, even with the code promotion infrastructure in place, if the solution tightly couples senders and receivers of information you still need to have a complete test cycle and deployment with every change.
That cycle is represented by a link icon in the diagram below. This tight coupling also makes error handling more complex, particularly in cases where one consumer rejects the transaction and additional consumers process it successfully.
Figure 6: iPaaS with multiple tightly coupled consumers
An event broker allows you to move into the realm of fully event-driven architecture. At the outset, the addition of an event broker may seem like it increases complexity — after all, there are now more components — but the addition of those components is more than offset by the fact that an event broker makes code more modular and reduces integration challenges.
By playing middleman between receivers and senders, an event broker lets you change a sender without redeploying and retesting the entire process. What’s more, you can very easily add new senders and consumers, and seamlessly integrate them into an existing solution.
With an event broker, you can design a new sender connecting to an innovative new application using iPaaS tools like graphical mapping, content-based filtering, and pre-built connectivity. Then once you add a new subscription to the event broker, events begin flowing to the new process and onward to the information consumer.
Figure 7: iPaaS with loosely coupled consumers using event broker
Learn more about the Boomi and Solace partnership in our blog post "Boomi and Solace Deliver Event-Driven iPaaS to Power Real-Time Integration."
At runtime, decoupling lets consumers process information independently. This can help to resolve several difficult architectural challenges:
Beyond decoupling and event-driven architecture between iPaaS components, an event broker provides an on-ramp to real-time capabilities for your iPaaS. The internet of things (IoT) and other real-time information sources such as Workday and Salesforce present a massive opportunity to increase your business’s ability to react, but you typically need to act on such information in real-time to fully exploit its value. Real-time information sources can also send a massive spike in traffic during periods of high demand.
The need to process incoming events in real-time, and deal with sudden unpredictable bursts of data can be challenges for any iPaaS. In the absence of an event broker, an iPaaS is typically triggered either through a synchronous API request, or on a polling interval defined when the process is developed and deployed.
In the case of a process invoked by a synchronous API request, dealing with traffic spikes within the structures of an iPaaS typically means increasing the number of processing threads available to handle the incoming flow of information and over allocation of memory and processing power. In the case of a polling interval, it is difficult to predict how often to poll far in advance, especially if you can’t change that interval when conditions change.
Figure 8: Using polling for real-time event processing
An event broker helps you overcome these challenges by providing an entry point for event-driven applications. Messages that are placed on event broker topics can trigger your iPaaS immediately, without having to worry about selecting a static polling interval. An event broker can also buffer incoming events and provide a throttling mechanism that prevents downstream processes from being overwhelmed in periods of high traffic.
Figure 9: Using an event broker for real-time event processing
Event brokers are a great potential partner for your iPaaS, enhancing their resilience, agility and real-time capabilities. Recognizing those potential benefits is the first step in the journey. The next steps are recognizing and overcoming some challenges that event brokers can introduce. In my next post, I’m looking forward to answering questions such as:
Together, Boomi and Solace provide integration and data movement for any hybrid environment at scale, regardless of complexity. Find out more in our partner solution brief.