Understanding Modern Enterprise Integration Requirements
Understanding Modern Enterprise Integration Requirements
See how microservices are helping overcome enterprise integration barriers in the age of containers, DevOps, and high velocity.
Join the DZone community and get the full member experience.Join For Free
Record growth in microservices is disrupting the operational landscape. Read the Global Microservices Trends report to learn more.
Enterprise Application Integration (EAI) is a complex problem to solve and different software vendors have produced different types of software products like ESBs, application servers, message brokers, API gateways, load balancers, proxy servers, and many others. These products have evolved from a monolithic, heavyweight, high-performing runtimes to lean, modularized, micro-runtimes. Microservices Architecture (MSA) is having a major impact on the way architects design their enterprise software systems. The requirements ten years ago have drastically changed due to modern advancements in MSA, containers, DevOps, agility and crazy customer demands.
If you came here to understand new EIP patterns, this is not the right place. In reality, the 65 EIP patterns introduced by Gregor Hohpe are still in action with a few more additional patterns coming through. Refer to this link to understand the EIP patterns you can still use with the modern architecture.
Another misconception is that "integration" means just connecting different systems which communicate using heterogeneous protocols and messaging formats. In practice, integration architects need to understand the whole ecosystem of an enterprise software architecture.
The above figure depicts a typical integration architecture within a modern enterprise. It consists of the following layers.
- Data layer — This is the layer in which your business data resides. Databases can be RDBMS, NoSQL, or ERP/CRM systems like SAP, PeopleSoft, or MS Dynamics, or any proprietary systems you have built.
- Service layer — This is where the MSA and the technologies like containers, Docker, Kubernetes are started to make an impact. With the concepts like Domain Driven Design (DDD) or "doing one task at best" type of approaches have made this layer more lean.
- Integration layer — This is where the traditional integration happens with service orchestrations, data transformations, protocol switching, and all EIP patterns mentioned above play a major role at this layer.
- API management layer — This has become an integral part of any integration project where it provides a clean abstraction for the business as well as consumers to act upon. Capabilities like authentications, authorization, caching, throttling, and analytics are provided at this layer so that every integration or backend service does not need to worry about implementing it.
- Load balancing layer — This is the layer where network and threat protection alongside load balancing is implemented. External attacks like DOS or XML injections are prevented at this layer
It is essential to know about these layers if you are designing an enterprise software system. But knowing them and the features at each layer is not enough to make decisions when you are actually doing the implementation. You need to understand the main requirements of each and every layer at an architectural level before deciding on specific technology or vendor(s).
This is where the business critical data resides in your system. You should have the following main requirements covered by any of the technology/vendors you are going to select for the implementation.
- Consistency — The data you stored in this layer should be consistent. Any tampering of data needs to be avoided and necessary measures need to be taken to prevent such attempts.
- Persistence at scale — Your system should be capable of storing events at very high rates without losing consistency. In the meantime, it should be available to read the persisted data.
- Security — This is a critical requirement of your data stores since this data derives the value for your business. Any unintended use or access needs to be prevented.
This layer consists of actual business logic and services which offers the information which is needed by most of the client systems. In the SOA world, you may have a few services with each service covering a particular area of functionality at a macro level. But in an MSA type of implementation, each service covers a unique functionality which is at a micro level. As an example, you may have a flight booking service in SOA and a flight information service, flight schedule service, and flight rate service as separate microservices in MSA. In any of the situations, you need to deploy these services independently so that they will not have an impact on each other when there is a failure. At this layer, the system should be capable of fulfilling the below main requirements:
- Agility — This layer should be implemented in an agile manner with proper testing and automation strategies built around that. Since this layer provides the more up-to-date, business-critical information, new services need to be implemented more frequently when the business grows. Proper CI/CD processes and build pipelines needs to be integrated with the selected technology.
- Latency — Another important aspect of this layer is the latency. The results should be delivered with minimum latency since there are additional layers which add up to the overall latency due to network interference. The services implemented at this layer should more clean and efficient to provide better latencies.
- Automation — The other important factor to consider here is the level of automation and the different levels of testing carried out before releases. Since this layer is at the heart of the enterprise system and the core of your business, any false data or service outages can cause loss of business.
This layer provides the mediation capabilities required for different systems to interact with each other. You can find tons of different integration products and technologies which provides the feature set which is required at this level, but not every technology/vendor is capable of fulfilling these critical but future-proof requirements:
- Simplicity — Integration of services needs not be as hard as implementing business logic at the services layer. You should have easy-to-use syntax and visual tooling to build these integrations without much fuss. In the meantime, it should provide the necessary building blocks to deal with integration requirements at production deployments. Concepts like data types, resiliency, containerization need to be supported.
- Agility — This might not have been a requirement in the past where you had bulky ESBs, Message Brokers deployed for integration. But in a modern enterprise, your integration technology needs to be agile and capable of supporting frequent releases, automation, and DevOps integrations.
- Stability — When you deal with heterogeneous systems, there is a high chance that the system might interact with unexpected data types. This can cause stability issues in your integration layer. It is essential that your integration system should be capable of handling such scenarios without losing stability.
- Resiliency — Another aspect of integrating heterogeneous systems is that those system can fail at any moment and the integration layer should be able to withstand those failures without propagating those failures to upper layers. Technologies like circuit breakers and retry need to be supported for resiliency.
- Performance — When it comes to performance at the integration layer, it should be capable of accepting concurrent requests at a higher rates while dispatching them at a lower rates to the backend services which are not that capable of handling concurrent connections. This capability is critical when you have systems which has different message processing rates.
- Caching — This capability is nice to have at the integration layer, given that it is much closer to the actual back-end service. If the backend services layer is malfunctioning, the integration layer can provide some sort of cached data in some scenarios.
API Management Layer
Once you have the backend services layer and the integration layer implemented, the system is ready to expose valuable information to external systems. If your system is dealing with internal systems, you might not need an API management layer. But in most pragmatic integration projects, an API management layer is a core requirement. Instead of implementing common requirements at each and every services level, it is more efficient and scalable to add an API management layer. You will find dozens of API management vendors in the market which offer the same set of basic features required for traditional API management requirements. A more modern API management solution should be able to cater to the below set of requirements:
- Authentication and Authorization — The system should be capable of authenticating users who request access to the services layer. Authorization can be an added feature to secure the services at a fine-grained level. OAuth2 has become the defacto standard for access delegation and authentication. The system should be capable of handling OAuth2.
- Throttling — The system needs to be properly controlled so that users get a fair share of access across the board and if you are charging for usage; you need to apply more advanced throttling policies.
- Analytics — Monitoring and analyzing your API usage is critical to improve your business processes and drive new revenue. Business level analytics is more critical here alongside with system status monitoring.
- Availability — High availability is critical at this layer since users don't want to see empty pages or internal server errors at any given time. By making this layer highly available, we can account for failures at upstream layers like integration and back-end services through caching at the API gateway.
- Resiliency — Another critical requirement at the API management layer is the ability to withstand failures at upstream layers. It should not propagate the failures of those systems to other downstream layers like load balancers. This can be achieved by having mechanisms like a circuit breaker, bulkhead, and retry at this layer.
- Caching — Intelligent and adaptive caching technologies can be useful to provide uninterrupted service to the consumers. Caching helps to improve the overall performance of the system by reducing the load on the backend systems. When the backend services are not available, intelligent caching systems can still provide cached results to the consumers.
- Developer experience — If you are exposing your APIs to the developers to browse through the APIs and create their own applications, developer experience is another key requirement of the system. Sometimes developers may need to build their own APIs by composing existing APIs. This API composition capability is a nice-to-have requirement in most cases.
Load Balancer/Proxy Layer
Exposing your business functionality to the public internet needs to be done very carefully given the number of data breaches and hacking activities happen around the world. That is why you need to have a proper load balancing or proxy layer to provide basic protection to your information system. Main requirements which need to be fulfilled at this layer are
- Security — This layer should be able to handle basic security threats like DOS attacks, XML injection, CSS, CRLF, and others. It should also be able to support SSL.
- Resiliency — At this layer, resiliency means that any failures in the upstream systems must not take this layer down. When the upstream systems are not available, it should be properly handled at this layer through techniques like circuit breaker and retry.
- Performance — This is the layer which will serve all the requests coming into the system. We can reduce the load on backend systems by using caching technologies at other layers. But this layer is not able to do that since it is at the forefront of the users. Having good performance in terms of concurrency handling is essential at this layer.
- Caching — Caching capabilities at this layer will eventually improve the overall system performance by reducing the load on the upstream systems. The caching needs to be intelligent and adaptive since multiple layers of data invalidation happen in the upstream systems.
- Load balancing — Obviously, this is one of the core capabilities required at this layer to handle the enormous load coming into the system.
- Availability — This is the layer where availability is most critical. Since there are no other ways to mimic the availability of this layer (similar to caching), having high availability at this layer is a must.
As discussed in the above sections, having a layered architecture allows us to think deeply about those layers and their requirements. It also makes the way to choose the best-suited technology or vendor at each layer. There are technologies/vendors who are highly praised by analyst reports for each and every layer. It is the task of solution architects and CTOs to make the final decision.
Published at DZone with permission of Chanaka Fernando , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.