Integration

Jolie is a service-oriented programming language, developed with the aim of making some key principles of service-oriented programming syntactically manifest. Express is a leading framework in the world of JavaScript for the creation of REST services. The concepts behind Express/JavaScript and Jolie are not that distant, but they are offered under different shapes: one aimed at building on top of JavaScript and Node.js, the other aimed at expressing these concepts declaratively. The aim of this article is to identify the first differences that arise between Express and Jolie in the development of a REST service by comparing how the same concepts are codified. We ground our comparison in a concrete example: a simple REST service that exposes an API for retrieving information about users. Users are identified by username and associated to data that includes name, e-mail address, and an integer representing "karma" that the user has in the system. In particular, two operations are possible: Getting information about a specific user (name, e-mail, and karma counter) by passing its username, for example by requesting /api/users/jane. Listing the usernames of the users in the system, with the possibility of filtering them by karma. For example, to get the list of usernames associated to minimum karma 5, we could request /api/users?minKarma=5. We assume some familiarity with JavaScript. Express The following code implements our simple example with Express. JavaScript const express = require('express') const http = require('http') const { validationResult , query } = require('express-validator') const app = express() const users = { jane: { name: "Jane Doe", email: "jane@doe.com", karma: 6 }, john: { name: "John Doe", email: "john@doe.com", karma: 4 } } app.get('/api/users/:username', (req, res) => { const errors = validationResult(req); if (!errors.isEmpty()) { return res.status(400).json({ errors: errors.array() }); } if (users[req.params.username]) { return res.status(200).json(users[req.params.username]) } else { return res.status(404).json(req.params.username) } }) app.get('/api/users', query('minKarma').isNumeric(), (req, res) => { const errors = validationResult(req); if (!errors.isEmpty()) { return res.status(400).json({ errors: errors.array() }); } let usernames = [] for (username in users) { if (req.query.minKarma && req.query.minKarma < users[username].karma) { usernames.push(username) } } const responseBody = {} if (usernames.length != 0) { responseBody['usernames'] = usernames } res.status(200).json(responseBody) }) app.listen(8080) We start by importing from some library modules, using require, and then we create the application (const app = express()). To represent data about users, we use a simple object (users) that associates usernames (as properties) to objects with name, e-mail address, and karma. We have two users, respectively identified by the usernames john and jane. We then add two routes to our application: The first route is for getting information about a specific user. We start by validating the request and returning immediately in case there are any validation errors. Then, we check if the requested username is in our users data object: if so, we return information about that user; otherwise, we return an error with status code 404. The second route is for listing users. The boilerplate for validation and error checking is similar. We use a for loop to find the usernames of users with sufficient karma and accumulate them in an array, which is then returned to the caller. After we are done configuring our application, we launch it by listening on TCP port 8080. Jolie We now implement the same example in Jolie. type User { name: string, email: string, karma: int } type ListUsersRequest { minKarma?: int } type ListUsersResponse { usernames*: string } type ViewUserRequest { username: string } interface UsersInterface { RequestResponse: viewUser( ViewUserRequest )( User ) throws UserNotFound( string ), listUsers( ListUsersRequest )( ListUsersResponse ) } service App { execution: concurrent inputPort Web { location: "socket://localhost:8080" protocol: http { format = "json" osc << { listUsers << { template = "/api/user" method = "get" } viewUser << { template = "/api/user/{username}" method = "get" statusCodes.UserNotFound = 404 } } } interfaces: UsersInterface } init { users << { john << { name = "John Doe", email = "john@doe.com", karma = 4 } jane << { name = "Jane Doe", email = "jane@doe.com", karma = 6 } } } main { [ viewUser( request )( user ) { if( is_defined( users.(request.username) ) ) { user << users.(request.username) } else { throw( UserNotFound, request.username ) } } ] [ listUsers( request )( response ) { i = 0 foreach( username : users ) { user << users.(username) if( !( is_defined( request.minKarma ) && user.karma < request.minKarma ) ) { response.usernames[i++] = username } } } ] } } We start by declaring the API (types and interface) of the service, with the two operations for listing and viewing users. Then, we define the implementation of our service by using a service block, which is configured to handle client requests concurrently (execution: concurrent). The input port Web defines an HTTP access point for clients, and contains a protocol configuration that maps our two operations to the expected HTTP resource paths (using URI templates), method, and status codes. The data structure about users is defined in the block for service initialisation (init). The << operator is used in Jolie to make deep copies of data trees. The behaviour of the service is defined by the main procedure, which offers a choice ([..]{..} [..] {..}) between our two operations. The operations are implemented using a logic similar to that seen in the previous JavaScript code snippet. In viewUser, if the requested username cannot be found, we throw a fault (UserNotFound)—users.(request.username) in Jolie roughly translates to users[request.username] in JavaScript. Comparison From this simple example, we can observe a few interesting differences. We focus on concepts, leaving minor details or aestethic differences out of the discussion. API-First and Validation The Jolie code starts by defining explicitly the API of the service, whereas in Express the API is implicitly given by the routes created on the application object. (Message types can actually be omitted in Jolie, but it is considered a best practice to include them.) We might say that Jolie follows an "API-first" approach, whereas in Express APIs can be defined after a service is coded (or not at all, even). There are two important features that Jolie provides out-of-the-box if you provide message types. Runtime Checks: all messages are checked at runtime to respect their expected types. So writing code for validating the request is not necessary in Jolie. Automatic Casting: parameters in querystrings are automatically converted from strings to their expected types. If casting is not possible, an error is returned to the client automatically. In Express, req.query.minKarma is a string, which JavaScript can cast automatically to a number when needed. To check that this would not be a problem, we had to add the validation code query('minKarma').isNumeric(). Manual versions of these features can be implemented in an Express application. For example, one can use the express-validator middleware in the configuration of the application in order to check that messages respect some JSON schemas. Jolie offers additional integrated mechanisms for performing validation, for example refinement types and service decorators. We leave a comparison of validation techniques in Express and Jolie to another article. Code Structure Defining a service requires dealing with its access points (how it can be accessed by clients) and its implementation (see also the 5 principles of service-oriented programming). In Express, these aspects are coded by calling methods of the application object. The approach is multi-paradigmatic: it combines objects, functions, and imperative programming. The service is represented as an object (app in our example), and the operations offered by the service are defined together with their implementations step-by-step. The definition of a service is given by the internal state of the application object (modified by invoking the get method in our example). In Jolie, the configuration of access points and the business logic of the operations that the service offers are kept separate. The definition of a service and its access points (service App and inputPort Web in our example, respectively) are given in a declarative approach, which makes the structure of a service syntactically manifest. Jolie is service-oriented: the necessary concepts for defining a service are supported by linguistic primitives (in contrast to libraries). The implementation of each operation in Jolie is then given in the main block, and is pure business logic that abstracts from the concrete data format (and even transport) used by the access point. The mapping from resource paths to operations (and faults to status codes) is given in the configuration of the access point. In general, operation implementations can be reused with different access point configurations. For example, we could easily reconfigure our service to send XML responses by changing format = "json" in the configuration of the input port to format = "xml". Conclusion In this article, we have explored the differences between Express and Jolie in the context of a simple REST service. True to the dynamic nature of JavaScript, Express sets up the API and access point of a service by using subsequent method invocations, whereas in Jolie this is achieved with a more static and declarative approach. Another interesting difference lies in the approaches to business logic: the syntax of Jolie naturally guides the developer towards keeping business logic abstract from concrete data formats.

Developers always need meaningful insights about their code to be effective in their day-to-day programming activities regarding building APIs for either internal or external use. Logging and analytics are always instrumental in providing adequate and necessary insights to developers. Likewise, API logging and analytics give developers the insight they need to ensure APIs perform their intended functions effectively. To appreciate the importance of API logging and analytics, it is necessary to understand what API logging and analytics are. What Are API Logging and Analytics? Source: Sitepoint API logging is keeping relevant records of input, process, and output of an API, which can be used for various purposes. API logging is carried out whenever we record operations of an API, whether they failed or succeeded. On the other hand, analytics is used for the discovery, interpretation, and communication of meaningful patterns in data. Therefore, API analytics can be described as a way of getting useful insight into the operation of an API. API logging and analytics are instrumental in providing the necessary insights developers need to be more efficient in building products, achieving business goals, and delivering a great customer experience. Importance of API Logging and Analytics for Developers API logging and analytics are important for many reasons: they provide diagnoses, prescriptions, descriptions, and predictions, all of which make development more effective for developers. Simplified Diagnosis of API Errors APIs are meant to be consumed and errors are bound to manifest sometimes. If no data is logged that means there is nothing to analyze to troubleshoot the errors, so dealing with errors would be based on guesses. Some errors may not even be discovered in time, so they would end up affecting user experience. API logging and analytics are advantageous in this case because logs can be analyzed to get insights about errors the APIs run through every time consumers make requests. Developers can then pick up the errors from the logs, analyze them to gain insights, and fix them with precision. Understanding How APIs Are Being Consumed Many APIs don’t even log consumer interactions, and that makes it hard to understand how such APIs are being used by consumers. So, it is hard to measure consumer behavior regarding their interactions with the APIs, leaving developers with no insight into the harm and opportunities that might have come with such interactions. Understanding how your APIs are being used guides developers toward providing better security, support, and experience to their consumers. For example, some consumers might be making malicious requests to your APIs, and they either fail or succeed at this. API logging and analytics bring such occurrences to the developers’ attention as quickly as possible so they can tackle the occurrence favorably. Without API logging and analytics, it is likely that nothing would be known until serious damage occurred. Anyway, some portals for developers such as the Backstage developer portal makes all the moving parts easy to handle. Source: API Portal Gathering of Important Metrics Measuring an API’s operations gives you the opportunity to gather valuable metrics that can help developers make such APIs better. The frequency of requests, number of users, demography of users, and several other stats can be great insights for developers to set plans to make their APIs better. It might help in determining their choice of infrastructure and more. For example, having detailed logs and analytics of a consistent increase in user base and retained users can be an indication for developers to improve infrastructures in order to not go out of service. It can also help them move their server close to the country of their main audience because that will translate to a better experience for the audience. Provision of Business Insight APIs are built to create value for consumers and make profits eventually. API calls can be attributed to individuals interacting with them and that can be used to measure user behavior analytics, which can be effective in calculating revenue, describing user activities, and forecasting business and user needs in the future. Now, the question is this: How are business insights important to developers? The answer is that it prompts developers to design adequate systems and architectures that can withstand the business requirements as indicated by the insights derived from API logging and analytics. Conclusion Building APIs to withstand the test of time requires a lot of work for developers and it goes beyond what could be done with guesswork and anecdotal decisions. This is why it is important that developers be well equipped with API logging and analytics to gain the information and insights they need to build APIs that consumers enjoy and love to use. API logging and analytics are important to developers because it gives them the information and insight to do their job adequately, which enables them to improve on serving their users and advance business goals.

Web Speech API is a web technology that allows you to incorporate voice data into apps. It converts speech into text and vice versa using your web browser. The Web Speech API was introduced in 2012 by the W3C community. A decade later, this API is still under development and has limited browser compatibility. This API supports short pieces of input, e.g., a single spoken command, as well as lengthy, continuous input. The capability for extensive dictation makes it ideal for integration with the Applause app, while brief inputs work well for language translation. Speech recognition has had a dramatic impact on accessibility. Users with disabilities can navigate the web more easily using their voices. Thus, this API could become key to making the web a friendlier and more efficient place. The text-to-speech and speech-to-text functionalities are handled by two interfaces: speech synthesis and speech recognition. Speech Recognition In the speech recognition (SpeechRecognition) interface, you speak into a microphone and then the speech recognition service then checks your words against its own grammar. The API protects the privacy of its users by first asking permission to access your voice by a microphone. If the page using the API uses the HTTPS protocol, it asks for permission only once. Otherwise, the API will ask in every instance. Your device probably already includes a speech recognition system, e.g., Siri for iOS or Android Speech. When using the speech recognition interface, the default system will be used. After the speech is recognized, it is converted and returned as a text string. In "one-shot" speech recognition, the recognition ends as soon as the user stops speaking. This is useful for brief commands, like a web search for app testing sites, or making a call. In "continuous" recognition, the user must end the recognition manually by using a "stop" button. At the moment, speech recognition for Web Speech API is only officially supported by two browsers: Chrome for Desktop and Android. Chrome requires the use of prefixed interfaces. However, the Web Speech API is still experimental, and specifications could change. You can check whether your current browser supports the API by searching for the webkitSpeechRecognition object. Speech Recognition Properties Let’s learn a new function: speechRecognition(). var recognizer = new speechRecognition(); Now let’s examine the callbacks for certain events: onStart:onStart is triggered when the speech recognizer begins to listen to and recognize your speech. A message may be displayed to notify the user that the device is now listening. onEnd:onEnd generates an event that is triggered each time the user ends the speech recognition. onError: Whenever a speech recognition error occurs, this event is triggered using the SpeechRecognitionError interface. onResult: When the speech recognition object has obtained results, this event is triggered. It returns both the interim results and the final results. onResult must use the SpeechRecognitionEvent interface. The SpeechRecognitionEvent object contains the following data: results[i]: An array of the result objects of the speech recognition, each element representing a recognized word resultindex: The current recognition index results[i][j]: The j-th alternative to a recognized word; the first word to appear is the word deemed most probable results[i].isFinal: A Boolean value that displays whether the result is interim or final results[i][j].transcript: The word’s text representation results[i][j].confidence: The probability of the result being correct (value range from 0 to 1) What properties should we configure on the speech recognition object? Let’s take a look. Continuous vs One-Shot Decide whether you need the speech recognition object to listen to you continuously until you turn it off, or whether you only need it to recognize a short phrase. Its default setting is "false." Let’s say you are using the technology for note-taking, to integrate with an inventory tracking template. You need to be able to speak at length with enough time to pause without sending the app back to sleep. You can set continuous to true like so: speechRecognition.continuous = true; Language Image Sourced from Google What language do you want the object to recognize? If your browser is set to English by default, it will automatically select English. However, you can also use locale codes. Additionally, you could allow the user to select the language from a menu: speechRecognition.lang = document.querySelector("#select_dialect").value; Interim Results Interim results refer to results that are not yet complete or final. You can enable the object to display interim results as feedback to users by setting this property to true: speechRecognition.interimResults = true; Start and Stop If you have configured the speech recognition object to “continuous,” then you will need to set the onClick property of the start and stop buttons, like so: JavaScript document.querySelector("#start").onclick = () => { speechRecognition.start(); }; document.querySelector("#stop").onclick = () => { speechRecognition.stop(); }; This will allow you to control when your browser begins "listening" to you, and when it stops. So, we’ve taken an in-depth look at the speech recognition interface, its methods, and its properties. Now let’s explore the other side of Web Speech API. Speech Synthesis Speech synthesis is also known as text-to-speech or TTS. Speech synthesis means taking text from an app and converting it into speech, then playing it from your device’s speaker. You can use speech synthesis for anything from driving directions to reading out lecture notes for online courses, to screen-reading for users with visual impairments. In terms of browser support, speech synthesis for Web Speech API can be used in Firefox desktop and mobile from Gecko version 42+. However, you have to enable permissions first. Firefox OS 2.5+ supports speech synthesis by default; no permissions are required. Chrome and Android 33+ also support speech synthesis. So, how do you get your browser to speak? The main controller interface for speech synthesis is SpeechSynthesis, but a number of related interfaces are required, e.g., for voices to be used for the output. Most operating systems will have a default speech synthesis system. Put simply, you need to first create an instance of the SpeechSynthesisUtterance interface. The SpeechSynthesisUtterance interface contains the text the service will read, as well as information such as language, volume, pitch, and rate. After specifying these, put the instance into a queue that tells your browser what to speak and when. Assign the text you need to be spoken to its "text" attribute, like so: newUtterance.text = The language will default to your app or browser’s language unless specified otherwise using the .lang attribute. After your website has loaded, the voiceschanged event can be fired. To change your browser’s voice from its default, you can use the getvoices() method within SpeechSynthesisUtterance. This will show you all of the voices available. The variety of voices will depend on your operating system. Google has its own set of default voices, as does Mac OS X. Finally, choose your preferred voice using the Array.find() method. Customize your SpeechSynthesisUtterance as you wish. You can start, stop and pause the queue, or change the talking speed (“rate”). Web Speech API: The Pros and Cons When should you use Web Speech API? It’s great fun to play with, but the technology is still in development. Still, there are plenty of potential use cases. Integrating APIs can help modernize IT infrastructure. Let’s look at what Web Speech API is able to do well, and which areas are ripe for improvement. Boosts Productivity Talking into a microphone is quicker and more efficient than typing a message. In today’s fast-paced working life, we may need to be able to access web pages while on the go. It’s also fantastic for reducing the admin workload. Improvements in speech-to-text technology have the potential to significantly cut time spent on data entry tasks. STT could be integrated into audio-video conferencing to speed up note-taking in your stand-up meeting. Accessibility As previously mentioned, both STT and TTS can be wonderful tools for users with disabilities or support needs. Additionally, users who may struggle with writing or spelling for any reason may be better able to express themselves through speech recognition. In this way, speech recognition technology could be a great equalizer on the internet. Encouraging the use of these tools in an office also promotes workplace accessibility. Translation Web Speech API could be a powerful tool for language translation due to its capability for both STT and TSS. At the moment, not every language is available; this is one area in which Web Speech API has yet to reach its full potential. Offline Capability One drawback is that an internet connection is necessary for the API to function. At the moment, the browser sends the input to its servers, which then returns the result. This limits the circumstances in which Web Speech can be used. Accuracy Incredible strides have been made to refine the accuracy of speech recognizers. You may occasionally still encounter some struggles, such as with technical jargon and other specialized vocabularies, or with regional accents. However, in 2022, speech recognition software is now reaching human-level accuracy. In Conclusion Although Web Speech API is experimental, it could be an amazing addition to your website or app. From top PropTech companies to marketing, all workplaces can use this API to supercharge efficiency. With a few simple lines of JavaScript, you can open up a whole new world of accessibility. Speech recognition makes it easier and more efficient for your users to navigate the web. Get excited to watch this technology grow and evolve!

Request-response communication with REST / HTTP is simple, well-understood, and supported by most technologies, products, and SaaS cloud services. Contrarily, data streaming with Apache Kafka is a fundamental change to process data continuously. HTTP and Kafka complement each other in various ways. This post explores the architectures and uses cases to leverage request-response together with data streaming in the control plane for management or in the data plane for producing and consuming events. Request-Response (HTTP) Versus Data Streaming (Apache Kafka) Prior to discussing the relationship between HTTP/REST and Apache Kafka, let's explore the concepts behind both. Traditionally, request-response and data streaming are two different paradigms. Request-Response With REST/HTTP The following characteristics make HTTP so prevalent in software engineering for request-response (aka request-reply) communication: The foundation of data communication for the World Wide Web The standard application layer protocol in the internet protocol suite, commonly known as TCP/IP Simple and well understood Supported by most open-source frameworks, proprietary products, and SaaS cloud services Pre-defined API with GET, POST, PUT, and DELETE commands Typically synchronous communication, but chunked transfer encoding (i.e., streaming) is also possible Point-to-point message exchange between two applications (like a client and server or two independent microservices) Data Streaming With Apache Kafka HTTP is about communication between two applications. On the contrary, data streaming is much more than just data communication between a client and a server. Hence, data streaming platforms like Apache Kafka have very different characteristics: No official web standard like HTTP Plenty of open-source and proprietary implementations exist An event-driven system with asynchronous communication using truly decoupled producers and consumers due to the storage of the streaming platform General-purpose events instead of pre-defined APIs - contract management using schemas is crucial in larger projects for API enforcement and data governance Continuous data processing in real-time at any scale - a fundamental change for developers that are used to web services and databases for building applications Backpressure handling, slow consumers, and replayability of historical events are core concepts built-in out-of-the-box Data integration and data processing capabilities are built into the data streaming platform, i.e., it is not just a message queue Please note that this article specifically discusses Apache Kafka as it is the established de facto standard for data streaming. It powers most data streaming distributions like Confluent, Red Hat, IBM, Cloudera, TIBCO, and many more, plus cloud services like Confluent Cloud and Amazon MSK. Nevertheless, other frameworks and cloud services like Apache Pulsar, Redpanda, Apache Flink, AWS Kinesis, and many other data streaming technologies follow the same principles. Just be aware of the technical differences and trade-offs between data streaming products. Request-Response and Data Streaming Are Complimentary Most architectures need request-response for point-to-point communication (e.g., between a server and mobile app) and data streaming for continuous event processing. Event sourcing with CQRS is the better design for most data streaming scenarios. However, developers can implement the request-response message exchange pattern natively with Apache Kafka. Nevertheless, direct HTTP communication with Kafka is the easier and often better approach for appropriate use cases. With this in mind, let’s look at use cases where HTTP is used with Kafka and how they complement each other. Why Is HTTP/REST So Popular? Most developers and administrators are familiar with REST APIs. They are the natural option for many best practices and security guidelines. Here are some good reasons why this will not change in the future: Avoiding technology lock-in: Sometimes, you want to embed the communication or proxy it with a more agnostic API. Familiarity with a known technology: Developers are familiar with REST endpoints and if they are under pressure or need a quick result, it’s quicker than learning how to use a new API. Supported by almost all products: Most open-source frameworks, commercial products, and SaaS cloud provide HTTP APIs. Security: HTTP ports are much easier to open by security teams compared to the TCP ports of the Kafka-native protocol used by client APIs from programming languages such as Java, Go, C++, or Python. For instance, in DMZ pass-through requirements, InfoSec owns the F5 proxies in the DMZ. A Kafka REST Proxy makes the integration easier. Domain-driven design (DDD): Often, HTTP/REST and Kafka are combined to leverage the best of both worlds: Kafka for decoupling and HTTP for synchronous client-server communication. A service mesh using Kafka with REST APIs is a common architecture. Other great implementations exist for request-response communication. For instance: gRPC: Efficient request-response communication via a cross-platform open source high-performance Remote Procedure called Framework GraphQL Data query and manipulation language for APIs and a runtime for fulfilling queries with existing data Nevertheless, HTTP is the first choice in most projects. gRPC, GraphQL, or other implementations are chosen for specific problems if HTTP is not good enough. Use Cases for HTTP/REST APIs With Apache Kafka RESTful interfaces to an Apache Kafka cluster make it easy to produce and consume messages, view the cluster's metadata, and perform administrative actions using standard HTTP(S) instead of the native TCP-based Kafka protocol or clients. Each scenario differs significantly in its purpose. Some use cases are implemented out of convenience, while others are required because of technical specifications. There are two major categories of use cases for combining HTTP with Kafka. In terms of cloud-native architectures, this can be divided into the management plane (i.e., administration and configuration) and the data plane (i.e., producing and consuming data). Management Plane With HTTP and Kafka The management and administration of a Kafka cluster involve various tasks, such as: Cluster management: Creation of a cluster, expanding or shrinking a cluster, etc. Cluster configuration: Management of Kafka topics, consumer groups, key management, role-based access control (RBAC), etc. CI/CD and DevOps integration: HTTP APIs are the most popular way to build delivery pipelines and automate administration, instead of using Python or other alternative scripting options. Data governance: Tools for data lineage, data catalogues, and policy enforcement need to be configured using APIs. 3rd party monitoring integration: Connect metrics APIs, alerts, and other notifications into systems like Datadog, Slack, etc. Data Plane With HTTP and Kafka Many scenarios require or prefer the usage of REST APIs for producing and consuming messages to/from Kafka, such as Natural request-response applications such as mobile apps: These applications and the frameworks almost always require integration via HTTP and request-response. WebSockets, Server-Sent Events (SSE), and similar concepts are a better fit for data streaming with Kafka. They are in the client framework, though often not supported. Legacy application and third-party tool integration: Legacy applications, standard software, and traditional middleware are often proprietary. The only integration capabilities are HTTP/REST. Extract, transform, load (ETL), enterprise service bus (ESB), and other third-party tools are complementary to data streaming with Kafka. Mainframe integration using REST APIs from COBOL to Kafka is another example. API Gateway: Most API management tools do not provide native support for data streaming and Kafka today and only work on top of REST interfaces. Kafka (via the REST interface) and API management are still very complementary for some use cases, such as service monetization or integration with partner systems. Other programming languages: Kafka provides Java and Scala clients. Confluent provides and supports additional clients, including Python, .NET, C, C++, and Go. More Kafka clients exist from the community, including Erlang, Kotlin, Node.js, PHP, Ruby, and Rust. Many of these community clients are not battle tested or supported. Therefore, calling the REST API from your favourite programming language is sometimes the better and easier option. Others, such as COBOL on the mainframe, don’t even provide a Kafka client at all. Hence, a REST API is the only viable solution example: HTTP + Kafka With Confluent REST Proxy. Example: HTTP + Kafka With Confluent Rest Proxy The Confluent REST Proxy has been around for a long time and is available under the Confluent Community License. Many companies use it in production as a management plane and data plane as a self-managed component in conjunction with open-source Apache Kafka, Confluent Platform, or Confluent Cloud. While not being a lawyer, the short version is that you can use the Confluent REST Proxy for free - even for your production workloads at any scale - as long as you don't build a competitive cloud service with it (say, e.g., the "AWS Kafka HTTP Proxy") and charge per hour or volume for the serverless offering. The Confluent REST Proxy and REST APIs are separated into both a data plane and a management plane: While some applications require both, in many scenarios, only one or the other is used. The management plane is typically used for very low throughout and a few API calls. The data plane, on the other hand, varies. Many applications produce and consume data continuously. The biggest limitation of the REST Proxy data plane is that it is a synchronous request-response protocol. What Scale and Volumes Does a REST Proxy for Kafka Support? Don’t underestimate the power of the REST Proxy as a data plane because Kafka provides batch capabilities to scale up to many parallel REST Proxy instances. There are deployments where four REST Proxy instances can handle ~20,000 events per second, which is sufficient for many use cases. Many HTTP use cases do not require millions of events per second. Hence, the Confluent REST Proxy is often good enough. This is even true for many IoT use cases I have seen in the wild where devices or machines connect to Kafka via HTTP. How Does HTTP’s Streaming Data Transfer Fit Into the Architecture? Please note that chunked transfer encoding is a streaming data transfer mechanism available in version 1.1 of HTTP. In chunked transfer encoding, the data stream is divided into a series of non-overlapping "chunks". The chunks are sent out and received independently of one another. Some Kafka REST Produce APIs support a streaming mode that allows sending multiple records over a single stream. The stream remains open unless explicitly terminated. The streaming mode can be achieved by setting an additional header "Transfer-Encoding: chunked" on the initial request. Check if your favourite Kafka proxy or cloud API supports the HTTP streaming mode. Architecture Options for Kafka + REST Proxy Different deployment options exist for the Confluent REST Proxy. The self-managed REST Proxy instance or cluster of instances (as a “dedicated node”) is still decoupled from the open-open source Kafka broker or commercial Confluent Server. This is the ideal option for a data plane to produce and consume messages. The management plane is also embedded as a unified REST API into Confluent Server (as a “broker plugin”) and Confluent Cloud for administrative operations. This simplifies the architecture because no additional nodes are required for using the administration APIs. In some deployments, both approaches may be combined: The management plane is used via the embedded REST APIs in Confluent Server or in Confluent Cloud. Meanwhile, data plane use cases are decoupled into their own REST Proxy instances to easily handle scalability and be independent of the server side. The developer does not have to care about the infrastructure or architecture for REST APIs in Confluent Cloud. The HTTP interfaces are fully managed by the vendor. The REST APIs of the self-managed REST Proxy and Confluent Cloud are compatible. Hybrid architectures and cloud migration are possible without implementing any breaking changes. Data Governance for Data Streaming and HTTP Services With a Schema Registry Data governance is an important part of most data streaming projects. Kafka deployments usually include various decoupled producers and consumers, often following the DDD principle for microservice architectures. Hence, Confluent Schema Registry is used in most projects for schema enforcement and versioning. Any Kafka client built by Confluent can leverage the Schema Registry using Avro, Protobuf, or JSON Schema. This includes programming APIs like Java, Python, Go, or Python, but also Kafka Connect sources and sink, Kafka Streams, ksqlDB, and the Confluent REST Proxy. Like the REST Proxy, Schema Registry is available under the Confluent Community License. Schema Registry lives separately from your Kafka brokers. Confluent REST Proxy still talks to Kafka to publish and read data (messages) to topics. Concurrently, the REST Proxy can also talk to Schema Registry to send and retrieve schemas that describe the data models for the messages. A Schema Registry provides a serving layer for your metadata and enables data governance and schema enforcement for all events. It provides a RESTful interface for storing and retrieving your Avro, JSON Schema, and Protobuf schemas. The Schema Registry stores a versioned history of all schemas based on a specified subject name strategy, provides multiple compatibility settings, and allows the evolution of schemas according to the configured compatibility settings and expanded support for these schema types. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in any of the supported formats. Schema enforcement happens on the client side. Additionally, Confluent Platform and Confluent Cloud provide server-side schema validation. The latter is helpful if incorrect or malicious client applications send messages to Kafka without using the client-side Schema Registry integration. API Management and Data Sharing API Management is a term from the Open API world that puts a layer on top of HTTP / REST APIs for the management, monitoring, and monetization of APIs. Solutions include Apigee, MuleSoft Anypoint, Kong, IBM API Connect, TIBCO Mashery, and many more. Features of API Gateways and API Management Products API Gateway and API Management Tools provide many outstanding features: API Portal for creating and publishing APIs Enforcing usage policies and controlling access Technical features for data transformations Nurturing the subscriber community Collecting and analyzing usage statistics Reporting on performance Monetization and billing These features are unavailable in any data streaming platform like Kafka, Pulsar, Kinesis, et al. on the other side, API tools like MuleSoft or Kong are not built for processing real-time data at scale with low latency. API Management and Data Streaming Are Complimentary Hence, API Management and data streaming are complementary, not competitive! The blog post "Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?" explores this. API == REST/HTTP for most API Management products and related API gateways. Vendors start to integrate either the Kafka API or event standards like AsyncAPI to get into event-based architectures. That's great news! Sharing of Streaming Data Requires a Stream Data Exchange Instead of HTTP APIs Data sharing becomes crucial to modern and flexible enterprise architectures that build on concepts like microservices and data mesh. Real-time data beats slow data. That's not just true for applications but also for data replication across business units, organizations, B2B, clouds, hybrid environments, and other scenarios. Therefore, the next generation of data sharing is built on top of data streaming. HTTP APIs make little sense in many data streaming scenarios, especially if you expect high volumes or require low latency. Hence, data sharing in real-time by linking Kafka clusters or using a stream data exchange is a much better approach. I won't go into more detail here. The dedicated blog post "Streaming Data Exchange with Kafka for Real-Time Data Sharing" explores the idea, its trade-offs, and some real-world examples. Not One or the Other, Instead Combine Kafka and HTTP/Rest in Your Projects! Various use cases employ HTTP/REST with Apache Kafka as a management plane or data plane. This combination will not go away in the future. The Confluent REST Proxy can be used for HTTP(S) communication with your favourite client interface. No matter if you run open-source Apache Kafka, Confluent Platform, or Confluent Cloud. Check out the source code on GitHub or get started with an intuitive tutorial. How do you combine data streaming with request-response principles? How do you combine HTTP and Kafka? What proxy are you using? Let’s connect on LinkedIn and discuss it!

I had a really time-limited effort to do to prove how to write a command line wrapper for an open API a customer is developing. The target REST API is the jquants-api, as presented in a previous article. I chose to implement the wrapper in Golang, which proved to be extremely fast and pleasant to do. The task was eventually done in a short evening, and the resulting Golang wrapper with core features has been uploaded on GitHub. This is the short story on the process to write the API and the few different programming steps to get there. Goals So first, let’s list the programming tasks that we will have to deal with: Create a test, and supporting code, checking we can save the username and password in an edn file compatible with the jquants-api-jvm format Write another test and supporting code to retrieve the refresh token Write another test and supporting code to retrieve the ID token Write another test and supporting code using the ID token to retrieve daily values Publish our wrapper to GitHub Use our Go library in another program Start by Writing a Test Case, Preparing and Saving the Login Struct to Access the API We always talk about writing code using TDD — now’s the day to do it. Check that we have code to enter and save the username and password in an edn file compatible with the jquants-api-jvm format. In a helper_test.go file, let’s write the skeleton test for a PrepareLogin function. Go package jquants_api_go import ( "fmt" "os" "testing" ) func TestPrepareLogin(t *testing.T) { PrepareLogin(os.Getenv("USERNAME"), os.Getenv("PASSWORD")) } Here, we pick up the USERNAME and PASSWORD from the environment, using os.GetEnv. We will write the prepare function in a helper.go file. It will: Get the username and password as parameters Instantiate a Login struct Marshal this as an EDN file content func PrepareLogin(username string, password string) { var user = Login{username, password} encoded, _ := edn.Marshal(&user) writeConfigFile("login.edn", encoded) } Our Login struct will first simply be: type Login struct { UserName string `edn:"mailaddress"` Password string `edn:"password"` } And the call to edn.Marshal will create a byte[] array content that we can write to file, and so writeConfigFile will simply call os.WriteFile with the array returned from the EDN marshaling. func writeConfigFile(file string, content []byte) { os.WriteFile(getConfigFile(file), content, 0664) } To be able to use the EDN library, we will need to add it to the go.mod file with: require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 Before running the test, be sure to enter your jquants API’s credential: export USERNAME="youremail@you.com" export PASSWORD="yourpassword" And at this stage, you should be able to run go test in the project folder, and see the following output: PASS ok github.com/hellonico/jquants-api-go 1.012s You should also see that the content of the login.edn file is properly filled: cat ~/.config/jquants/login.edn {:mailaddress "youremail@you.com" :password "yourpassword"} Use the Login to Send an HTTP Request to the jQuants API and Retrieve the RefreshToken The second function to be tested is TestRefreshToken, which sends a HTTP post request with the username and password and retrieve the refresh token as an answer of the API call. We update the helper_test.go file with a new test case: func TestRefreshToken(t *testing.T) { token, _ := GetRefreshToken() fmt.Printf("%s\n", token) } The GetRefreshToken func will: Load user stored in file previously and prepare it as JSON data Prepare the HTTP request with the URL and the JSON formatted user as body content Send the HTTP request The API will returns data that will store in a RefreshToken struct And let’s store that refresh token as an EDN file The supporting GetUser will now load the file content that was written in the step before. We already have the Login struct, and will then just use edn.Unmarshall() with the content from the file. func GetUser() Login { s, _ := os.ReadFile(getConfigFile("login.edn")) var user Login edn.Unmarshal(s, &user) return user } Note, that, while we want to read/write our Login struct to a file in EDN format, we also want to marshal the struct to JSON when sending the HTTP request. So the metadata on our Login struct needs to be slightly updated: type Login struct { UserName string `edn:"mailaddress" json:"mailaddress"` Password string `edn:"password" json:"password"` } We also need a new struct to read the token returned by the API, and we also want to store it as EDN, just like we are doing for the Login struct: type RefreshToken struct { RefreshToken string `edn:"refreshToken" json:"refreshToken"` } And now, we have all the bricks to write the GetRefreshToken function: func GetRefreshToken() (RefreshToken, error) { // load user stored in file previously and prepare it as json data var user = GetUser() data, err := json.Marshal(user) // prepare the http request, with the url, and the json formatted user as body content url := fmt.Sprintf("%s/token/auth_user", BASE_URL) req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(data)) // send the request client := http.Client{} res, err := client.Do(req) // the API will returns data that will store in a RefreshToken struct var rt RefreshToken json.NewDecoder(res.Body).Decode(&rt) // and let's store that refresh token as an EDN file encoded, err := edn.Marshal(&rt) writeConfigFile(REFRESH_TOKEN_FILE, encoded) return rt, err } Running go test is a little bit more verbose, because we print the refreshToken to the standard output, but the tests should be passing! {eyJjdHkiOiJKV1QiLC...} PASS ok github.com/hellonico/jquants-api-go 3.231s Get the ID Token From the Refresh Token, you can retrieve the IdToken which is the token then used to send requests to the jquants API. This is has almost the same flow as GetRefreshToken, and to support it we mostly introduce a new struct IdToken with the necessary metadata to marshal to/from edn/json. type IdToken struct { IdToken string `edn:"idToken" json:"idToken"` } And the rest of the code this time is: func GetIdToken() (IdToken, error) { var token = ReadRefreshToken() url := fmt.Sprintf("%s/token/auth_refresh?refreshtoken=%s", BASE_URL, token.RefreshToken) req, err := http.NewRequest(http.MethodPost, url, nil) client := http.Client{} res, err := client.Do(req) var rt IdToken json.NewDecoder(res.Body).Decode(&rt) encoded, err := edn.Marshal(&rt) writeConfigFile(ID_TOKEN_FILE, encoded) return rt, err } Get Daily Quotes We come to the core of the wrapper code, where we use the IdToken, and request daily quote out of the jquants HTTP API via a HTTP GET request. The code flow to retrieve the daily quotes is: As before, read ID token from the EDN file Prepare the target URL with parameters code and dates parameters Send the HTTP request using the idToken as a HTTP header Parse the result as a daily quotes struct, which is a slice of Quote structs The test case simply checks on non-nul value returned and prints the quotes for now. func TestDaily(t *testing.T) { var quotes = Daily("86970", "", "20220929", "20221003") if quotes.DailyQuotes == nil { t.Failed() } for _, quote := range quotes.DailyQuotes { fmt.Printf("%s,%f\n", quote.Date, quote.Close) } } Supporting code for the func Daily is shown below: func Daily(code string, date string, from string, to string) DailyQuotes { // read id token idtoken := ReadIdToken() // prepare url with parameters baseUrl := fmt.Sprintf("%s/prices/daily_quotes?code=%s", BASE_URL, code) var url string if from != "" && to != "" { url = fmt.Sprintf("%s&from=%s&to=%s", baseUrl, from, to) } else { url = fmt.Sprintf("%s&date=%s", baseUrl, date) } // send the HTTP request using the idToken res := sendRequest(url, idtoken.IdToken) // parse the result as daily quotes var quotes DailyQuotes err_ := json.NewDecoder(res.Body).Decode(&quotes) Check(err_) return quotes } Now we need to fill in a few blanks: The sendRequest needs a bit more details The parsing of DailyQuotes is actually not so straightforward So, first let’s get the sendRequest func out of the way. It sets a header using http.Header, and note that you can add as many headers as you want there. Then it sends the HTTP GET request and returns the response as-is. func sendRequest(url string, idToken string) *http.Response { req, _ := http.NewRequest(http.MethodGet, url, nil) req.Header = http.Header{ "Authorization": {"Bearer " + idToken}, } client := http.Client{} res, _ := client.Do(req) return res } Now to the parsing of the daily quotes. If you use Goland as your editor, you’ll notice that if you copy-paste a JSON content into your Go file, the editor will ask to convert the JSON to go code directly! Pretty neat. type Quote struct { Code string `json:"Code"` Close float64 `json:"Close"` Date JSONTime `json:"Date"` AdjustmentHigh float64 `json:"AdjustmentHigh"` Volume float64 `json:"Volume"` TurnoverValue float64 `json:"TurnoverValue"` AdjustmentClose float64 `json:"AdjustmentClose"` AdjustmentLow float64 `json:"AdjustmentLow"` Low float64 `json:"Low"` High float64 `json:"High"` Open float64 `json:"Open"` AdjustmentOpen float64 `json:"AdjustmentOpen"` AdjustmentFactor float64 `json:"AdjustmentFactor"` AdjustmentVolume float64 `json:"AdjustmentVolume"` } type DailyQuotes struct { DailyQuotes []Quote `json:"daily_quotes"` } While the defaults are very good, we need to do a bit more tweaking to unmarshal Dates properly. What follows comes from the following post on how to marshal/unmarshal JSON dates. The JSONTime type will store its internal date as a 64bits integer, and we add the functions to JSONTime to marshall/unmarshall JSONTime. As shown, the time value coming from the JSON content can be either a string or an integer. type JSONTime int64 // String converts the unix timestamp into a string func (t JSONTime) String() string { tm := t.Time() return fmt.Sprintf("\"%s\"", tm.Format("2006-01-02")) } // Time returns a `time.Time` representation of this value. func (t JSONTime) Time() time.Time { return time.Unix(int64(t), 0) } // UnmarshalJSON will unmarshal both string and int JSON values func (t *JSONTime) UnmarshalJSON(buf []byte) error { s := bytes.Trim(buf, `"`) aa, _ := time.Parse("20060102", string(s)) *t = JSONTime(aa.Unix()) return nil } The test case written at first now should pass with go test. "2022-09-29",1952.000000 "2022-09-30",1952.500000 "2022-10-03",1946.000000 PASS ok github.com/hellonico/jquants-api-go 1.883s Our helper is now ready and we can adding some CI to it. CircleCI Configuration The configuration is character to character close to the official CircleCI doc on testing with Golang. We will just update the Docker image to 1.17. version: 2.1 jobs: build: working_directory: ~/repo docker: - image: cimg/go:1.17.9 steps: - checkout - restore_cache: keys: - go-mod-v4-{{ checksum "go.sum" } - run: name: Install Dependencies command: go get ./... - save_cache: key: go-mod-v4-{{ checksum "go.sum" } paths: - "/go/pkg/mod" - run: go test -v Now we are ready to set up the project on CircleCI: The required parameters USERNAME and PASSWORD in our helper_test.go can be set up directly from the Environment Variables settings of the CircleCI project: Any commit on the main branch will trigger the CircleCI build (or you can manually trigger it of course) and if you’re all good, you should see the success steps: Our wrapper is well-tested. Let’s start publishing it. Publishing the Library on GitHub Providing our go.mod file has the content below: module github.com/hellonico/jquants-api-go go 1.17 require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3 The best way to publish the code is to use git tags. So let’s create a git tag and push it to GitHub with: git tag v0.6.0 git push --tags Now, a separate project can depend on our library by using it in their go.mod. require github.com/hellonico/jquants-api-go v0.6.0 Using the Library from an External Program Our simplistic program will parse parameters using the flag module, and then call the different functions just like it was done in the test cases for our wrapper. package main import ( "flag" "fmt" jquants "github.com/hellonico/jquants-api-go" ) func main() { code := flag.String("code", "86970", "Company Code") date := flag.String("date", "20220930", "Date of the quote") from := flag.String("from", "", "Start Date for date range") to := flag.String("to", "", "End Date for date range") refreshToken := flag.Bool("refresh", false, "refresh RefreshToken") refreshId := flag.Bool("id", false, "refresh IdToken") flag.Parse() if *refreshToken { jquants.GetRefreshToken() } if *refreshId { jquants.GetIdToken() } var quotes = jquants.Daily(*code, *date, *from, *to) fmt.Printf("[%d] Daily Quotes for %s \n", len(quotes.DailyQuotes), *code) for _, quote := range quotes.DailyQuotes { fmt.Printf("%s,%f\n", quote.Date, quote.Close) } } We can create our CLI using go build. go build And the run it with the wanted parameters here: Refreshing the ID token Refreshing the refresh token Getting daily values for entity with code 86970 between 20221005 and 20221010 ./jquants-example --id --refresh --from=20221005 --to=20221010 --code=86970 Code: 86970 and Date: 20220930 [From: 20221005 To: 20221010] [3] Daily Quotes for 86970 "2022-10-05",2016.500000 "2022-10-06",2029.000000 "2022-10-07",1992.500000 Nice work. We will leave it to the user to write the remaining statements and listedInfo that are part of the JQuants API but not yet implemented in this wrapper.

OAuth is often employed in processes requiring permissions to be granted to front-end applications and end users. Yet what we typically need in API systems integrations is a way to secure connections between the integration middleware and backend systems without a need for any ongoing human interactions. OAuth can be a good choice for that scenario. This article shows how it can be achieved in Python with backend systems using REST and HL7 FHIR. What We Would Like To Have Let’s say we have a typical integration scenario as in the diagram below: External systems and applications invoke the interoperability layer (Zato) which is expected to further invoke a few backend systems (e.g., a REST and HL7 FHIR one) so as to return a combined result of backend API invocations. It does not matter what technology the client systems use, i.e., whether they are REST ones or not. The interoperability layer needs to identify itself with the backend systems before it is allowed to invoke them. They need to make sure that it really is Zato and that it accesses only the resources allowed. An OAuth server issues time-based access tokens, which are simple strings, like web browser session cookies, confirming that such and such bearer of the said token is allowed to make such and such requests. Note that the tokens have an explicit expiration time; e.g., they will become invalid after one hour. Also observe that Zato stores the tokens as-is: they are genuinely opaque strings. If a client system invokes the interoperability layer, the layer will obtain a token from the OAuth server and keep it in an internal cache. Next, Zato will invoke the backend systems, bearing the token among other HTTP headers. Each invoked backend system will extract the token from the incoming request and validate it. How the validation looks in practice is something that Zato will not be aware of because it treats the token as an opaque string. However, in practice, if the token is self-contained (e.g., JWT data) the system may validate it on its own, and if it is not self-contained, the system may invoke an introspection endpoint on the OAuth server to validate the access token from Zato. Once the validation succeeds, the backend system will reply with the business data and the interoperability layer will combine the results for the calling application’s benefit. In subsequent requests, the same access token will be reused by Zato with the same flow of messages as previously. However, if the cached token expires, Zato will request a new one from the OAuth server - this will be transparent to the calling application - and the flow will resume. In OAuth terminology, what is described above has specific names, the overall flow of messages between Zato and the OAuth server is called a “Client Credential Flow” and Zato is then considered a “client” from the OAuth server’s perspective. How To Do It Configuring OAuth First, we need to create an OAuth security definition that contains the OAuth server’s connection details. In this case, the server is Okta. Note the scopes field: it is a list of permissions (“scopes”) that Zato will be able to make use of. What exactly the list of scopes should look like is something to be coordinated with the people who are responsible for the configuration of the OAuth server. If it is you personally, simply ensure that what is in the OAuth server and in Zato is in sync. Calling REST To invoke REST services, fill out a form as below, pointing the “Security” field to the newly created OAuth definition. This suffices for Zato to understand when and how to obtain new tokens from the underlying OAuth server. Here is a sample code to invoke a backend REST system. Note that we merely refer to a connection by its name, without having to think about security at all. It is Zato that knows how to get and use OAuth tokens as required. # -*- coding: utf-8 -*- # Zato from zato.server.service import Service class GetClientBillingPlan(Service): """ Returns a billing plan for the input client. """ def handle(self): # In a real service, this would be read from input payload = {'client_id': 123} # Get a connection to the server .. conn = self.out.rest['REST Server'].conn # .. invoke it .. response = conn.get(self.cid, payload) # .. and handle the response here. ... Calling HL7 FHIR Similarly to REST endpoints, to invoke HL7 FHIR servers, fill out a form as below and let the “Security” field point to the OAuth definition just created. This will suffice for Zato to know when and how to use tokens received from the underlying OAuth server. Here is a sample code to invoke an FHIR server system. As with REST servers above, observe that we only refer to a connection by its name and Zato takes care of OAuth. # -*- coding: utf-8 -*- # Zato from zato.server.service import Service class GetPractitioner(Service): """ Returns a practictioner matching input data. """ def handle(self) -> 'None': # Connection to use conn_name = 'My EHR' # In a real service, this would be read from input practitioner_id = 456 # Get a connection to the server .. with self.out.hl7.fhir[conn_name].conn.client() as client: # Get a reference to a FHIR resource .. practitioners = client.resources('Practitioner') # .. look up the practitioner .. result = practitioners.search(active=True, _id=practitioner_id).get() # .. and handle the response here. ... What About the API Clients? One aspect omitted above is the initial API clients. This is on purpose. How they invoke Zato, using what protocols, with what security mechanisms, and how to build responses based on their input data, is completely independent of how Zato uses OAuth in its own communication with backend systems. All of these aspects can and will be independent in practice; e.g., clients will use Basic Auth rather than OAuth. Or perhaps the clients will use AMQP, Odoo, SAP, or IBM MQ, without any HTTP, or maybe there will be no explicit API invocations, and what we call “clients” will be actually CSV files in a shared directory that your services will be scheduled to periodically pick up. Yet, once more, regardless of what makes the input data available, the backend OAuth mechanism will work independently of it all.

Reusing common functionalities is very important in any coding language to avoid redundancy. In MuleSoft, we will also come across situations where we will have to reuse common functionalities, for example, error handling. In this article, I will explain how to reuse common flows in your implementation flow. There are two ways of doing it: Create Maven dependencies in the Maven repository and reuse them in the main flow. Create the JAR file and use it in the main flow. Create Maven Dependencies in the Maven Repository and Reuse Them in the Main Flow Create the common Mule Project in Studio that you are planning to reuse. In Studio, under Package Explorer, right click -> click New - > select "Mule Project." Enter the Project Name (in my case, common-lib-prj) and click Finish. Create two flows in it with a logger inside the flow as below. Open the pom.xml file and modify the Mule Maven Plugin by adding the classifier as "mule-plugin" as below. Open the command prompt and go to the workspace project directory of common-lib-prj. Enter the command "mvn clean install." By doing this, common-lib-prj will be created as a plugin project, and it will be available to use as a Maven dependency once the build is a success. Create the new Mule project (in my example, I am creating sample-main-prj) in which you will be using this common-lib-prj. Create a sample flow in the Mule configuration file as below. Drag the HTTP listener from the palette, add the default connector configuration with port 8081 and path as /api/*, and then add one Logger and Flow Reference Component. Copy groupId, artifactId, and version from pom.xml file from common-lib-prj as below. Add the above-copied properties and also the classifier into the pom.xml file in sample-main-prj as a dependency. Once you save the pom.xml file, you will see common-lib-prj getting added in to sample-main-prj in the Package Explorer. Go to sample-main-prj Mule configuration file and click on the tab Global elements to add Import as Global configuration. Add the common-lib-prj.xml Mule configuration file name as an Import Global Configuration as below. Now you should be able to see both flows which we built under common-lib-prj in the Flow Reference drop-down menu in sample-main-prj as below. Now deploy the flow sample-main-prj by running the project. Open Postman or Advanced REST Client and send the request as below. Check the Studio Console output and you should see the logger getting printed from common-lib-prj as below. Create the JAR File and Use It in the Main Flow Create the common Mule Project in Studio that you are planning to reuse. In Studio, under Package Explorer, right click -> click new - > select "Mule Project." Enter the Project Name (in my case, common-lib-prj) and click Finish. Create two flows in it with a logger inside the flow as below. Deploy the common-lib-prj by running the project. Once the flow is successfully deployed, just export as a Mule Project. Create a new Project called implementation-prj where you will be using common-lib-prj flows. Right-click on implementation-prj and add common-lib-prj.jar into the implementation project as a Maven dependency. In the next pop-up window, click on "Install a local dependency," browse the common-lib-prj.jar file and install it. Then click Finish. The JAR file will be successfully added to the local repository. You should be able to see the dependency of common-lib-prj in the pom.xml file of implementation-prj as below. And also you should be able to see common-lib-prj added under "Project Libraries." Create the flow in implementation-prj as below with the default HTTP listener configuration, logger, and Flow Reference. Go to Global Elements and import the common-lib-prj Mule Configuration file. With this, we can reference any common flows and configurations which we can utilize or access as shared resources. We can import multiple files. Now you should be able to see both the flows which we built under common-lib-prj in the Flow Reference drop-down menu in implementation-prj as below. Now deploy the flow implementation-prj by running the project. Open Postman or Advanced REST Client and send the request as below. Check the Studio Console output and you should see the logger getting printed from common-lib-prj as below. You can deploy these two flows in CloudHub and see the result.

This post is about a repository I've shared on GitHub at dalelane/app-connect-tekton-pipeline. It contains an example of how to use Tekton to create a CI/CD pipeline that builds and deploys an App Connect Enterprise application to Red Hat OpenShift. The pipeline uses the IBM App Connect Operator to easily build, deploy and manage your applications in containers. The pipeline runs on OpenShift to allow it to easily be integrated into an automated continuous delivery workflow without needing to build anything locally from a developer's workstation. For background information about the Operator, and the different types of Kubernetes resources that this pipeline will create (e.g. IntegrationServer and Configuration), see these blog posts: What is an Operator and why did we create one for IBM App Connect? Exploring the IntegrationServer Resource of the IBM App Connect Operator Pipeline The pipeline builds and deploys your App Connect Enterprise application. You need to run this every time your application has changed and you want to deploy the new version to OpenShift. When running App Connect Enterprise in containers, there is a lot of flexibility about how much of your application is built into your container image, and how much is provided when the container starts. For background reading on some of the options, and some of the considerations about them, see the blog post: Comparing styles of container-based deployment for IBM App Connect Enterprise.This pipeline provides almost all parts of your application at runtime when the container starts. The only component that is baked into the image is the application BAR file. Baking the BAR files into custom App Connect images prevents the need to run a dedicated content server to host BAR files, however, if you would prefer to do that see the documentation on Mechanisms for providing BAR files to an integration server for more details on how to do this. (The pipelines in the repository use the approach described as "Custom image" in that documentation.) Running the Pipeline pipeline spec: pipeline.yaml example pipeline runs: simple-pipelinerun.yaml complex-pipelinerun.yaml helper scripts: 1-deploy-simple-integration-server.sh 1-deploy-complex-integration-server.sh What the Pipeline Does Builds your IBM App Connect Enterprise application and deploys it to the OpenShift cluster. Outcome From Running the Pipeline A new version of your application is deployed with zero-downtime - replacing any existing version of the app once it is ready. Background As discussed above, most of your application configuration will be provided to your application container at runtime by the Operator using Configuration resources. As shown in the screenshot above, this example pipeline currently supports many, but not all, of the types of Configuration resource: Loopback data source type Policy project type setdbparms.txt type server.conf.yaml type Truststore type For more information about the other Configuration types, see the documentation on Configuration types for integration servers. Adding support for any of these additional types would involve adding additional tasks to the tasks provided in the repo - the existing tasks are commented to help assist with this. Each of these configuration resources is individually optional. Two example App Connect applications are provided to show how the pipeline supports different application types. Simple Stand-Alone Applications The pipeline can be used to deploy a stand-alone application with no configuration dependencies. sample application simple-demo pipeline run config simple-pipelinerun.yaml demo script: 1-deploy-simple-integration-server.sh This is a simple App Connect application with no external configuration. When deploying this, the pipeline skips all of the Configuration tasks: Watching the pipeline run looks like this (except it takes longer). Complex Applications The pipeline can be used to deploy complex applications with multiple configuration dependencies and support Java projects. sample application sample-ace-application pipeline run config complex-pipelinerun.yaml demo script: 1-deploy-complex-integration-server.sh This is an example of an App Connect application that needs configuration for connecting to: a PostgreSQL database an external HTTP API an Apache Kafka cluster When deploying this, the pipeline runs all of the Configuration tasks required for this application: Watching the pipeline run (also sped up!) it looks like this. To avoid needing to store credentials in Git with your application code, the pipeline retrieves credentials from Kubernetes secrets. When configuring the pipeline for your application (see the section below) you need to specify the secrets it should use to do this. Sample Apps I've put notes on how I set up the sample apps to demonstrate the pipeline in demo-pre-reqs/README.md however, neither of the sample apps is particularly useful and was purely used to test and demo the pipeline. You can import them into App Connect Toolkit to edit them if you want to by: File -> Import... -> Projects from Folder or Archive Put the location of the ace-projects folder as the Import source. Tick all of the projects That will let you open the projects and work on them locally. If you're curious what they do, I'll include some brief notes below: Simple App It provides an HTTP endpoint that returns a Hello World message. Running this: Shell curl http://$(oc get route -nace-demo hello-world-http -o jsonpath='{.spec.host}')/hello" returns this: JSON { "hello" : "world" } Complex App It provides an intentionally contrived event-driven flow that: "Kafka consumer todo updates" receives a JSON message from a Kafka topic "get id from update message" parses the JSON message and extracts an ID number from it uses the id number to create an HTTP URL for an external API "retrieve current todo details" makes an HTTP GET call to the external API "base64 encode the description" transforms the response from the external API using a custom Java class "insert into database" inserts the transformed response payload into a PostgreSQL database The aim of this application was to demonstrate an ACE application that needed a variety of Configuration resources. But it means that running this: echo '{"id": 1, "message": "quick test"}' | kafka-console-producer.sh \ --bootstrap-server $BOOTSTRAP \ --topic TODO.UPDATES \ --producer-property "security.protocol=SASL_SSL" \ --producer-property "sasl.mechanism=SCRAM-SHA-512" \ --producer-property "sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="appconnect-kafka-user" password="$PASSWORD";" \ --producer-property "ssl.truststore.location=ca.p12" \ --producer-property "ssl.truststore.type=PKCS12" \ --producer-property "ssl.truststore.password=$CA_PASSWORD" gets you this: store=# select * from todos; id | user_id | title | encoded_title | is_completed ----+---------+--------------------+--------------------------------------+-------------- 1 | 1 | delectus aut autem | RU5DT0RFRDogZGVsZWN0dXMgYXV0IGF1dGVt | f (1 row) Configuring the Pipeline for Your App Connect Enterprise Application To run the pipeline for your own application, you need to first create a PipelineRun. The sample pipeline runs described above provide a good starting point for this, which you can modify to your own needs. You need to specify the location of your App Connect Enterprise application code and configuration resources. All of the available parameters are documented in the pipeline spec if further guidance is needed. Alternative Approaches Running App Connect Enterprise in containers is ideally suited to a variety of CI/CD approaches. The pipeline described in this post was useful for a project I recently worked on, but you can find a variety of other pipeline approaches for managing your ACE application. For another great option, see github.com/ot4i/ace-demo-pipeline.

There’s no denying that Jira is one of the best project management solutions for software development. The platform is feature-rich and exceptionally flexible when it comes to bending it to meet your needs through custom scripts or integrations. That said, there’s always room for improvement, and the Atlassian Marketplace is one of the best places to start your Jira transformation. Workflow Management Add-Ons These add-ons may not directly impact code quality, but they are excellent quality-of-life improvements. Look into them if you are missing some clarity and transparency in your workflows. Smart Checklist for Jira PRO Abandoned tasks and subtasks will become a problem sooner or later. The lack of visibility and awareness of who’s doing what at the moment is quite irritating. The absence of clearly described templates for routine tasks burdens everybody on the team.Smart Checklist for Jira is an elegant tool for dealing with these challenges. The add-on allows you to use a markdown editor to create context-rich checklists or templates for common issue types (DoD, Acceptance Criteria, etc.).Each checklist item can have a drop-down with more info, deadline, and assignees, making sure that all of your sub-tasks are collected neatly in the parent ticket. The progress bar will allow you to see how close the team is to close the ticket from both the issue and the board views.Review score: 3.5Price: There's a 30-day free trial. After that, the price starts at $0.50 per user per month. Jira MISC Workflow Extensions This add-on brings in a set of custom conditions, post functions, and validators that allow you to go beyond vanilla Jira functionality when it comes to workflow automation. Built-in syntax checking is a great asset for building the functionality you need from scratch. It is intuitive enough that you don’t have to be a coder.Review score: 3.5Price: The app is free for teams of up to 10 users. The price starts at $1.31 per user per month for larger teams. Tempo Planner - Resource and Capacity Planning This add-on is a good asset for planning your team’s time and workload. You can easily see the current status and progress and rely on custom reports to make data-driven plans or estimations.Review score: 3Price: The pricing starts at $1 per user per month. Deployment Add-Ons By definition, deployment is the process of bringing resources into effective action, and within the context of software delivery, these add-ons follow the definition to a T. GitHub for Jira This add-on is a GitHub to Jira integration that gives developers a bit of breathing space as they can focus on delivering value rather than managing reports. The integration allows you to get pull requests, deployments, branches, builds, and commits straight into Jira. Additionally, you'll have some nice DevOps tools to check performance insights, such as deployment frequency or cycle time. That said, please note that the add-on is by no means perfect, and you’ll have to troubleshoot a bit from time to time.Review score: 2.5Price: Free Jenkins for Jira (Official) The add-on allows users to send, build and deploy data from Jenkins into Jira automatically. Some of the handier features of the add-on are the ability to see all Jenkins deployments on a timeline and the ability to confirm when features have been shipped.Review score: 4Price: Free Octopus Deploy for Jira As the DevOps tooling and deployment automation leader, Octopus offers a Jira integration that greatly improves visibility across the board. You’ll be able to see which issues contributed to the current release making the generation of release notes more fluid and intelligent. Plus, there’s no longer a need to jump between the apps.Review score: 2.5Price: Free Azure DevOps Integration for Jira This integration allows you to easily link commits from Azure DevOps to Jira Issue Ids, record time logs, assign commits to users, and transition Jira issues. Pull requests, and commits are easily visible, giving you the data to manage the team’s productivity more efficiently.Review score: 3.5Price: Starting at $0.80 per user per month. Code Quality Add-Ons Code quality add-ons are designed to make life easier for developers and QA engineers. QMetry Test Management for Jira This add-on allows you to import-export test cases and supports automated authoring. Test case templates are also a great feature to have, as they remove the necessity of redoing your work. As for the Jira side of things, the addon offers cross-project traceability between stories, test cases, executions, and defects.Review score: 4Price: Starting at $1 per user per month. Cucumber for Jira This add-on helps you create custom documentation in order to push the test results from Cucumber or your CI tool of choice into Jira. Jira issues are linked to corresponding Git files, allowing you to create issues directly from Cucumber. Lastly, the embedded Gherkin editor gives your team more room to collaborate without swapping apps.Review score: 3Price: Starting at $0.53 per user per month. Build Management Add-Ons These add-ons are designed to help with assembling the work you’ve done into a viable software release. Version Manager for Jira This add-on offers a higher level of collaboration for development teams. People who are not Jira/Project admins can order, create, edit, and delete new versions of projects.Review score: 3.5Price: There is a free 30-day trial for teams of all sizes. After that, the prices start at $10 per 10 users per year. REST API Browser for Jira This dev tool helps discover REST APIs available for Atlassian products such as Jira, Confluence, and Trello. The interface will offer easy access to APIs and descriptions of methods and endpoints.Review score: 4Price: Starting at $1 per user per month. Did I miss anything? Feel free to mention the add-ons and integrations you use to improve your software delivery processes in the comments section.

The serverless offerings of AWS (or any other public cloud provider today) are getting more and more popular. One of the biggest reasons is of course not needing to think about infra (server provisioning and so on). The other reasons include the offerings themselves, which at times fit well into the technical need for finding solutions to the given problems. But on the other side, it remains a challenge to know the offerings in detail to leverage them properly. There could be always some new feature hidden deep inside the bulk of the documentation. Background A retail company decided to leverage the AWS Serverless offering for modernizing their on-premises integrations for order management business process. We can go serverless in the cloud in various ways, but the initial solutioning was decided to leverage the Step Function and Lambda Function along with some more integration offerings from AWS, namely SNS, SQS, and EventBridge. The coding was done, so the unit testing and system integration testing followed by the UAT. Everything looked fine. It went live and started handling the transactions quite well. And then one day, there was an unplanned AWS outage spanning almost a whole business day. What could happen next? Let’s see. Challenge With the Initial Solution The outage brought on a disastrous situation. While the integration was designed to get triggered by an EventBridge schedule, it was the Lambda Function which would pull the inbound payloads from the SQS which would in turn subscribe to the SNS topic where the order management system would publish the payloads when an order would be placed from the portal, due to the outage, the inbound payloads got piled up in the SQS. Not only that, but the messages which were ingested into the database got piled up unprocessed. And then the real challenge began once the outage was restored. The flows started progressing, but the rate of the payloads getting processed were too low to cope with the business need. That low rate being allowed, the whole set of pending transactions would take months to complete, which was obviously not rationally acceptable. So, the need for the moment became to enhance the solution in a way that scales up when needed to reduce the backlogs. Improved Solution The current design had three Lambda Functions — the first one would pull the payloads from the SQS and insert it into the database, the second one would parse those and insert the values as records in another table in the database, the third one would process the records, i.e., invoke Oracle ERP APIs (the company is using Oracle ERP as matter of their choice) to create, update, or delete the purchase orders there. The Step Function would be triggered by the EventBridge rule. At first, the Lambda 1 would be executed inserting the pulled payloads into the table. The Lambda 2 would then iterate over the unparsed payloads in the database to parse each of the payloads and insert the respective records into the database. Lambda 3 would then iterate over the unprocessed records in the table and process those as required by invoking APIs and so on. The catch is: The queries used to select the unparsed or unprocessed records from the database would be limited by the maximum number of records those could fetch based on how much time the respective Lambda Function would take to process those. This is required because Lambda Function in AWS can run only up to 15 minutes, beyond which it will time out and stop its execution abruptly. That’s fair enough from the AWS offering perspective, because Lambda Function is not intended to be used for long-running jobs. Also, it's fair to raise a question on the initial design decision to use Lambda Function for this integration. A better solution could be leveraging an EC2 instance or a containerized solution leveraging AWS Fargate, if staying serverless was one of the key requirements. But here, we would focus only on the capability of the Step Function to scale up to handle the given situation. The same feature of the Step Function can be leveraged in other scenarios that might appear when modernizing integrations, especially when there is a need for orchestration or workflow and when we prefer to go serverless. AWS Step Function comes with some features that help achieving parallel execution. The State Machine of a Step Function is basically a collection of states and there are some defined state types which must be mentioned for each state. The state type that is commonly used to trigger a Lambda Function is "Task." As the name explains, the Lambda Function is supposed to perform some tasks. Out of the remaining state types, two which are relevant to this topic are "Map" and "Parallel." Well, the answer to this problem is Map State, but why am I mentioning the "Parallel?" There is a thin line of difference between them, but the names can really be misleading at times. It is worth explaining the difference in this context. To explain in simple words — a Parallel State is for performing two or more set of tasks in parallel but on the same input. A Map State is for performing a same set of tasks concurrently on each item of a given list of inputs. The below diagrams can be helpful to understand it better. The above diagram may make it clear why Map State is the desired option in this case. To explain the improved solution more specifically in detail: The Lambda 2 and 3 need to be inside two Map States respective to each of them. A new Lambda 0 needs to be incorporated which would first prepare the input arrays of the Lambda 2 and 3. It would do that by querying the database tables to fetch the unparsed and unprocessed records. The Lambda 1 will not require to be in a Map State, as that would still pull the messages from the SQS and insert into the table. In every execution of the State Machine, the Lambda 0 would prepare the input arrays based on what was inserted into the tables by the Lambda 1 and the Lambda 2 in the previous execution of the State Machine which would respectively become the input for the Lambda 2 and the Lambda 3 in the current execution of the State Machine. The input array of Lambda 2 would be a list of the list of unique identifiers of the inserted payloads. The input array of Lambda 3 would be a list of the list of unique identifiers of the parsed records. Step Function requires the input array to be set at the field ItemsPath (this is specific to the Map State, i.e., “Type”: “Map”. It would be set from the Lambda Function code of the Lambda 0 and would be propagated through the Step Function. The value of the field MaxConcurrency would be set as 0 (the default value so also the same) which means there is no quota on parallelism and the State Machine would execute as much in parallel as possible. However, the architect/development team would decide the sizes of the inner and outer lists of the input arrays which would respectively mean the number of payloads/records that the Lambda Function would be allowed to process per instance of execution and the number of instances of the execution (in other words, the number of iterations) which in turn would be the limit of parallelism as well. The database would be queried by the Lambda 0 according to these configurable numbers (Lambda Function’s environment variable could be leveraged). After this solution was implemented, the backlog took around 24 hours to get cleared. However, it was not an all-green situation. The backlog was cleared but the Operations Team observed further challenges over time which were even more deep-rooted to the fundamental design decisions of this solution. Further Challenges Like it happens in any standard retail order management business process, the high-level steps of the order journey remain, like Purchase Order (PO) Creation > Receipt of the PO > Invoice Generation of the PO > Return/Replacement Processing (if any). There were integrations for each of these modules which were modernized following the same design pattern. That said, the challenges observed over the time were as below. Time Lag Issue – Many times, it was observed that the PO for which the integration tried to generate the invoice or receipt was not even created in the Oracle ERP at that point of time. It could be due to various reasons, but typically, the observation was depending on the transactional volume. Some PO creation and the respective receipt creation would get delayed, and by the time those were processed, the integration would get the invoice generation request. This is mainly because the source application responsible for the invoice generation is agnostic about the PO creation status. A possible solution could be usage of SNS FIFO topic with SQS FIFO queue to achieve the fan-out pattern across all these integrations to ensure the sequence of events are processed in the same sequence as generated. But there could be many other ways of solving this issue (e.g., publishing the PO creation status to an SNS topic and have all the involved systems subscribed to that topic to sync it across those systems and so on) subject to the technical feasibilities and dependencies. Intermittent Lack of Parallelism Issue – Despite using the Map State in the Step Function, it was observed that sometimes some iterations would not start until the previous ones are completed. This issue increases especially when the input array size is greater than 40. This is a known limitation as mentioned in the AWS documentation. There could be two possible ways to resolve this. The concurrency limit could be restricted such that it does not exceed 40. However, that might again impact or limit the desired performance of the integration. In order to achieve higher concurrency, the State Machines could be nested that cascades the Map States. In this way, the Step Function code would become more complex but higher concurrency can be achieved. Takeaway Modernization of integrations using Lambda Functions alone may not always be a favorable option, especially when the integration has huge transformation logic and a high volume of transactional requirements. It may shoot up the cost too high in those cases, or sometimes fail miserably to scale up. However, some of the Step Function features (when used to orchestrate the tasks defined in Lambda Functions) may be helpful to come up with decent and flexible solutions in terms of scalability. Sample Code of Step Function The below is sample code with similar orchestration logic as described in the improved solution for this case study. This is not the exact solution, but just an indicative sample. JSON { "Comment": "A description of my state machine", "StartAt": "Process ID Concurrently", "States": { "Process ID Concurrently": { "Type": "Map", "ItemsPath": "$.id", "Iterator": { "StartAt": "Perform ID Task", "States": { "Perform ID Task": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:495125011821:function:ProcessUniqueID:$LATEST", "End": true } } }, "ResultPath": null, "Next": "Process Name Concurrently" }, "Process Name Concurrently": { "Type": "Map", "ItemsPath": "$.name", "Iterator": { "StartAt": "Perform Name Task", "States": { "Perform Name Task": { "Type": "Task", "Resource": "arn:aws:lambda:us-east-1:495125011821:function:ProcessUniqueName:$LATEST", "End": true } } }, "End": true } } } Sample Execution The above sample Step Function, when executed with an input as seen in the screenshot below, identifies the iterations for each Map State and executes them concurrently. The below timestamps show that each of the three iterations of the first Map State started exactly at the same time, ensuring concurrency. The below timestamps show that each of the two iterations of the second Map State started exactly at the same time, ensuring concurrency. Also, it is noticeable that this state started after the first Map State was completed, i.e., two separate Map States are not concurrently executed. The below screenshot shows the three iterations for the first Map State have taken each item in the outer list of the JSON input as their individual inputs. The below screenshot shows the two iterations for the second Map State have taken each item in the outer list of the JSON input as their individual inputs. The below screenshot shows the CouldWatch Log for the Lambda (ProcessUniqueID) which is invoked from the first Map State to perform the ID Task. The below screenshot shows the CouldWatch Log for the Lambda (ProcessUniqueName) which is invoked from the second Map State to perform the Name Task.