DZone Spotlight

If you’re a Solidity developer, you’ll be excited to hear that Truffle now supports console logging in Solidity smart contracts. While Truffle has long been a leader in smart contract development tooling—providing an easy-to-use environment for creating, testing, and debugging smart contracts—a directly integrated console.log was a feature it still needed. But no more! Developers can now easily log messages and debug their smart contracts, all within the familiar Truffle (Ganache) environment. Let’s look at how. What Is Console.log? Console.log is a very popular feature in JavaScript and is widely used by developers to easily output logging messages and extract details directly from code. In the context of Web3 and smart contract development, console.log plays a similar role, allowing developers to print out Solidity variables and other information from their smart contracts. For example, you can use console.log to display the value of a variable or the output of a function call within your smart contract. This can be extremely useful when debugging or testing your smart contract. console.log("Console Logging: The Smart Contract Developer's Best Friend"); How To Use Console Logging in Truffle Making use of console.log is quite straightforward. First, you’ll have to ensure you have an up-to-date Truffle version running on your computer. If you have any issues, you might want to uninstall the package entirely and then reinstall it. For the commands used in this post, we’ll use NPM as our package manager. $ npm install -g truffle After a successful installation, I suggest that you modify the truffle configuration file (i.e. truffle-config.js) as follows: module.exports = { . . . solidityLog: { displayPrefix: ' :', // defaults to "" preventConsoleLogMigration: true, // defaults to false } displayPrefix: decorates the outputs from console.log to differentiate it from other contents displayed by the CLI. preventConsoleLogMigration: screens contract deployments from going through when on a test or mainnet. You can opt out of this if you wish to deploy your contract with the console.log included. However, if you choose to do this, keep in mind that console.log has unpredictable behavior when it comes to gas usage. Now you’re ready to try it out! Import the contract.sol contract into your Solidity code as usual. Now you’re ready to use the console.log() command as you would in JavaScript. This includes using string substitutions like %s and %f. pragma solidity ^0.8.9; import "truffle/console.sol"; contract BookStore { //... function transfer(address to, uint256 amount) external { console.log("Transferring %s tokens to %s", amount, to); require(balances[msg.sender] >= amount, "Not enough tokens"); balances[msg.sender] -= amount; balances[to] += amount; emit Transfer(amount, to, msg.sender); } } The above transfer function shows console.log in action. Imagine a call to the transfer function failing with the Not enough tokens error. The console.log line, in this case, will show the number of tokens the call is trying to transfer. This allows the developer to see the address and amount of tokens being transferred. The message will look like this. ... Transferring 10 tokens to 0x377bbcae5327695b32a1784e0e13bedc8e078c9c An even better way to debug this could be to add in the balances[msg.sender] to the console.log statement or print it out on a separate line. That way, the sender’s balance is visible in the console, too. You get the point! You can also leave logs in test and mainnets; this way, you’ll have a nice way to observe your smart contract. And it's worth mentioning that tools like Tenderly will integrate the scrapping of logs, which can be useful when debugging and testing smart contracts in a production environment. Finally, when using console logging, it's important to follow all the good usage rules you already know, such as using clear and descriptive log messages. This will make it easier to understand the output and identify any issues that may arise. Other Debugging Tools in Truffle While console logging is a powerful tool for debugging smart contracts, keep in mind that Truffle offers other debugging tools as well. Truffle has a powerful built-in debugger CLI tool that can be used to step through the execution of a smart contract and inspect the state of variables at different points in the execution. Additionally, events are a nice way to log messages and track the behavior of a smart contract. That said, it's worth noting that using the debugger for something as simple as variable output can be overkill. Similarly, event logging only works when the transaction succeeds, which can be a limitation in certain situations. The bottom line is that the console.log feature—in combination with the other debugging tools in Truffle—can provide a better developer experience thanks to its simplicity and ease of use. It gives developers the ability to quickly and easily log messages and monitor the behavior of their smart contracts, while the other debugging tools can be used for more advanced debugging and troubleshooting. Try It Out Truffle's new console logging feature is a valuable addition to smart contract development. It’s easy to use and can streamline the debugging and testing process. The ability to log messages and track the behavior of smart contracts in real-time can reduce inefficiencies and headaches. It’s a great tool to have in your toolbox.

In a Kubernetes context, multi-tenancy refers to sharing one large cluster among multiple teams, applications, or users primarily in lower environments such as development and testing—mainly to reduce cost and operational overhead around managing many clusters. Multi-tenancy is becoming an essential requirement for platform teams deploying Kubernetes infrastructure. Achieving Multi-Tenancy Using Kubernetes Namespaces By far, the most popular approach to achieve multi-tenancy is by using Kubernetes namespaces. Kubernetes namespaces provide an easy way to divide a set of resources, such as pods, services, and deployments, which are only accessible within that namespace. Platform teams usually manage and operate the clusters and have full cluster-level permissions. They accomplish multi-tenancy by creating one or more namespaces specific to each team/application/user and restricting access to those namespaces for end users who are developers, DevOps engineers, and application owners. End users can only perform operations that are specific to the namespaces they have ownership of. This works well for the vast majority of multi-tenancy use cases; however, there is one corner use case when the vcluster framework can be useful. Multi-Tenancy Using the Vcluster Open-Source Framework Restricting end users’ access only to namespaces does not work in some corner cases where the end users need access to cluster-scope objects such as CRDs, ingress controllers, cluster API servers, etc., for their day-to-day development work. Typically, cluster-level access is needed for end users who are involved in developing custom resources and custom controllers to extend Kubernetes API, admission controllers to implement mutating and validating webhooks, and other services that might require custom Kubernetes configuration. Virtual Cluster (Vcluster) is an open-source framework that aims to solve this problem. Vcluster is essentially a virtual cluster that can be created in a physical cluster. Vcluster installs a K3s cluster by default (k0s, k8s, and EKS optionally) in a namespace of the host cluster for every vcluster instance and installs the core Kubernetes components such as API server, controller manager, storage backend, and optionally, a scheduler. End users interact with the virtual cluster API server and get full access to the virtual cluster, still maintaining resource isolation and security as they are restricted to the host namespaces and don’t have access to the host cluster API server. Platform teams create a namespace in the host cluster, configure resource quotas and policies for the host namespace, create a vcluster instance, and hand over the virtual cluster to the end users. Key Questions to Answer Before Deploying Vclusters While deploying vcluster for a small group of end users is fairly simple, platform teams must ask themselves the following questions and implement additional solutions around vcluster to meet their organization’s automation, security, governance, and compliance requirements before making large-scale vcluster deployments: How do you create host-level namespaces and resource quotas for each namespace and map it with internal users/teams? Platform teams still need a solution for namespace-level multi-tenancy, as host namespaces must be created first for deploying vcluster instances. How do you automate the lifecycle management of vcluster for large-scale usage? Platform teams need to solve for things like creation/modification/deletion of vcluster instances, exposing the vcluster API server to end users (using ingress or load balancers), securely distributing vcluster Kubeconfig files to end users, and upgrading vcluster instances (K3s) for software updates and security vulnerabilities. How do you ensure only approved and mandated cluster-wide services are running in each vcluster? Do you deploy them in every vcluster? How do you guarantee there is no drift? These services typically include security plugins, logging, monitoring, service mesh, ingress controllers, storage plugins, etc. How do you create network policies at the namespace level for the host namespaces? This level of network isolation is still needed as the physical cluster may be shared among multiple BUs and application teams—requiring network isolation among host namespaces. How do you enforce security policies such as OPA in each vcluster? Do you deploy these policies in every vcluster? Most of the platform teams standardize a set of security policies based on the recommendations from their security teams and deploy them in every cluster to maintain security posture even in lower environments. How do you retrieve Kubectl audit logs for each vcluster? For some organizations, Kubectl audit is a key requirement, even in lower environments. How do you handle cost allocation? Since the resources are shared by different development teams, they may belong to different cost centers and platform teams need to implement the appropriate chargeback policies for cost allocation. How do you make other developer tools like ArgoCD work with vcluster? GitOps tools like ArgoCD require cluster access for application deployment, so each vcluster instance must be configured in the ArgoCD for end users to leverage ArgoCD GitOps deployment. This may apply to other common tools such as observability, logging, and monitoring tools. What kind of additional resource and operational overhead do you incur with every vcluster instance? Each vcluster essentially is a K3s/K8s cluster with all the add-ons, such as your security plugins, logging plugins, monitoring plugins, ingress controllers, etc. These per vcluster Kubernetes resources and the add-ons can potentially incur substantial overhead as you deploy more vcluster instances. Similarly, since each vcluster essentially is a Kubernetes cluster, platform teams may incur additional overhead managing these clusters for Kubernetes version updates, patch management, and add-on management. Does the vcluster match your real production environment? For some organizations, it’s important that the development environment closely matches the production environment. Vcluster supports other distributions such as K8s and EKS, but platform teams must check whether it’s equivalent to running a standalone cluster for the use cases that require near-production environments. For instance, EKS supports many advanced features, including third-party CNIs, various storage classes, auto-scaling, IRSA, and add-ons, which may not be available in the virtual EKS clusters. Conclusion For most platform teams, multi-tenancy based on namespaces with additional automation around namespace life cycle management, security, and cost control solves their multi-tenancy use cases. Vcluster addresses a specific gap in Kubernetes namespace multi-tenancy whereby end users that don’t have cluster-level privileges can now access cluster-scoped objects within their virtual cluster. Platform teams must internally validate whether there is such a requirement for their end users, do a thorough cost-benefit analysis taking their security, compliance, and governance requirements into consideration, and implement additional automation around it to ensure it’s enterprise-ready before deployment.

Regularly checking for vulnerabilities in your pipeline is very important. One of the steps to execute is to perform a vulnerability scan of your Docker images. In this blog, you will learn how to perform the vulnerability scan, how to fix the vulnerabilities, and how to add it to your Jenkins pipeline. Enjoy! 1. Introduction In a previous blog from a few years ago, it was described how you could scan your Docker images for vulnerabilities. A follow-up blog showed how to add the scan to a Jenkins pipeline. However, Anchore Engine, which was used in the previous blogs, is not supported anymore. An alternative solution is available with grype, which is also provided by Anchore. In this blog, you will take a closer look at grype, how it works, how you can fix the issues, and how you can add it to your Jenkins pipeline. But first of all, why check for vulnerabilities? You have to stay up-to-date with the latest security fixes nowadays. Many security vulnerabilities are publicly available and therefore can be exploited quite easily. It is therefore a must-have to fix security vulnerabilities as fast as possible in order to minimize your attack surface. But how do you keep up with this? You are mainly focused on business and do not want to have a full-time job fixing security vulnerabilities. That is why it is important to scan your application and your Docker images automatically. Grype can help with scanning your Docker images. Grype will check operating system vulnerabilities but also language-specific packages such as Java JAR files for vulnerabilities and will report them. This way, you have a great tool that will automate the security checks for you. Do note that grype is not limited to scanning Docker images. It can also scan files and directories and can therefore be used for scanning your sources. In this blog, you will create a vulnerable Docker image containing a Spring Boot application. You will install and use grype in order to scan the image and fix the vulnerabilities. In the end, you will learn how to add the scan to your Jenkins pipeline. The sources used in this blog can be found on GitHub. 2. Prerequisites The prerequisites needed for this blog are: Basic Linux knowledge Basic Docker knowledge Basic Java and Spring Boot knowledge 3. Vulnerable Application Navigate to Spring Initializr and choose a Maven build, Java 17, Spring Boot 2.7.6, and the Spring Web dependency. This will not be a very vulnerable application because Spring already ensures that you use the latest Spring Boot version. Therefore, change the Spring Boot version to 2.7.0. The Spring Boot application can be built with the following command, which will create the jar file for you: Shell $ mvn clean verify You are going to scan a Docker image, therefore a Dockerfile needs to be created. You will use a very basic Dockerfile which just contains the minimum instructions needed to create the image. If you want to create production-ready Docker images, do read the posts Docker Best Practices and Spring Boot Docker Best Practices. Dockerfile FROM eclipse-temurin:17.0.1_12-jre-alpine WORKDIR /opt/app ARG JAR_FILE COPY target/${JAR_FILE} app.jar ENTRYPOINT ["java", "-jar", "app.jar"] At the time of writing, the latest eclipse-temurin base image for Java 17 is version 17.0.5_8. Again, use an older one in order to make it vulnerable. For building the Docker image, a fork of the dockerfile-maven-plugin of Spotify will be used. The following snippet is therefore added to the pom file. XML <plugin> <groupId>com.xenoamess.docker</groupId> <artifactId>dockerfile-maven-plugin</artifactId> <version>1.4.25</version> <configuration> <repository>mydeveloperplanet/mygrypeplanet</repository> <tag>${project.version}</tag> <buildArgs> <JAR_FILE>${project.build.finalName}.jar</JAR_FILE> </buildArgs> </configuration> </plugin> The advantage of using this plugin is that you can easily reuse the configuration. Creating the Docker image can be done by a single Maven command. Building the Docker image can be done by invoking the following command: Shell $ mvn dockerfile:build You are now all set up to get started with grype. 4. Installation Installation of grype can be done by executing the following script: Shell $ curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin Verify the installation by executing the following command: Shell $ grype version Application: grype Version: 0.54.0 Syft Version: v0.63.0 BuildDate: 2022-12-13T15:02:51Z GitCommit: 93499eec7e3ce2704755e9f51457181b06b519c5 GitDescription: v0.54.0 Platform: linux/amd64 GoVersion: go1.18.8 Compiler: gc Supported DB Schema: 5 5. Scan Image Scanning the Docker image is done by calling grype followed by docker:, indicating that you want to scan an image from the Docker daemon, the image, and the tag: Shell $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT Application: grype Version: 0.54.0 Syft Version: v0.63.0 Vulnerability DB [updated] Loaded image Parsed image Cataloged packages [50 packages] Scanned image [42 vulnerabilities] NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY busybox 1.34.1-r3 1.34.1-r5 apk CVE-2022-28391 High jackson-databind 2.13.3 java-archive CVE-2022-42003 High jackson-databind 2.13.3 java-archive CVE-2022-42004 High jackson-databind 2.13.3 2.13.4 java-archive GHSA-rgv9-q543-rqg4 High jackson-databind 2.13.3 2.13.4.1 java-archive GHSA-jjjh-jjxp-wpff High java 17.0.1+12 binary CVE-2022-21248 Low java 17.0.1+12 binary CVE-2022-21277 Medium java 17.0.1+12 binary CVE-2022-21282 Medium java 17.0.1+12 binary CVE-2022-21283 Medium java 17.0.1+12 binary CVE-2022-21291 Medium java 17.0.1+12 binary CVE-2022-21293 Medium java 17.0.1+12 binary CVE-2022-21294 Medium java 17.0.1+12 binary CVE-2022-21296 Medium java 17.0.1+12 binary CVE-2022-21299 Medium java 17.0.1+12 binary CVE-2022-21305 Medium java 17.0.1+12 binary CVE-2022-21340 Medium java 17.0.1+12 binary CVE-2022-21341 Medium java 17.0.1+12 binary CVE-2022-21360 Medium java 17.0.1+12 binary CVE-2022-21365 Medium java 17.0.1+12 binary CVE-2022-21366 Medium libcrypto1.1 1.1.1l-r7 apk CVE-2021-4160 Medium libcrypto1.1 1.1.1l-r7 1.1.1n-r0 apk CVE-2022-0778 High libcrypto1.1 1.1.1l-r7 1.1.1q-r0 apk CVE-2022-2097 Medium libretls 3.3.4-r2 3.3.4-r3 apk CVE-2022-0778 High libssl1.1 1.1.1l-r7 apk CVE-2021-4160 Medium libssl1.1 1.1.1l-r7 1.1.1n-r0 apk CVE-2022-0778 High libssl1.1 1.1.1l-r7 1.1.1q-r0 apk CVE-2022-2097 Medium snakeyaml 1.30 java-archive GHSA-mjmj-j48q-9wg2 High snakeyaml 1.30 1.31 java-archive GHSA-3mc7-4q67-w48m High snakeyaml 1.30 1.31 java-archive GHSA-98wm-3w3q-mw94 Medium snakeyaml 1.30 1.31 java-archive GHSA-c4r9-r8fh-9vj2 Medium snakeyaml 1.30 1.31 java-archive GHSA-hhhw-99gj-p3c3 Medium snakeyaml 1.30 1.32 java-archive GHSA-9w3m-gqgf-c4p9 Medium snakeyaml 1.30 1.32 java-archive GHSA-w37g-rhq8-7m4j Medium spring-core 5.3.20 java-archive CVE-2016-1000027 Critical ssl_client 1.34.1-r3 1.34.1-r5 apk CVE-2022-28391 High zlib 1.2.11-r3 1.2.12-r0 apk CVE-2018-25032 High zlib 1.2.11-r3 1.2.12-r2 apk CVE-2022-37434 Critical What does this output tell you? NAME: The name of the vulnerable package INSTALLED: Which version is installed FIXED-IN: In which version the vulnerability is fixed TYPE: The type of dependency, e.g., binary for the JDK, etc. VULNERABILITY: The identifier of the vulnerability; with this identifier, you are able to get more information about the vulnerability in the CVE database SEVERITY: Speaks for itself and can be negligible, low, medium, high, or critical. As you take a closer look at the output, you will notice that not every vulnerability has a confirmed fix. So what do you do in that case? Grype provides an option in order to show only the vulnerabilities with a confirmed fix. Adding the --only-fixed flag will do the trick. Shell $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [50 packages] Scanned image [42 vulnerabilities] NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY busybox 1.34.1-r3 1.34.1-r5 apk CVE-2022-28391 High jackson-databind 2.13.3 2.13.4 java-archive GHSA-rgv9-q543-rqg4 High jackson-databind 2.13.3 2.13.4.1 java-archive GHSA-jjjh-jjxp-wpff High libcrypto1.1 1.1.1l-r7 1.1.1n-r0 apk CVE-2022-0778 High libcrypto1.1 1.1.1l-r7 1.1.1q-r0 apk CVE-2022-2097 Medium libretls 3.3.4-r2 3.3.4-r3 apk CVE-2022-0778 High libssl1.1 1.1.1l-r7 1.1.1n-r0 apk CVE-2022-0778 High libssl1.1 1.1.1l-r7 1.1.1q-r0 apk CVE-2022-2097 Medium snakeyaml 1.30 1.31 java-archive GHSA-3mc7-4q67-w48m High snakeyaml 1.30 1.31 java-archive GHSA-98wm-3w3q-mw94 Medium snakeyaml 1.30 1.31 java-archive GHSA-c4r9-r8fh-9vj2 Medium snakeyaml 1.30 1.31 java-archive GHSA-hhhw-99gj-p3c3 Medium snakeyaml 1.30 1.32 java-archive GHSA-9w3m-gqgf-c4p9 Medium snakeyaml 1.30 1.32 java-archive GHSA-w37g-rhq8-7m4j Medium ssl_client 1.34.1-r3 1.34.1-r5 apk CVE-2022-28391 High zlib 1.2.11-r3 1.2.12-r0 apk CVE-2018-25032 High zlib 1.2.11-r3 1.2.12-r2 apk CVE-2022-37434 Critical Note that the vulnerabilities for the Java JDK have disappeared, although there exists a more recent update for the Java 17 JDK. However, this might not be a big issue, because the other (non-java-archive) vulnerabilities show you that the base image is outdated. 6. Fix Vulnerabilities Fixing the vulnerabilities is quite easy in this case. First of all, you need to update the Docker base image. Change the first line in the Docker image: Dockerfile FROM eclipse-temurin:17.0.1_12-jre-alpine into: Dockerfile FROM eclipse-temurin:17.0.5_8-jre-alpine Build the image and run the scan again: Shell $ mvn dockerfile:build ... $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [62 packages] Scanned image [14 vulnerabilities] NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY jackson-databind 2.13.3 2.13.4 java-archive GHSA-rgv9-q543-rqg4 High jackson-databind 2.13.3 2.13.4.1 java-archive GHSA-jjjh-jjxp-wpff High snakeyaml 1.30 1.31 java-archive GHSA-3mc7-4q67-w48m High snakeyaml 1.30 1.31 java-archive GHSA-98wm-3w3q-mw94 Medium snakeyaml 1.30 1.31 java-archive GHSA-c4r9-r8fh-9vj2 Medium snakeyaml 1.30 1.31 java-archive GHSA-hhhw-99gj-p3c3 Medium snakeyaml 1.30 1.32 java-archive GHSA-9w3m-gqgf-c4p9 Medium snakeyaml 1.30 1.32 java-archive GHSA-w37g-rhq8-7m4j Medium As you can see in the output, only the java-archive vulnerabilities are still present. The other vulnerabilities have been solved. Next, fix the Spring Boot dependency vulnerability. Change the version of Spring Boot from 2.7.0 to 2.7.6 in the POM. XML <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.7.6</version> <relativePath/> <!-- lookup parent from repository --> </parent> Build the JAR file, build the Docker image, and run the scan again: Shell $ mvn clean verify ... $ mvn dockerfile:build ... $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [62 packages] Scanned image [10 vulnerabilities] NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY snakeyaml 1.30 1.31 java-archive GHSA-3mc7-4q67-w48m High snakeyaml 1.30 1.31 java-archive GHSA-98wm-3w3q-mw94 Medium snakeyaml 1.30 1.31 java-archive GHSA-c4r9-r8fh-9vj2 Medium snakeyaml 1.30 1.31 java-archive GHSA-hhhw-99gj-p3c3 Medium snakeyaml 1.30 1.32 java-archive GHSA-9w3m-gqgf-c4p9 Medium snakeyaml 1.30 1.32 java-archive GHSA-w37g-rhq8-7m4j Medium So, you got rid of the jackson-databind vulnerability, but not of the snakeyaml vulnerability. So, in which dependency is snakeyaml 1.30 being used? You can find out by means of the dependency:tree Maven command. For brevity purposes, only a part of the output is shown here: Shell $ mvnd dependency:tree ... com.mydeveloperplanet:mygrypeplanet:jar:0.0.1-SNAPSHOT [INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.7.6:compile [INFO] | +- org.springframework.boot:spring-boot-starter:jar:2.7.6:compile [INFO] | | +- org.springframework.boot:spring-boot:jar:2.7.6:compile [INFO] | | +- org.springframework.boot:spring-boot-autoconfigure:jar:2.7.6:compile [INFO] | | +- org.springframework.boot:spring-boot-starter-logging:jar:2.7.6:compile [INFO] | | | +- ch.qos.logback:logback-classic:jar:1.2.11:compile [INFO] | | | | \- ch.qos.logback:logback-core:jar:1.2.11:compile [INFO] | | | +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.17.2:compile [INFO] | | | | \- org.apache.logging.log4j:log4j-api:jar:2.17.2:compile [INFO] | | | \- org.slf4j:jul-to-slf4j:jar:1.7.36:compile [INFO] | | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:compile [INFO] | | \- org.yaml:snakeyaml:jar:1.30:compile ... The output shows us that the dependency is part of the spring-boot-starter-web dependency. So, how do you solve this? Strictly speaking, Spring has to solve it. But if you do not want to wait for a solution, you can solve it by yourself. Solution 1: You do not need the dependency. This is the easiest fix and is low risk. Just exclude the dependency from the spring-boot-starter-web dependency in the pom. XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <exclusions> <exclusion> <groupId>org.yaml</groupId> <artifactId>snakeyaml</artifactId> </exclusion> </exclusions> </dependency> Build the JAR file, build the Docker image, and run the scan again: Shell $ mvn clean verify ... $ mvn dockerfile:build ... $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [61 packages] Scanned image [3 vulnerabilities] No vulnerabilities found No vulnerabilities are found anymore. Solution 2: You do need the dependency. You can replace this transitive dependency by means of dependencyManagement in the pom. This is a bit more tricky because the updated transitive dependency is not tested with the spring-boot-starter-web dependency. It is a trade-off whether you want to do this or not. Add the following section to the pom: XML <dependencyManagement> <dependencies> <dependency> <groupId>org.yaml</groupId> <artifactId>snakeyaml</artifactId> <version>1.32</version> </dependency> </dependencies> </dependencyManagement> Build the jar file, build the Docker image, and run the scan again: Shell $ mvn clean verify ... $ mvn dockerfile:build ... $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [62 packages] Scanned image [3 vulnerabilities] No vulnerabilities found Again, no vulnerabilities are present anymore. Solution 3: This is the solution when you do not want to do anything or whether it is a false positive notification. Create a .grype.yaml file where you exclude the vulnerability with High severity and execute the scan with the --config flag followed by the .grype.yaml file containing the exclusions. The .grype.yaml file looks as follows: YAML ignore: - vulnerability: GHSA-3mc7-4q67-w48m Run the scan again: Shell $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed Vulnerability DB [no update available] Loaded image Parsed image Cataloged packages [62 packages] Scanned image [10 vulnerabilities] NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY snakeyaml 1.30 1.31 java-archive GHSA-98wm-3w3q-mw94 Medium snakeyaml 1.30 1.31 java-archive GHSA-c4r9-r8fh-9vj2 Medium snakeyaml 1.30 1.31 java-archive GHSA-hhhw-99gj-p3c3 Medium snakeyaml 1.30 1.32 java-archive GHSA-9w3m-gqgf-c4p9 Medium snakeyaml 1.30 1.32 java-archive GHSA-w37g-rhq8-7m4j Medium The High vulnerability is not shown anymore. 7. Continuous Integration Now you know how to manually scan your Docker images. However, you probably want to scan the images as part of your continuous integration pipeline. In this section, a solution is provided when using Jenkins as a CI platform. The first question to answer is how you will be notified when vulnerabilities are found. Up until now, you only noticed the vulnerabilities by looking at the standard output. This is not a solution for a CI pipeline. You want to get notified and this can be done by failing the build. Grype has the --fail-on <severity> flag for this purpose. You probably do not want to fail the pipeline when a vulnerability with severity negligible has been found. Let’s see what happens when you execute this manually. First of all, introduce the vulnerabilities again in the Spring Boot application and in the Docker image. Build the JAR file, build the Docker image and run the scan with flag --fail-on: Shell $ mvn clean verify ... $ mvn dockerfile:build ... $ grype docker:mydeveloperplanet/mygrypeplanet:0.0.1-SNAPSHOT --only-fixed --fail-on high ... 1 error occurred: * discovered vulnerabilities at or above the severity threshold Not all output has been shown here, but only the important part. And, as you can see, at the end of the output, a message is shown that the scan has generated an error. This will cause your Jenkins pipeline to fail and as a consequence, the developers are notified that something went wrong. In order to add this to your Jenkins pipeline, several options exist. Here it is chosen to create the Docker image and execute the grype Docker scan from within Maven. There is no separate Maven plugin for grype, but you can use the exec-maven-plugin for this purpose. Add the following to the build-plugins section of the POM. XML <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>3.1.0</version> <configuration> <executable>grype</executable> <arguments> <argument>docker:mydeveloperplanet/mygrypeplanet:${project.version}</argument> <argument>--scope</argument> <argument>all-layers</argument> <argument>--fail-on</argument> <argument>high</argument> <argument>--only-fixed</argument> <argument>-q</argument> </arguments> </configuration> </plugin> </plugins> </build> Two extra flags are added here: --scope all-layers: This will scan all layers involved in the Docker image. -q: This will use quiet logging and will show only the vulnerabilities and possible failures. You can invoke this with the following command: Shell $ mvnd exec:exec You can add this to your Jenkinsfile inside the withMaven wrapper: Plain Text withMaven() { sh 'mvn dockerfile:build dockerfile:push exec:exec' } 8. Conclusion In this blog, you learned how to scan your Docker images by means of grype. Grype has some interesting, user-friendly features which allow you to efficiently add them to your Jenkins pipeline. Also, installing grype is quite easy. Grype is definitely a great improvement over Anchor Engine.

It’s the start of a new year, and while lots of folks are promising to be more active, I’m going to show you how to make Promises to be lazier: JavaScript Promises, that is. It will make more sense in a moment. First, let’s look at a basic Promise example. Here I have a function called sleep that takes a time in milliseconds and a value. It returns a promise that will execute a setTimeout for the number of milliseconds that we should wait, then the Promise resolves with the value. /** * @template ValueType * @param {number} ms * @param {ValueType} value * @returns {Promise<ValueType>} */ function sleep(ms, value) { return new Promise((resolve) => { setTimeout(() => resolve(value), ms); }); } It works like this: We can await the sleep function with the arguments 1000 and 'Yawn & stretch', and after one second the console will log the string, ‘Yawn & stretch’. There’s nothing too special about that. It probably behaves as you would expect, but it gets a little weird if we store it as a variable to await later on, rather than awaiting the returned Promise right away. const nap = sleep(1000, 'Yawn & stretch') Now let’s say we do some other work that takes time (like typing out the next example), and then await the nap variable. You might expect a one-second delay before resolving, but in fact, it resolves immediately. Anytime you create a Promise, you instantiate whatever asynchronous functionality it’s responsible for. In our example, the moment we define the nap variable, the Promise gets created which executes the setTimeout. Because I’m a slow typer, the Promise will be resolved by the time we await it. In other words, Promises are eager. They do not wait for you to await them. In some cases, this is a good thing. In other cases, it could lead to unnecessary resource use. For those scenarios, you may want something that looks like a Promise, but uses lazy evaluation to only instantiate when you need it. Before we continue, I want to show you something interesting. Promises are not the only things that can be awaited in JavaScript. If we create a plain Object with a .then() method, we can actually await that object just like any Promise. This is kind of weird, but it also allows us to create different objects that look like Promises, but aren’t. These objects are sometimes called “Thenables." With that in mind, let’s create a new class called LazyPromise that extends the built-in Promise constructor. Extending Promise isn’t strictly necessary, but it makes it appear more similar to a Promise using things like instanceof. class LazyPromise extends Promise { /** @param {ConstructorParameters<PromiseConstructor>[0]} executor */ constructor(executor) { super(executor); if (typeof executor !== 'function') { throw new TypeError(`LazyPromise executor is not a function`); } this._executor = executor; } then() { this.promise = this.promise || new Promise(this._executor); return this.promise.then.apply(this.promise, arguments); } } The part to focus on is the then() method. It hijacks the default behavior of a standard Promise to wait until the .then() method is executed before creating a real Promise. This avoids instantiating the asynchronous functionality until you actually call for it, and it works whether you explicitly call .then() or use await. Now, let’s see what happens if we replace the Promise in the original sleep function with a LazyPromise. Once again, we’ll assign the result to a nap variable. function sleep(ms, value) { return new LazyPromise((resolve) => { setTimeout(() => resolve(value), ms); }); } const nap = sleep(1000, 'Yawn & stretch') Then we take our time to type out the await nap line and execute it. This time, we see a one-second delay before the Promise resolves, regardless of how much time passed since the variable was created. (Note that this implementation only creates the new Promise once and references it in subsequent calls. So if we were to await it again, it would resolve immediately like any normal Promise.) Of course, this is a trivial example that you probably won’t find in production code, but there are many projects that use lazy-evaluated Promise-like objects. Probably the most common example is with database ORMs and query builders like Knex.js or Prisma. Consider the pseudo-code below. It’s inspired by some of these query builders: const query = db('user') .select('name') .limit(10) const users = await query We create a database query that goes to the "user" table and selects the first ten entries and returns their names. In theory, this would work fine with a regular Promise. But what if we wanted to modify the query based on certain conditions like query string parameters? It would be nice to be able to continue modifying the query before ultimately awaiting the Promise. const query = db('user') .select('name') .limit(10) if (orderBy) { query.orderBy(orderBy) } if (limit) { query.limit(limit) } if (id) { query.where({ id: id }) } const users = await query If the original database query was a standard Promise, it would eagerly instantiate the query as soon as we assigned the variable, and we wouldn’t be able to modify it later on. With lazy evaluation, we can write code like this that’s easier to follow, improves the developer experience, and only executes the query once when we need it. That’s one example where lazy evaluation is great. It might also be useful for things like building, modifying, and orchestrating HTTP requests. Lazy Promises are very cool for the right use cases, but that’s not to say that they should replace every Promise. In some cases, it’s beneficial to instantiate eagerly and have the response ready as soon as possible. This is another one of those “it depends” scenarios. But the next time someone asks you to make a Promise, consider being lazy about it ( ͡° ͜ʖ ͡°). Thank you so much for reading. If you liked this article, please share it.

Whether you're building an application from scratch with zero users, or adding features to an existing application, working with data during development is a necessity. This can take different forms, from mock data APIs reading data files in development, to seeded database deployments closely mirroring an expected production environment. I prefer the latter as I find fewer deviations from my production toolset leads to fewer bugs. A Humble Beginning For the sake of this discussion, let's assume we're building an online learning platform offering various coding courses. In its simplest form, our Node.js API layer might look like this. JavaScript // server.js const express = require("express"); const App = express(); const courses = [ {title: "CSS Fundamentals", "thumbnail": "https://fake-url.com/css"}], {title: "JavaScript Basics", "thumbnail": "https://fake-url.com/js-basics"}], {title: "Intermediate JavaScript", "thumbnail": "https://fake-url.com/intermediate-js"} ]; App.get("/courses", (req, res) => { res.json({data: courses}); }); App.listen(3000); If all you need is a few items to start building your UI, this is enough to get going. Making a call to our /courses endpoint will return all of the courses defined in this file. However, what if we want to begin testing with a dataset more representative of a full-fledged database-backed application? Working With JSON Suppose we inherited a script exporting a JSON-array containing thousands of courses. We could import the data, like so. JavaScript // courses.js module.exports = [ {title: "CSS Fundamentals", "thumbnail": "https://fake-url.com/css"}], {title: "JavaScript Basics", "thumbnail": "https://fake-url.com/js-basics"}], {title: "Intermediate JavaScript", "thumbnail": "https://fake-url.com/intermediate-js"}, ... ]; // server.js ... const courses = require("/path/to/courses.js"); ... This eliminates the need to define our mock data within our server file, and now we have plenty of data to work with. We could enhance our endpoint by adding parameters to paginate the results and set limits on how many records are returned. But, what about allowing users to post their own courses? How about editing courses? This solution gets out of hand quickly as you begin to add functionality. We'll have to write additional code to simulate the features of a relational database. After all, databases were created to store data. So, let's do that. Bulk Loading JSON With Sequelize For an application of this nature, PostgreSQL is an appropriate database selection. We have the option of running PostgreSQL locally or connecting to a PostgreSQL-compatible cloud-native database, like YugabyteDB Managed. Apart from being a highly-performant distributed SQL database, developers using YugabyteDB benefit from a cluster that can be shared by multiple users. As the application grows, our data layer can scale out to multiple nodes and regions. After creating a YugabyteDB Managed account and spinning up a free database cluster, we're ready to seed our database and refactor our code, using Sequelize. The Sequelize ORM allows us to model our data to create database tables and execute commands. Here's how that works. First, we install Sequelize from our terminal. Shell // terminal > npm i sequelize Next, we use Sequelize to establish a connection to our database, create a table, and seed our table with data. JavaScript // database.js // JSON-array of courses const courses = require("/path/to/courses.js"); // Certificate file downloaded from YugabyteDB Managed const cert = fs.readFileSync(CERTIFICATE_PATH).toString(); // Create a Sequelize instance with our database connection details const Sequelize = require("sequelize"); const sequelize = new Sequelize("yugabyte", "admin", DB_PASSWORD, { host: DB_HOST, port: "5433", dialect: "postgres", dialectOptions: { ssl: { require: true, rejectUnauthorized: true, ca: cert, }, }, pool: { max: 5, min: 1, acquire: 30000, idle: 10000, } }); // Defining our Course model export const Course = sequelize.define( "course", { id: { type: DataTypes.INTEGER, autoIncrement: true, primaryKey: true, }, title: { type: DataTypes.STRING, }, thumbnail: { type: DataTypes.STRING, }, } ); async function seedDatabase() { try { // Verify that database connection is valid await sequelize.authenticate(); // Create database tables based on the models we've defined // Drops existing tables if there are any await sequelize.sync({ force: true }); // Creates course records in bulk from our JSON-array await Course.bulkCreate(courses); console.log("Courses created successfully!"); } catch(e) { console.log(`Error in seeding database with courses: ${e}`); } } // Running our seeding function seedDatabase(); By leveraging Sequelize’s bulkCreate method, we’re able to insert multiple records in one statement. This is more performant than inserting requests one at a time, like this. JavaScript . . . // JSON-array of courses const courses = require("/path/to/courses.js"); async function insertCourses(){ for(let i = 0; i < courses.length; i++) { await Course.create(courses[i]); } } insertCourses(); Individual inserts come with the overhead of connecting, sending requests, parsing requests, indexing, closing connections, etc. on a one-off basis. Of course, some of these concerns are mitigated by connection pooling, but generally speaking the performance benefits of inserting in bulk are immense, not to mention far more convenient. The bulkCreate method even comes with a benchmarking option to pass query execution times to your logging functions, should performance be of primary concern. Now that our database is seeded with records, our API layer can use this Sequelize model to query the database and return courses. JavaScript // server.js const express = require("express"); const App = express(); // Course model exported from database.js const { Course } = require("/path/to/database.js") App.get("/courses", async (req, res) => { try { const courses = await Course.findAll(); res.json({data: courses}); } catch(e) { console.log(`Error in courses endpoint: ${e}`); } }); App.listen(3000); Well, that was easy! We've moved from a static data structure to a fully-functioned database in no time. What if we're provided the dataset in another data format, say, a CSV file exported from Microsoft Excel? How can we use it to seed our database? Working With CSVs There are many NPM packages to convert CSV files to JSON, but none are quite as easy to use as csvtojson. Start by installing the package. Shell // terminal > npm i csvtojson Next, we use this package to convert our CSV file to a JSON-array, which can be used by Sequelize. // courses.csv title,thumbnail CSS Fundamentals,https://fake-url.com/css JavaScript Basics,https://fake-url.com/js-basics Intermediate JavaScript,https://fake-url.com/intermediate-js JavaScript // database.js ... const csv = require('csvtojson'); const csvFilePath = "/path/to/courses.csv"; // JSON-array of courses from CSV const courses = await csv().fromFile(csvFilePath); ... await Course.bulkCreate(courses); ... Just as with our well-formatted courses.js file, we're able to easily convert our courses.csv file to bulk insert records via Sequelize. Conclusion Developing applications with hardcoded data can only take us so far. I find that investing in tooling early in the development process sets me on the path toward bug-free coding (or so I hope!) By bulk-loading records, we’re able to work with a representative dataset, in a representative application environment. As I’m sure many agree, that’s often a major bottleneck in the application development process.

Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios. This, on the one hand, is heavily influenced by the sustained rising and popularity of big-data processing frameworks, including but not limited to Hadoop, Spark, and NoSQL databases; on the other hand, as more and more data are to be analyzed in a correlated and multi-dimensional fashion, it's getting difficult to pack all data into one graph on one instance, having a truly distributed and horizontally scalable graph database is a must-have. Do Not Be Misled Designing and implementing a scalable graph database system has never been a trivial task. There is a countless number of enterprises, particularly Internet giants, that have explored ways to make graph data processing scalable. Nevertheless, most solutions are either limited to their private and narrow use cases or offer scalability in a vertical fashion with hardware acceleration which only proves, again, that the reason why mainframe architecture computer was deterministically replaced by PC-architecture computer in the 90s was mainly that vertical scalability is generally considered inferior and less-capable-n-scalable than horizontal scalability, period. It has been a norm to perceive that distributed databases use the method of adding cheap PC(s) to achieve scalability (storage and computing) and attempt to store data once and for all on demand. However, doing the same cannot achieve equivalent scalability without massively sacrificing query performance on graph systems. Why scalability in a graph (database) system is so difficult (to get)? The primary reason is that graph system is high-dimensional; this is in deep contrast to traditional SQL or NoSQL systems, which are predominantly table-centric, essentially columnar and row stores (and KV stores in a more simplistic way) and have been proved to be relatively easy to implement with a horizontally scalable design. A seemingly simple and intuitive graph query may lead to deep traversal and penetration of a large amount of graph data, which tends to otherwise cause a typical BSP (Bulky Synchronous Processing) system to exchange heavily amongst its many distributed instances, therefore causing significant (and unbearable) latencies. On the other hand, most existing graph systems prefer to sacrifice performance (computation) while offering scalability (storage). This would render such systems impractical and useless in handling many real-world business scenarios. A more accurate way to describe such systems is that they probably can store a large amount of data (across many instances) but cannot offer adequate graph-computing power — to put it another way, these systems fail to return with results when being queried beyond meta-data (nodes and edges). This article aims to demystify the scalability challenge(s) of graph databases, meanwhile putting a lot of focus on performance issues. Simply put, you will have a better and unobstructed understanding of scalability and performance in any graph database system and gain more confidence in choosing your future graph system. There is quite a bit of noise in the market about graph database scalability; some vendors claim they have unlimited scalability, while others claim to be the first enterprise-grade scalable graph databases. Who should you believe or follow? The only way out is to equip yourself with adequate knowledge about scalability in graph database systems so that you can validate it by yourself and don't have to be misled by all those marketing hypes. Admittedly, there are many terms for graph database scalability; some can be dearly confusing, to name a few: HA, RAFT or Distributed Consensus, HTAP, Federation, Fabric, Sharding, Partitioning, etc. Can you really tell the difference, sometimes minute and often with overlapping features, of all these terms? We'll unravel them all. 3 Schools of Distributed Graph System Architecture Designs First, make sure you understand the evolution pathway from a standalone (graph database) instance to a fully distributed and horizontally scalable cluster of graph database instances. Graph 1: Evolution of Distributed (Graph) Systems. A distributed system may take many forms, and this rich diversification may lead to confusion. Some vendors misleadingly (and ironically) claim their database systems to be distributed evenly on a single piece of underpinning hardware instance, while other vendors claim their sharded graph database cluster can handle zillion-scale graph datasets while, in reality, the cluster can't even handle a typical multi-hop graph query or graph algorithm that reiteratively traverse the entire dataset. Simply put, there are ONLY three schools of scalable graph database architecture designs, as captured in the table: Table 1: Comparison of three schools of Distributed Graph Systems. HTAP Architecture The first school is considered a natural extension to the master-slave model, and we are calling it distributed consensus cluster where typically three instances form a graph database cluster. The only reason to have three or an odd number of instances in the same cluster is that it's easier to vote for a leader of the cluster. As you can see, this model of cluster design may have many variations; for instance, Neo4j's Enterprise Edition v4.x supports the original RAFT protocol, and only one instance handles workload, while the other two instances passively synchronize data from the primary instance — this, of course, is a naïve way of putting RAFT protocol to work. A more practical way to handle workload is to augment the RAFT protocol to allow all instances to work in a load-balanced way. For instance, having the leader instance handle read-and-write operations, while the other instances can at least handle read type of queries to ensure data consistencies across the entire cluster. A more sophisticated way in this type of distributed graph system design is to allow for HTAP (Hybrid Transactional and Analytical Processing), meaning there will be varied roles assigned amongst the cluster instances; the leader will handle TP operations, while the followers will handle AP operations, which can be further broken down into roles for graph algorithms, etc. The pros and cons of graph system leveraging distributed consensus include: Small hardware footprint (cheaper). Great data consistency (easier to implement). Best performance on sophisticated and deep queries. Limited scalability (relying on vertical scalability). Difficult to handle a single graph that's over ten billion-plus nodes and edges. What's illustrated below is a novel HTAP architecture from Ultipa with key features like: High-Density Parallel Graph Computing. Multi-Layer Storage Acceleration (Storage is in close proximity to compute). Dynamic Pruning (Expedited graph traversal via dynamic trimming mechanism). Super-Linear Performance (i.e., when computing resource such as the number of CPU cores is doubled, the performance gain can be more than doubled). Graph 2: HTAP Architecture Diagram by Ultipa Graph. Note that such HTAP architecture works wonderfully on graph data size that's below 10B nodes + edges. Because lots of computing acceleration are done via in-memory computing, and if every billion nodes and edges consume about 100GB of DRAM, it may take 1TB of DRAM on a single instance to handle a graph of ten billion nodes and edges. The upside of such design is that the architecture is satisfactory for most real-world scenarios. Even for G-SIBs (Globally Systemically Important Banks), a typical fraud detection, asset-liability management, or liquidity risk management use case would consume around one billion data; a reasonably sized virtual machine or PC server can decently accommodate such data scale and be very productive with an HTAP setup. The downside of such a design is the lack of horizontal (and unlimited) scalability. And this challenge is addressed in the second and third schools of distributed graph system designs (see Table 1). The two graphs below show the performance advantages of HTAP architecture. There are two points to watch out for: Linear Performance Gain: A 3-instance Ultipa HTAP cluster's throughput can reach ~300% of a standalone instance. The gain is reflected primarily in AP type of operations such as meta-data queries, path/k-hop queries, and graph algorithms, but not in TP operations such as insertions or deletions of meta-data because these operations are done primarily on the main instance before synchronized with secondary instances. Better performance = Lower Latency and Higher Throughput (TPS or QPS). Graph 3: Performance Advantages of HTAP Architecture. Graph 4: TPS comparison of Ultipa and Neo4j. Grid Architecture In the second school, there are also quite a few naming variations for such types of distributed and scalable graph system designs (some are misleading). To name a few: Proxy, Name server, MapReduce, Grid, or Federation. Ignore the naming differences; the key difference between the secondary school and the first school lies with the name server(s) functioning as a proxy between the client side and server side. When functioning as a proxy server, the name server is only for routing queries and forwarding data. On top of this, except for the running graph algorithm, the name server has the capacity to aggregate data from the underpinning instances. Furthermore, in federation mode, queries can be run against multiple underpinning instances (query-federation); for graph algorithms, however, the federation's performance is poor (due to data migration, just like how map-reduce works). Note that the second school is different from the third school in one area: data is functionally partitioned but not sharded in this school of design. For graph datasets, functional partitioning is the logical division of graph data, such as per time series (horizontal partitioning) or per business logic (vertical partitioning). Sharding, on the other hand, aims to be automated, business logic or time series ignorant. Sharding normally considers the location of network storage-based partitioning of data; it uses various redundant data and special data distribution to improve performance, such as making cuttings against nodes and edges on the one hand and replicating some of the cut data for better access performance on the other hand. In fact, sharding is very complex and difficult to understand. Automated sharding, by definition, is designed to treat unpredictable data distribution with minimal-to-zero human intervention and business-logic ignorant, but this ignorance can be very problematic when facing business challenges entangled with specific data distribution. Let's use concrete examples to illustrate this. Assuming you have 12 months' worth of credit card transaction data. In artificial partition mode, you naturally divide the network of data into 12 graph sets, one graph set with one-month transactions on each cluster of three instances, and this logic is predefined by the database admin. It emphasizes dividing the data via the metadata of the database and ignoring the connectivity between the different graph sets. It's business-friendly, it won't slow down data migration, and has good query performance. On the other hand, in auto-sharding mode, it's up to the graph system to determine how to divide (cut) the dataset, and the sharding logic is transparent to the database admin. But it's hard for developers to immediately figure out where the data is stored, therefore leading to potential slow data migration problems. It would be imprudent to claim that auto-sharding is more intelligent than functional partitioning simply because auto-sharding involves less human intervention. Do you feel something is wrong here? It's exactly what we are experiencing with the ongoing rising of artificial intelligence, we are allowing machines to make decisions on our behalf, and it's not always intelligent! (In a separate essay, we will cover the topic of the global transition from artificial intelligence to augmented intelligence and why graph technology is strategically positioned to empower this transition.) In Graph-5, a grid architecture pertaining to the second school of design is illustrated; the two extra components added on top of Graph-2's HTAP architecture are name server(s) and meta server(s). Essentially all queries are proxied through the name-server, and the name-sever works jointly with the meta-server to ensure the elasticity of the grid; the server cluster instances are largely the same as the original HTAP instance (as illustrated in Graph 2). Graph 5: Grid Architecture w/ Name Server and Meta Server. Referring to Table 1, the pros and cons of the grid architecture design can be summarized as follows: All the pros/benefits of a typical HTAP architecture are retained. Scalability is achieved with performance intact (compared to HTAP architecture). Restricted scalability — server clusters are partitioned with DBA/admin intervention. Introduction of name-server/meta-server, making cluster management sophisticated. The name-server is critical and complex in ensuring business logic is performed distributively on the server clusters and with simple merge and aggregation functionalities on it before returning to the clients. Business logic may be required to cooperate with partitioning and querying. Shard Architecture Now, we can usher in the third school of distributed graph system design with unlimited scalability — the shard (see Table 1). On the surface, the horizontal scalability of a sharding system also leverages name server and meta server as in the second school of design, but the main differences lie with the: Shard servers are genuinely shared. Name servers do NOT have knowledge about business logic (as in the second school) directly. Indirectly, it can roughly judge the category of business logic via automatic statistics collection. This decoupling is important, and it couldn't be achieved elegantly in the second school. The sharded architecture has some variations; some vendor calls it fabrics (it's actually more like grid architecture in the secondary school), and others call it map-reduce, but we should deep dive into the core data processing logic to unravel the mystery. There are only two types of data processing logic in shard architecture: Type 1: Data is processed mainly on name servers (or proxy servers) Type 2: Data is processed on sharded or partitioned servers as well as name servers. Type 1 is typical, as you see in most map-reduce systems such as Hadoop; data are scattered across the highly distributed instances. However, they need to be lifted and shifted over to the name servers before they are processed there. Type 2 is different in that the shard servers have the capacity to locally process the data (this is called: compute near or collocated with storage or data-centric computing) before they are aggregated and secondarily processed on the name servers. As you would imagine, type 1 is easier to implement as it's a mature design scheme by many big-data frameworks; however, type 2 offers better performance with more sophisticated cluster design and query optimization. Shard servers in type-2 offer computing power, while type-1 has no such capability. The graph below shows a type-2 shard design: Graph 6: Shard Architecture w/ Name Server and Meta Server. Sharding is nothing new from a traditional SQL or NoSQL big-data framework design perspective. However, sharding on graph data can be Pandora's box, and here is why: Multiple shards will increase I/O performance, particularly data ingestion speed. But multiple shards will significantly increase the turnaround time of any graph query that spans across multiple shards, such as path queries, k-hop queries, and most graph algorithms (the latency increase can be exponential!). Graph query planning and optimization can be extremely sophisticated, most vendors today have done very shallowly on this front, and there are tons of opportunities in deepening query optimization on-the-fly: Cascades (Heuristic vs. Cost) Partition-pruning (shard-pruning, actually) Index-choosing Statistics (Smart Estimation) Pushdown (making computing as close to the storage as possible) and more. In Graph-7, we captured some preliminary findings on the Ultipa HTAP cluster and Ultipa Shard cluster; as you can see, data ingestion speed improves by four times (super-linear), but everything else tends to be slower by five times or more (PageRank slower by 10x, LPA by 16X, etc.) Graph 7: Preliminary findings on the performance difference between HTAP and Shard Architecture. Stay Tuned There are tons of opportunities to continuously improve the performance of the sharding architecture. The team at Ultipa has realized that having a truly advanced cluster management mechanism and deeper query optimization on a horizontally scalable system are the keys to achieving endless scalability and satisfactory performance. Lastly, the third schools of distributed graph system architectures illustrate the diversity and complexity involved when designing a sophisticated and competent graph system. Its course, it’s hard to say one architecture is absolutely superior to another, given cost, subjective preference, design philosophy, business logic, complexity-tolerance, serviceability, and many other factors — it would be prudent to conclude that the direction of architecture evolution for the long term clearly is to go from the first school to the second school and eventually to the third school. However, most customer scenarios can be satisfied with the first two schools, and human intelligence (DBA intervention) still makes pivotal sense in helping to achieve an equilibrium of performance and scalability, particularly in the second and third schools of designs. Long live the formula: Graph Augmented Intelligence = Human-intelligence + Machine’s-graph-computing-power

Gaining complexity in a microservices system certainly isn't for the faint of heart (though neither is complexity in monoliths!). When there are many services that need to communicate with one another, we might need to coordinate multiple services communicating with multiple other services. We also might code for varying environments such as local, development server, or the cloud. How do services know where to find one another? How can we avoid problems when a service is unavailable? How do we handle requests when we scale up or down certain parts of our system? This is where something like Spring Cloud Netflix Eureka comes into play. Eureka is a service discovery project that helps services interact with one another without hardwiring in instance-specific or environment-dependent details. Architecture We have carefully built up a system of microservices from generic application chatter to a system of services communicating among one another. In the last article, we migrated most of our standalone services into Docker Compose so that it could orchestrate startup and shutdown as a unit. In this article, we will add a service discovery component, so that services can find and talk to one another without hard-coding host and port information into applications or environments. Docker Compose manages most of the services (in dark gray area), with each containerized service encompassed in a light gray box. Neo4j is the only component managed externally with Neo4j's database-as-a-service (AuraDB). Interactions between services are shown using arrows, and the types of data objects passed to numbered services (1-4) are depicted next to each. Spring Cloud Netflix Eureka Spring Cloud Netflix originally contained a few open-sourced projects from Netflix, including Eureka, Zuul, Hystrix, and Ribbon. Since then, most of those have been migrated into other Spring projects, except for Eureka. Eureka handles service registry and discovery. A Eureka server is a central place for services to register themselves. Eureka clients register with the server and are able to find and communicate with other services on the registry without referencing hostname and port information within the service itself. Config + Eureka Architecture Decision I had to make a decision on architecture when using Spring Cloud Config and Eureka together in a microservices system. There are a couple of options: 1. Config-first approach. Applications (services1-4) will reach out to config server first before gathering up other properties. In this approach, the config server does not register with Eureka. 2. Discovery-first approach. Applications will register with Eureka before connecting to config and gathering properties. In this approach, config server becomes a Eureka client and registers with it. There is an excellent blog post that provides a clear explanation of each, along with pros and cons. I'd highly encourage checking that out! I opted for the config-first approach because there is already a bit of delay starting up applications in Docker Compose (see blog post detailing this). Going with discovery-first would mean an extra step in the chain before applications could connect to config and contact databases. Since I didn't want to slow this step down any further, I decided not to register the config server app with Eureka, leaving it separate. Without further ado, let's start coding! Applications: Eureka Server We will use the Spring Initializr at start.spring.io to set up the outline for our Eureka server application. On the form, we choose Maven for the Project, then leave Language and Spring Boot version fields defaulted. Under the Project Metadata section, I updated the group name for my personal projects, but you are welcome to leave it defaulted. I named the artifact eureka-server, though naming is up to you, as long as we map it properly where needed. All other fields in this section can remain as they are. Under the Dependencies section, we need only Eureka Server. Finally, we can click the Generate button at the bottom to download the project. The project will download as a zip, so we can unzip it and move it to our project folder with the other services. Open the project in your favorite IDE and let's get coding! The pom.xml contains the dependencies and software versions we set up on the Spring Initializr, so we can move to the application.properties file in the src/main/resources folder. Properties files server.port=8761 eureka.client.register-with-eureka=false eureka.client.fetch-registry=false We need to specify a port number for this application to use so that its traffic doesn't conflict with our other services. The default port for Spring Cloud Eureka server is 8761, so we will use that. Next, we don't need to register the server itself with Eureka (useful in systems with multiple Eureka servers), so we will set the eureka.client.register-with-eureka value to false. The last property is set to false because we also don't need this server to pull the registry from other sources (like other Eureka servers). A StackOverflow question and answer addresses these settings well. In the EurekaServerApplication class, we only need to add the annotation @EnableEurekaServer to set this up as a Eureka server. Let's test this locally by starting the application in our IDE and navigating a web browser window to localhost:8761. This should show us a page like the one below, which gives details about the server and a section for Instances currently registered with Eureka. Since we haven't connected any other services with Eureka, we don't have any services registered with the server. That's it for the server, so let's start retrofitting our other services as Eureka clients. Applications: Service1 We don't have many changes to add for Spring Cloud Eureka. Starting in the pom.xml, we need to add a dependency. XML <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId> </dependency> This dependency enables the application as a Eureka client. Most recommendations would also have us adding an annotation like @EnableEurekaClient (Eureka-specific) or @EnableDiscoveryClient (project-agnostic) to the main application class. However, that is not a necessary requirement, as it is defaulted to enabling this functionality when you add the dependency to the pom.xml. To run the service locally, we will also need to add a property to the `application.properties` file. Properties files eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka This tells the application where to look for the Eureka server. We will move this property to the config server file for this application, so we can comment this one out when we test everything together. However, for testing a siloed application, you will need it enabled here. Let's start on changes to service2, which interacts with service1. Applications: Service2 Just like with service1, we need to add the Eureka client dependency to service2's pom.xml to enable service discovery. XML <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-config</artifactId> </dependency> We also want to have this application use Spring Cloud Config for referencing the Eureka server, so we can retrofit that by adding the dependency. We will walk through the config file changes in a bit. Again, if we test locally, we would also need to add the following property to the application.properties file. Properties files eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka Since we will test everything together, it is commented out in the application for now. Instead, we will add a properties file for Spring Cloud Config to host, similar to our other services (next section). Next, we need to make some adjustments to the main application class to utilize Eureka over previously-defined hostname and port locations. Java public class Service2Application { public static void main(String[] args) { SpringApplication.run(Service2Application.class, args); } @Bean @LoadBalanced WebClient.Builder createLoadBalancedBuilder() { return WebClient.builder(); } @Bean WebClient client(WebClient.Builder builder) { return builder.baseUrl("http://mongo-client").build(); } } Eureka lets calling applications reference an application name, and it will map the hostname/port details behind-the-scenes, no matter where the application is running. This is where we see the mongo-client referenced in the second @Bean definition (11th line of above code). We also need to create a load-balanced bean (only required when using Eureka). Step-by-step, I created a WebClient.Builder bean, load balanced it with the @LoadBalanced annotation, then used that to create the actual WebClient bean that gets injected for use in method calls (in the BookController class). Applications: Service3 and Service4 Next, we need to add our other services to Eureka using the steps below. 1. Add the dependency to each pom.xml file. 2. For local testing, add the commented out property in the application.properties file. Now let's add the Eureka property to the Spring Cloud Config files for our applications! Spring Cloud Config For each config file the server hosts, we will need to add the following: YAML eureka: client: serviceUrl: defaultZone: http://goodreads-eureka:8761/eureka This tells the application where to look so it can register with Eureka. Full sample code for each config file is located in the related Github repository folder. We also need to create a whole new config file for service2 to use the config server. YAML spring: application: name: goodreads-client eureka: client: serviceUrl: defaultZone: http://goodreads-eureka:8761/eureka A sample is provided on the Github repository, but this file is created in a local repository initialized with git, and then referenced in the config server properties file for that project to serve up. More information on that is in a previous blog post. Let's make a few changes to the docker-compose.yml! docker-compose.yml We need to remove the dynamic environment property for service2 and to add the Eureka server project for Docker Compose to manage. YAML goodreads-svc2: #other properties... environment: - SPRING_APPLICATION_NAME=goodreads-client - SPRING_CONFIG_IMPORT=configserver:http://goodreads-config:8888 - SPRING_PROFILES_ACTIVE=docker We added environment variables for application name, config server location, and spring profiles like we see in our other services. Next, we need to add our Eureka server application to the compose file. YAML goodreads-eureka: container_name: goodreads-eureka image: jmreif/goodreads-eureka ports: - "8761:8761" environment: - EUREKA_CLIENT_REGISTER-WITH-EUREKA=false - EUREKA_CLIENT_FETCH-REGISTRY=false volumes: - $HOME/Projects/docker/goodreads/config-server/logs:/logs networks: - goodreads For our last step, we need to build all of the updated applications and create the Docker images. To do that we can execute the following commands from the project folder: Shell cd service1 mvn clean package -DskipTests=true cd ../service2 mvn clean package -DskipTests=true cd ../service3 mvn clean package -DskipTests=true cd ../service4 mvn clean package -DskipTests=true cd ../eureka-server mvn clean package Note: the Docker Compose file is using my pre-built images with Apple silicon architecture. If your machine has a different chip, you will need to do one of the following: 1) utilize the build option in the docker-compose.yml file (comment out image option), 2) create your own Docker images and publish to DockerHub (plus modify the docker-compose.yml file image options). We can run our system with the same command we have been using. Shell docker-compose up -d Note: If you are building local images with the `build` field in docker-compose.yml, then use the command `docker-compose up -d --build`. This will build the Docker containers each time on startup from the directories. Next, we can test all of our endpoints. Goodreads-config (mongo): command line with curl localhost:8888/mongo-client/docker. Goodreads-eureka: web browser with localhost:8761 and note the applications (might take a few minutes for everything to register). Goodreads-svc1: command line with curl localhost:8081/db, curl localhost:8081/db/books, and curl localhost:8081/db/book/623a1d969ff4341c13cbcc6b. Goodreads-svc2: command line with curl localhost:8080/goodreads and curl localhost:8080/goodreads/books. Goodreads-svc3: curl localhost:8082/db, curl localhost:8082/db/authors, and curl localhost:8082/db/author/623a48c1b6575ea3e899b164. Goodreads-config (neo4j): command line with curl localhost:8888/neo4j-client/docker. Neo4j database: ensure AuraDB instance is running (free instances are automatically paused after 3 days). Goodreads-svc4: curl localhost:8083/neo, curl localhost:8083/neo/reviews, and curl localhost:8083/neo/reviews/178186 or web browser with only URL. Bring everything back down again with the below command. Shell docker-compose down Wrapping Up! In this iteration of the project, we integrated service discovery through the Spring Cloud Netflix Eureka project. We created a Eureka server project, and then retrofitted our other services as Eureka clients with an added dependency. Finally, we integrated the new Eureka server project to Docker Compose and updated some of the options for the other services. We tested all of our changes by spinning up the entire microservices system and checking each of our endpoints. Keep following along in this journey to find out what comes next (or review previous iterations to see what we have accomplished). Happy coding! Resources Github: microservices-level10 repository Blog post: Baeldung's guide to Spring Cloud Netflix Eureka Blog post: Config First vs. Discovery First Documentation: Spring Cloud Netflix Interview questions: What is Spring Cloud Netflix?

Terraform is a deployment tool that you can use to provision and manage your infrastructure as code. Usually, the infrastructure is cloud-based, but Terraform can manage anything that can be controlled through an API. But why should you consider infrastructure as code (IaC) in the first place? IaC helps avoid some common pitfalls in deployment. Managing your infrastructure using special code opens the door to automation. It also creates a strong release-oriented approach to infrastructure similar to normal application code. In this tutorial, I will go through each step of using Terraform to provision an AWS EC2 instance. 1. How Terraform Works If you have ever deployed software, you would know the importance of repeatability and consistency in your infrastructure setup. You don’t want to mess with environment changes when deploying software on a Friday evening. Terraform makes infrastructure provisioning repeatable and consistent across public, private, and hybrid cloud platforms. The basic principle of Terraform is that you write human-readable configuration code to define your infrastructure. Once you have the code, you ask Terraform to deploy the infrastructure using a single command. The illustration below describes the working of Terraform on a high level. Terraform High-Level The answer is Terraform providers. Providers are basically plugins for Terraform that are made to talk with external APIs. They are written in Golang and distributed through the Terraform registry. A provider acts as the middleman between Terraform and the platform. It handles a bunch of different things such as: Authentication logic Making API requests Managing timeouts and errors Each cloud vendor maintains its own Terraform provider, and at this point, there are hundreds of published providers available on the Terraform registry. You can also write your own Terraform provider. Check out the illustration below. Terraform Providers 2. Terraform AWS EC2 Instance Configuration File Let’s look at how to provision an AWS EC2 instance using Terraform. The below illustration describes the overall process: Terraform EC2 $ mkdir terraform-aws-ec2-demo $ cd terraform-aws-ec2-demo Within the directory, create the main.tf file to write the initial Terraform configuration. See below. provider "aws" { region = "us-west-2" profile = "terraform-user" } resource "aws_instance" "hello_aws" { ami = "ami-0ceecbb0f30a902a6" instance_type = "t2.micro" tags = { Name = "HelloAWS" } } In case you are wondering, the configuration is written using a language known as HCL or Hashicorp Configuration Language. The language is human-readable (even more than JSON) and forms the core of Terraform. The code itself is simple enough to understand. The Terraform AWS Provider Block The first block configures the provider. provider "aws" { region = "us-west-2" profile = "terraform-user" } Providers only have one label: Name. This should be the official name of the provider as published in the Terraform registry. In our example, the value is “aws” for AWS Provider. The AWS Provider is responsible for understanding API interactions and making authenticated requests. You need to configure a provider by passing some inputs to the provider block. As an example, set the region to us-west-2. Also, direct the provider to use the profile for terraform-user. Since AWS APIs need authentication, you will have to create an IAM user, generate credentials from the AWS Console and set those credentials with your local system’s AWS CLI. For this demo, terraform-user is the IAM user I’ve created to manage our Terraform resources. You can read more about setting the AWS credentials. The Terraform AWS Resource Block The next block in the main.tf file describes the resource you want to provision i.e. the EC2 instance. resource "aws_instance" "hello_aws" { ami = "ami-0ceecbb0f30a902a6" instance_type = "t2.micro" tags = { Name = "HelloAWS" } } This EC2 code block is an example of a Terraform resource. Resources are undoubtedly the most important elements in Terraform as they are directly responsible for provisioning the infrastructure. A resource block has two labels. The first label specifies the type of the resource and the second is the name of the resource. For this example, the type of resource is aws_instance and the name is “hello_aws”. The name has no special significance and is only used to reference the resource within the module scope. Of course, the type and name together become a unique resource identifier. Each resource takes some inputs. In this case, the inputs are: AMI ID: This is the AMI id that should be used to create the EC2 instance. In case you are wondering, this is the AMI for Amazon Linux. You can, of course, customize these parameters using Terraform data sources. Instance Type: This is the instance type. I recommend using t2.micro instance that is available in Free Tier. Tags: These are special key-value properties you may want to associate with the instance. Resource also has some outputs that are generated after Terraform has provisioned the necessary infrastructure. In fact, the input arguments also end up as output attributes. However, Terraform also attaches new output attributes such as the id of the created resource. 3. Initializing the Terraform Provider Before Terraform can deploy our EC2 instance, you need to initialize the workspace. What this means is that you have to download and install the AWS Provider binary from the Terraform registry. This is needed at least once for every workspace. To initialize the workspace, run the command terraform init. You should see output like this: Initializing the backend... Initializing provider plugins... - Finding latest version of hashicorp/aws... - Installing hashicorp/aws v4.48.0... - Installed hashicorp/aws v4.48.0 (signed by HashiCorp) Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. Terraform fetches the latest version of the AWS provider and installs it in the workspace. At this point, you will find an additional folder .terraform popup in your workspace. This contains the downloaded binary for the AWS provider. Also, Terraform creates a lock file .terraform.lock.hcl to record the provider selections. It is good practice to include this file in the version control repository so that Terraform can make the same selections by default when you run terraform init in the future. 4. Deploying the AWS EC2 Instance Using Terraform Once the initialization is successful, you are ready to deploy the EC2 instance using Terraform. Execute the terraform apply command and answer “yes” to the confirmation prompt. You should see similar output once the provisioning is complete. aws_instance.hello_aws: Creating... aws_instance.hello_aws: Still creating... [11s elapsed] aws_instance.hello_aws: Still creating... [21s elapsed] aws_instance.hello_aws: Still creating... [31s elapsed] aws_instance.hello_aws: Creation complete after 37s [id=i-033fd8432fb59ff37] Apply complete! Resources: 1 added, 0 changed, 0 destroyed. Also, you can check the new instance in the AWS console. The instance is created in the us-west-2a region as specified in the Terraform configuration file. Terraform created EC2 instance At this point, you will also find a file named terraform.tfstate created in your project directory. It is basically a JSON file that describes the resources managed by Terraform. You can view the contents of the file in human-readable format by executing terraform show command. Basically, this file is used by Terraform to keep track of the infrastructure it is managing. Of course, this is a simple instance and we have not deployed anything on the instance. You can also use Terraform to deploy a Nginx webserver on an EC2 instance. 5. Destroying the Terraform AWS EC2 Instance Destroying the infrastructure managed by Terraform is as easy as executing the terraform destroy command. This command also prompts for a confirmation, and if you answer “yes," Terraform goes ahead and destroys the EC2 instance created in the previous section. I recommend doing this if you don’t need the infrastructure to continue running. Conclusion Terraform is a declarative IaC provisioning tool that helps deploy resources onto any public or private cloud. In this post, you saw how easy it makes the process of deploying an EC2 instance to AWS. All you had to do was create a configuration file and pass it to Terraform using terraform apply command. The code for this post is available on GitHub for reference. If you found this post useful, consider sharing it with friends and colleagues. Also, in case of any queries, please write them in the comments section below.

When you are selecting databases for your latest use case (or replacing one that’s not meeting your current needs), the good news these days is that you have a lot of options to choose from. Of course, that’s also the bad news. You have a lot to sort through. There are far more databases to consider and compare than ever before. In December 2012, the end of the first year DB-Engines.com first began ranking databases, they had a list of 73 systems (up significantly from the 18 they first started their list with). As of December 2022, they are just shy of 400 systems. This represents a Cambrian explosion of database technologies over the past decade. There is a vast sea of options to navigate: SQL, NoSQL, and a mix of “multi-model” databases that can be a mix of both SQL and NoSQL, or multiple data models of NoSQL (combining two or more options: document, key-value, wide column, graph, and so on). Further, users should not confuse outright popularity with fitness for their use case. While network effects definitely have advantages (“Can’t go wrong with X if everyone is using it”), it can also lead to groupthink, stifling innovation, and competition. My colleague Arthur Pesa and I recently walked through a consideration of five factors that users need to keep foremost when shortlisting and comparing databases. The Five Factors Let’s get straight into the list. Software Architecture — Does the database use the most efficient data structures, flexible data models, and rich query languages to support your workloads and query patterns? Hardware Utilization — Can it take full advantage of modern hardware platforms? Or will you be leaving a significant amount of CPU cycles underutilized? Interoperability — How easy is it to integrate into your development environment? Does it support your programming languages, frameworks, and projects? Was it designed to integrate into your microservices and event streaming architecture? RASP — Does it have the necessary qualities of Reliability, Availability, Scalability, Serviceability, and of course, Performance? Deployment — Does this database only work in a limited environment, such as only on-premises, or only in a single datacenter or a single cloud vendor? Or does it lend itself to being deployed exactly where and how you want around the globe? Any such breakdown is subjective. You may have your own list of 4 factors, or 12, or 20, or 100 criteria. And, of course, each of these factors like software architecture break down into subcategories, such as “storage engine,” “distributed processing architecture,” and even “query language.” But this is how I’d bucketize them into general categories. Software Architecture The critical consideration here is “Does the database use the most efficient data structures, flexible data models, and rich query languages to support your specific workloads and query patterns?” Workload — Do you need to do a write-heavy or mixed read-write transactional workload? Or are you going to do a mostly-read analytical workload? Do you need to have a hybrid workload with a mix of transactions and analytics? Is that workload real-time, batched or a mix? Is it a steady stream of events per second, or are there smooth, regular intraday predictable rises and falls? Or maybe do you need to plan to deal with stochastic shocks of sudden bursts of traffic (for example, breaking news, or any other sudden popularity of a record)? Data Model — Are you dealing with key-value pairs? Wide column stores (row-based “key-key-value” data)? A column store (columnar data)? Document? Graph? RDBMS (with tables and JOINs)? Or something else entirely. Do you really have the time and need to do fully normalized data, or will you be absorbing so much unstructured data so quickly that normalization is a fool’s errand, and you’d be better served with a denormalized data model to start with? There’s no singular “right” answer here. “It depends” should be embraced as your mantra. Query Language — Here there is definitely more of a bias. Because while your data engineering team may be able to mask or hide the back-end query model, many of your users have their own biases and preferences. This is one of the main reasons why SQL remains such a lock-in. At the same time, there are new query languages that are available. Some are SQL-like, such as the Cassandra Query Language (CQL) that is used by Cassandra and ScyllaDB. It has a passing familiarity to SQL users. But don’t be fooled – there are no table JOINs! Then there are a series of new school query languages which may use, for example JSON. This is how Amazon DynamoDB queries work. Again, here, ScyllaDB supports such a JSON query model using our Alternator interface, which is compatible with DynamoDB. Regardless of which way you lean, query language should not be an afterthought in your consideration. Transactions / Operations / CAP — Which is more important to you? Fully consistent ACID transactions? Or highly performant, highly available basic CRUD operations? The CAP theorem says you can have any two of three: consistency, availability or partition tolerance. Considering that distributed databases always need to be partition-tolerant, that leaves you with the choice between so-called “CP”-mode consistency-oriented systems, or “AP”-mode availability-oriented systems. And within these modes, there are implementation details to consider. For example, how you achieve strong consistency within a distributed system can vary widely. Consider even the choice of various consensus algorithms to ensure linearizability, like Paxos, Raft, Zookeeper (ZAB), and so on. Besides the different algorithms, each implementation can vary significantly from another. Data Distribution — When you say “distributed system,” what do you mean exactly? Are we talking about a local, single-datacenter cluster? Or are we talking multi-datacenter clustering? How do cross-cluster updates occur? Is it considered all one logical cluster, or does it require inter-cluster syncs? How does it handle data localization and, for example, GDPR compliance? Hardware Utilization We are in the middle of an ongoing revolution in underlying hardware that continues to push the boundaries of software. A lot of software applications, and many databases in particular, are still rooted in decades-old origins, designs, and presumptions. CPU Utilization / Efficiency — A lot of software is said to be running poorly if CPU utilization goes up beyond, say, 40% or 50%. That means you are supposed to run that software inefficiently, leaving half of your box underutilized on a regular basis. In effect, you’re paying for twice the infrastructure (or more) than you actually need. So it behooves you to look at the way your system handles distributed processing. RAM Utilization / Efficiency — Is your database consistently memory-bound? Is its caching too aggressive, or too bloated (such as having multiple layers of caching), keeping unneeded data in memory? How does it optimize its read and write paths? Storage Utilization / Efficiency — What storage format does your database use? Does it have compact mutable tables that may require heavyweight file locking mechanisms? Or does it use immutable tables that may produce fast writes, but come at a cost of space and read amplification? Does it allow for tiered storage? How does it handle concurrency? Are files stored row-wise (good for transactional use cases) or column-wise (good for analytics on highly repetitive data)? Note that there isn’t just one “right” answer. Each solution is optimizing for different use cases. Network Utilization / Efficiency — Here you should think both about the efficiency of client-server cluster communications, as well as intra-cluster communications. Client/server models can be made more efficient with concurrency, connection pooling, and so on. Intra-cluster communications span from typical operational/transactional chatter (replicating data in an update or a write), as well as administrative tasks such as streaming and balancing data between nodes during a topology change. Interoperability No database is an island. How easy is it to integrate into your development environment? Does it support your programming languages, frameworks, and projects? Was it designed to integrate into your microservices and event streaming architecture? Programming Languages / Frameworks — Over and over you hear “We’re an X shop,” where X stands for your preferred programming language or framework. If your database doesn’t have the requisite client, SDK, library, ORM, and/or other packages to integrate it into that language, it might as well not exist. To be fair, the massive explosion of databases is concurrent to the massive explosion in programming languages. Yet it pays to look at programming language support for the client. Note that this is not the same as what language the database may be written in itself (which may factor into its software architecture and efficiency). This is purely about what languages you can write apps in to connect to that back end database. Event Streaming / Message Queuing — Databases may be a single source of truth, but they are not the only systems running in your company. In fact, you may have different databases all transacting, analyzing, and sharing different slices of your company’s overall data and information space. Event streaming is the increasingly common media for modern enterprises to avoid data silos, and these days your database is only as good as its integration with real-time event streaming and message queuing technologies. Can your database act as both a sink and a source of data? Does it have Change Data Capture (CDC)? Does it connect to your favorite event streaming and message queuing technologies such as Apache Kafka, or Apache Pulsar, or RabbitMQ? APIs — To facilitate your database integration into your application and microservices architecture, does your database support one or more APIs, such as a RESTful interface, or GraphQL? Does it have an administrative API so you can programmatically provision it rather than do everything via a GUI interface? Using the GUI might seem convenient at first, until you have to manage and automate your deployment systems. Other Integrations — What about CI/CD toolchains? Observability platforms? How about using your database as a pluggable storage engine or underlying element of a broader architecture? How well does it serve as infrastructure, or fit into the infrastructure you already use? RASP This acronym goes back decades and generally is used in a hardware context. It stands for Reliability, Availability, Serviceability (or Scalability), and Performance. Basically these “-ilities” are “facilities” — things that make it easy to run your system. In a database, they are vital to consider how much manual intervention and “plate-spinning” you might need to perform to keep your system up and stable. They represent how much the database can take care of itself under general operating conditions, and even mitigate failure states as much as possible. Typical platform engineer spinning up a bunch of new nodes. Reliability —How much tinkering do you need to put in to keep this thing from collapsing, or from data disappearing? Does your database have good durability capabilities? How survivable is it? What anti-entropy mechanisms does it include to get a cluster back in sync? How good are your backup systems? Even more important, how good are your restore systems? And are there operational guardrails to keep individuals from bringing the thing down with a single “Oops!” Availability — What does your database do when you have short term network partitions and transient node unavailability? What happens when a node fully fails? What if that network failure stretches out to more than a few hours? Serviceability — These days the common buzzword is “observability,” which generally encompasses the three pillars of logging, tracing, and metrics. Sure, your database needs to have observability built-in. Yet serviceability goes beyond that. How easy is it to perform upgrades without downtime? How pain-free are maintenance operations? Scalability — Some databases are great to get started with. Then… you hit walls. Hard. Scalability means you don’t have to worry about hitting limits either in terms of total data under management, total operations per second, or geographic limits — such as going beyond a single datacenter to truly global deployability. Plus, there’s horizontal scalability — the scale out of adding more nodes to a cluster — as well as vertical scalability — putting your database on servers that have ever increasing numbers of CPUs, ever more RAM, and more storage (refer back to the hardware section above). Performance — Bottom line: if the database just can’t meet your latency or throughput SLAs, it’s just not going to fly in production. Plus, linked to scalability, many databases seem like they’ll meet your performance requirements at small scale or based on a static benchmark using test data but, when hit with real-world production workloads, just can’t keep up with the increasing frequency, variability, and complexity of queries. So performance requires a strong correlation to linear scale. Deployment All of the above then needs to run where you need it to. Does this database only work in a limited environment, such as only on-premises, or only in a single datacenter or a single cloud vendor? Or does it lend itself to being deployed exactly where and how you want around the globe? Ask yourself these question: Lock-ins — Can this run on-premises? Or, conversely, is it limited to only on-premises deployments? Is it limited to only a certain public cloud vendor, or can this run in the cloud vendor of your choice? What are your hybrid cloud or multicloud deployment options? Management / Control — Similarly, is this only available as a self-managed database, or can it be consumed as a fully-managed Database-as-a-Service (DBaas)? The former allows teams full control of the system, and the latter relieves teams of administrative burden. Both have their tradeoffs. Can you select only one, or does the database allow users to switch between these two business models? Automation and Orchestration — Does it have a Kubernetes Operator to support it in production? Terraform and Ansible scripts? While this is the last item in the list, rest assured, this should not be an afterthought in any production consideration.

ChatGPT is taking the world by storm and redefining the definition of hype in the process. Dutch high school students use it for their homework and instantly give themselves away by handing in essays with perfect spelling and grammar. Everybody’s talking about it; I’m already late to the party. Comments range from admiration to stoic resignation, to plain fear for one’s job or the fate of humanity. I’m no pundit with a crystal ball, but my instinct is that our jobs as software developers are safe. For now. Thanks to my background in linguistics and short stint as a translator, I took an early interest in machine translation and am still unimpressed by the state of the art. It’s hardly surprising. Perfect comprehension of human language is the holy grail of AI. It’s not about self-driving cars. If those performed as poorly as the best translation engine, none would be allowed off the test lot. The native speaker’s intuition of a meaningful sentence is not something you need to be taught. You start picking it up as a baby. For most of human history, we didn’t have schools, and yet we understood each other perfectly — within the same tribe, naturally. This capacity is a defining part of being human. We’re born to speak, with no neat separation into hardware and software. All that makes human language a very hard problem to express in algorithms. Machines are better at systems we designed ourselves, with predictable and exceptionless rules. No board game is safe from AI domination, it would seem. This doesn’t diminish the impressive mental prowess of professional chess or go players. The computer just happens to be better at it, like a deer can outrun you and salmons are better at swimming. None of this should spoil your appetite to compete in these pursuits with other human begins. Just don’t cheat. AI does an impressive job of stealing and interpolating. It invents the room where the original Giaconda posed for da Vinci, but it’s still the result of an algorithm and by nature predictable. Predictable is bland. Great art emerges when the combination of otherwise mundane elements (words, notes and chords, or brush strokes) combines into something that is more than the sum of its parts. You can’t predict, much less force this originality and few artists manage to produce a consistent stream of genius output. They all have their off days. When AI can write a sequel to Macbeth and an album of original Beatles songs worthy of Shakespeare and McCartney, the singularity has arrived. Humans are effectively redundant. It won’t happen in our lifetime, so let’s set our present expectations lower. Surely it can help us write Java? Yes, it can. The upbeat, competitive mood around the annual Advent of Code challenge was seriously soured when it turned out ChatGPT did a more than decent job at solving the tough daily puzzles. I’m not surprised. All these brain teasers are variations on problems that have long been solved. This is AI’s forte: digging up, sorting, and repackaging such collective knowledge. I don’t like puzzles in general, and I was never any good at math. I have failed job interviews for not being able to implement a linked list fast enough to the interviewer’s taste. He considered it basic stuff. It’s basic, all right. We solved it so we could deal with greater levels of abstraction. I don’t want to re-invent it, not even as an exercise. Sure, there are software niches where such mental acuity comes in handy, but I haven’t worked in them. I can’t remember or reproduce the proof of the Pythagoras Theorem. I trust it’s correct. Back to ChatGPT. I signed up and tasked it with a coding challenge I imagined would be right up its street. “Write me a piece of Java code that checks if a String is in camel case.” Unsurprisingly, the answer looked fine. But what a waste of perfectly good processing power and network packets! Of all the sensible options, I chose the laziest, most wasteful, least maintainable one. I did not check whether a common String library or the JRE itself had what I needed. I could have written it myself in ten minutes, but how boring. Instead, I went straight to the oracle. I should not be surprised if my ten lines of code were lifted straight from an Apache commons library. That’s where they should stay, to be properly imported as a library dependency, along with dozens of other useful functions you didn’t know about and instead scavenged from the web to add to your homegrown StringUtils.java. I know, we don’t live in a world of perfect re-use where we only ever write a line of new code if it serves a novel solution to a problem. I know we should not take do not repeat yourself to the extreme. One can in fact over-dry, Jerry Seinfeld. But Github Copilot and ChatGPT are StackOverflow on steroids. They make it all too easy. It will lead to more clueless copy-pasting instead of sensible re-use. I call for DROP: do not repeat other people. From a business perspective, new code is a liability, especially in complex organizations. Senior developers deal with code, but we’re not writing many new lines. Most of our days are spent keeping what we already have in working. That’s a good thing. If you want to code for fun, start or join an open-source project. I can’t pinpoint many activities in a typical workday that I could safely entrust to AI. Some days I don’t deal with code at all. I simply can’t, because coding is the formulation of the answer, and they’re not telling me the question. I don’t mean the question as the stakeholder jotted it down in JIRA, but formulated in crystal-clear human language to ensure we’re not wasting money building the wrong thing from day one. ChatGPT isn’t going to help. There’s the famous and hilarious scene from the Hitchhiker’s Guide where supercomputer Deep Thought has found the succinct answer to the Ultimate Question after seven million years of number-crunching. It’s 42. Too focused on getting to the answer, it had forgotten how complex the question was. Even if AI can work out 95% of the answers as well as the questions, I’m sure you don’t want to be a machine minder, sipping coffee and making three manual adjustments every fifteen minutes? Where’s the fun in that? Perhaps it’s only fair that the promise of a perfect code generator is making developers nervous. We’ve been encroaching on people’s livelihood for half a century. We’re not so evil that we want to rob people of a professional purpose in life. It’s just that automating things for its own sake is such darn fun. And sometimes we assume we’re doing people a favor by cutting out a task that only seems monotonous and trivial. I’ll leave you with a case in point. Eight years ago, I worked on the vessel traffic control software for the port authority in Rotterdam. One of the enhancements was a component to visually manage the passage of vessels coming through the sea lock complex in IJmuiden on their way to Amsterdam. Each ship gets a time slot to pass through one of the six locks in an assigned order, depending on its size. Traffic control wants to squeeze in as many ships as is safely possible because each passage is costly. “As a developer, I want the software to assign each incoming vessel a time slot and location within the lock, to use the available capacity optimally and maximize throughput.” I want it because it’s an irresistible coding challenge. Wrong user story. All traffic control wanted was a graphic depiction of the incoming vessels, true to scale, which they could manually drag and drop into the available locks. As developers, we had over-abstracted the “problem.” We knew the exact size of the locks and the vessels from the maritime database. It looked like another traveling salesman’s problem, only way simpler. Maybe it worked 90% of the time, but what did we know? What about the cargo of the vessel? Does that make a difference? Surely bad weather must affect the minimum safe distance between vessels. Add a few more of these chaotic parameters and you have one epic of a user story. Traffic control wanted a simple system not to save us work or their employer money. Doing the planning was one of the highlights of the day, an opportunity to use their intuition and decades of experience. No one was going to take that away from them.