DZone's Cloud Native Research: Join Us for Our Survey (and $750 Raffle)!
Distributed Caching: Enhancing Performance in Modern Applications
The Modern DevOps Lifecycle
While DevOps is here to stay, as the years pass, we must continuously assess and seek improvements to our existing software processes, systems, and culture — and DevOps is no exception to that rule. With business needs and customer demands constantly shifting, so must our technology, mindsets, and architecture in order to keep pace.Now is the time for this movement that's all about "shifting left" to essentially shift.In our annual DevOps Trend Report, we explore both its fundamental principles as well as the emerging topics, methodologies, and challenges surrounding the engineering ecosystem. Within our "Key Research Findings" and featured articles from our expert community members, readers will find information on core DevOps topics as well as new insights on what's next for DevOps in 2024 and beyond. Join us to learn about the state of CI/CD pipelines, the impact of technical debt, patterns for supply chain management<>DevOps, the rise of platform engineering, and even more!
Core PostgreSQL
AI Automation Essentials
This article presents an in-depth analysis of the service mesh landscape, focusing specifically on Istio, one of the most popular service mesh frameworks. A service mesh is a dedicated infrastructure layer for managing service-to-service communication in the world of microservices. Istio, built to seamlessly integrate with platforms like Kubernetes, provides a robust way to connect, secure, control, and observe services. This journal explores Istio’s architecture, its key features, and the value it provides in managing microservices at scale. Service Mesh A Kubernetes service mesh is a tool that improves the security, monitoring, and reliability of applications on Kubernetes. It manages communication between microservices and simplifies the complex network environment. By deploying network proxies alongside application code, the service mesh controls the data plane. This combination of Kubernetes and service mesh is particularly beneficial for cloud-native applications with many services and instances. The service mesh ensures reliable and secure communication, allowing developers to focus on core application development. A Kubernetes service mesh, like any service mesh, simplifies how distributed applications communicate with each other. It acts as a layer of infrastructure that manages and controls this communication, abstracting away the complexity from individual services. Just like a tracking and routing service for packages, a Kubernetes service mesh tracks and directs traffic based on rules to ensure reliable and efficient communication between services. A service mesh consists of a data plane and a control plane. The data plane includes lightweight proxies deployed alongside application code, handling the actual service-to-service communication. The control plane configures these proxies, manages policies, and provides additional capabilities such as tracing and metrics collection. With a Kubernetes service mesh, developers can separate their application's logic from the infrastructure that handles security and observability, enabling secure and monitored communication between microservices. It also supports advanced deployment strategies and integrates with monitoring tools for better operational control. Istio as a Service Mesh Istio is a popular open-source service mesh that has gained significant adoption among major tech companies like Google, IBM, and Lyft. It leverages the data plane and control plane architecture common to all service meshes, with its data plane consisting of envoy proxies deployed as sidecars within Kubernetes pods. The data plane in Istio is responsible for managing traffic, implementing fault injection for specific protocols, and providing application layer load balancing. This application layer load balancing differs from the transport layer load balancing in Kubernetes. Additionally, Istio includes components for collecting metrics, enforcing access control, authentication, and authorization, as well as integrating with monitoring and logging systems. It also supports encryption, authentication policies, and role-based access control through features like TLS authentication. Find the Istio architecture diagram below: Below, find the configuration and data flow diagram of Istio: Furthermore, Istio can be extended with various tools to enhance its functionality and integrate with other systems. This allows users to customize and expand the capabilities of their Istio service mesh based on their specific requirements. Traffic Management Istio offers traffic routing features that have a significant impact on performance and facilitate effective deployment strategies. These features allow precise control over the flow of traffic and API calls within a single cluster and across clusters. Within a single cluster, Istio's traffic routing rules enable efficient distribution of requests between services based on factors like load balancing algorithms, service versions, or user-defined rules. This ensures optimal performance by evenly distributing requests and dynamically adjusting routing based on service health and availability. Routing traffic across clusters enhances scalability and fault tolerance. Istio provides configuration options for traffic routing across clusters, including round-robin, least connections, or custom rules. This capability allows traffic to be directed to different clusters based on factors such as network proximity, resource utilization, or specific business requirements. In addition to performance optimization, Istio's traffic routing rules support advanced deployment strategies. A/B testing enables the routing of a certain percentage of traffic to a new service version while serving the majority of traffic to the existing version. Canary deployments involve gradually shifting traffic from an old version to a new version, allowing for monitoring and potential rollbacks. Staged rollouts incrementally increase traffic to a new version, enabling precise control and monitoring of the deployment process. Furthermore, Istio simplifies the configuration of service-level properties like circuit breakers, timeouts, and retries. Circuit breakers prevent cascading failures by redirecting traffic when a specified error threshold is reached. Timeouts and retries handle network delays or transient failures by defining response waiting times and the number of request retries. In summary, Istio's traffic routing capabilities provide a flexible and powerful means to control traffic and API calls, improving performance and facilitating advanced deployment strategies such as A/B testing, canary deployments, and staged rollouts. The following is a code sample that demonstrates how to use Istio's traffic routing features in Kubernetes using Istio VirtualService and DestinationRule resources: In the code below, we define a VirtualService named my-service with a host my-service.example.com. We configure traffic routing by specifying two routes: one to the v1 subset of the my-service destination and another to the v2 subset. We assign different weights to each route to control the proportion of traffic they receive. The DestinationRule resource defines subsets for the my-service destination, allowing us to route traffic to different versions of the service based on labels. In this example, we have subsets for versions v1 and v2. Code Sample YAML # Example VirtualService configuration apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-service spec: hosts: - my-service.example.com http: - route: - destination: host: my-service subset: v1 weight: 90 - destination: host: my-service subset: v2 weight: 10 # Example DestinationRule configuration apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: my-service spec: host: my-service subsets: - name: v1 labels: version: v1 - name: v2 labels: version: v2 Observability As the complexity of services grows, it becomes increasingly challenging to comprehend their behavior and performance. Istio addresses this challenge by automatically generating detailed telemetry for all communications within a service mesh. This telemetry includes metrics, distributed traces, and access logs, providing comprehensive observability into the behavior of services. With Istio, operators can easily access and analyze metrics that capture various aspects of service performance, such as request rates, latency, and error rates. These metrics offer valuable insights into the health and efficiency of services, allowing operators to proactively identify and address performance issues. Distributed tracing in Istio enables the capturing and correlation of trace spans across multiple services involved in a request. This provides a holistic view of the entire request flow, allowing operators to understand the latency and dependencies between services. With this information, operators can pinpoint bottlenecks and optimize the performance of their applications. Full access logs provided by Istio capture detailed information about each request, including headers, payloads, and response codes. These logs offer a comprehensive audit trail of service interactions, enabling operators to investigate issues, debug problems, and ensure compliance with security and regulatory requirements. The telemetry generated by Istio is instrumental in empowering operators to troubleshoot, maintain, and optimize their applications. It provides a deep understanding of how services interact, allowing operators to make data-driven decisions and take proactive measures to improve performance and reliability. Furthermore, Istio's telemetry capabilities are seamlessly integrated into the service mesh without requiring any modifications to the application code, making it a powerful and convenient tool for observability. Istio automatically generates telemetry for all communications within a service mesh, including metrics, distributed traces, and access logs. Here's an example of how you can access metrics and logs using Istio: Commands in Bash # Access metrics: istioctl dashboard kiali # Access distributed traces: istioctl dashboard jaeger # Access access logs: kubectl logs -l istio=ingressgateway -n istio-system In the code above, we use the istioctl command-line tool to access Istio's observability dashboards. The istioctl dashboard kiali command opens the Kiali dashboard, which provides a visual representation of the service mesh and allows you to view metrics such as request rates, latency, and error rates. The istioctl dashboard jaeger command opens the Jaeger dashboard, which allows you to view distributed traces and analyze the latency and dependencies between services. To access access logs, we use the kubectl logs command to retrieve logs from the Istio Ingress Gateway. By filtering logs with the label istio=ingressgateway and specifying the namespace istio-system, we can view detailed information about each request, including headers, payloads, and response codes. By leveraging these observability features provided by Istio, operators can gain deep insights into the behavior and performance of their services. This allows them to troubleshoot issues, optimize performance, and ensure the reliability of their applications. Security Capabilities Microservices have specific security requirements, such as protecting against man-in-the-middle attacks, implementing flexible access controls, and enabling auditing tools. Istio addresses these needs with its comprehensive security solution. Istio's security model follows a "security-by-default" approach, providing in-depth defense for deploying secure applications across untrusted networks. It ensures strong identity management, authenticating and authorizing services within the service mesh to prevent unauthorized access and enhance security. Transparent TLS encryption is a crucial component of Istio's security framework. It encrypts all communication within the service mesh, safeguarding data from eavesdropping and tampering. Istio manages certificate rotation automatically, simplifying the maintenance of a secure communication channel between services. Istio also offers powerful policy enforcement capabilities, allowing operators to define fine-grained access controls and policies for service communication. These policies can be dynamically enforced and updated without modifying the application code, providing flexibility in managing access and ensuring secure communication. With Istio, operators have access to authentication, authorization, and audit (AAA) tools. Istio supports various authentication mechanisms, including mutual TLS, JSON Web Tokens (JWT), and OAuth2, ensuring secure authentication of clients and services. Additionally, comprehensive auditing capabilities help operators track service behavior, comply with regulations, and detect potential security incidents. In summary, Istio's security solution addresses the specific security requirements of microservices, providing strong identity management, transparent TLS encryption, policy enforcement, and AAA tools. It enables operators to deploy secure applications and protect services and data within the service mesh. Code Sample YAML # Example DestinationRule for mutual TLS authentication apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: my-service spec: host: my-service trafficPolicy: tls: mode: MUTUAL clientCertificate: /etc/certs/client.pem privateKey: /etc/certs/private.key caCertificates: /etc/certs/ca.pem # Example AuthorizationPolicy for access control apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: my-service-access spec: selector: matchLabels: app: my-service rules: - from: - source: principals: ["cluster.local/ns/default/sa/my-allowed-service-account"] to: - operation: methods: ["*"] In the code above, we configure mutual TLS authentication for the my-service destination using a DestinationRule resource. We set the mode to MUTUAL to enforce mutual TLS authentication between clients and the service. The clientCertificate, privateKey, and caCertificates fields specify the paths to the client certificate, private key, and CA certificate, respectively. We also define an AuthorizationPolicy resource to control access to the my-service based on the source service account. In this example, we allow requests from the my-allowed-service-account service account in the default namespace by specifying its principal in the principals field. By applying these configurations to an Istio-enabled Kubernetes cluster, you can enhance the security of your microservices by enforcing mutual TLS authentication and implementing fine-grained access controls. Circuit Breaking and Retry Circuit breaking and retries are crucial techniques in building resilient distributed systems, especially in microservices architectures. Circuit breaking prevents cascading failures by stopping requests to a service experiencing errors or high latency. Istio's CircuitBreaker resource allows you to define thresholds for failed requests and other error conditions, ensuring that the circuit opens and stops further degradation when these thresholds are crossed. This isolation protects other services from being affected. Additionally, Istio's Retry resource enables automatic retries of failed requests, with customizable backoff strategies, timeout periods, and triggering conditions. By retrying failed requests, transient failures can be handled effectively, increasing the chances of success. Combining circuit breaking and retries enhances the resilience of microservices, isolating failing services and providing resilient handling of intermittent issues. Configuration of circuit breaking and retries in Istio is done within the VirtualService resource, allowing for customization based on specific requirements. Overall, leveraging these features in Istio is essential for building robust and resilient microservices architectures, protecting against failures, and maintaining system reliability. In the code below, we configure circuit breaking and retries for my-service using the VirtualService resource. The retries section specifies that failed requests should be retried up to 3 times with a per-try timeout of 2 seconds. The retryOn field specifies the conditions under which retries should be triggered, such as 5xx server errors or connect failures. The fault section configures fault injection for the service. In this example, we introduce a fixed delay of 5 seconds for 50% of the requests and abort 10% of the requests with a 503 HTTP status code. The circuitBreaker section defines the circuit-breaking thresholds for the service. The example configuration sets the maximum number of connections to 100, maximum HTTP requests to 100, maximum pending requests to 10, sleep window to 5 seconds, and HTTP detection interval to 10 seconds. By applying this configuration to an Istio-enabled Kubernetes cluster, you can enable circuit breaking and retries for your microservices, enhancing resilience and preventing cascading failures. Code Sample YAML apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-service spec: hosts: - my-service http: - route: - destination: host: my-service subset: v1 retries: attempts: 3 perTryTimeout: 2s retryOn: 5xx,connect-failure fault: delay: fixedDelay: 5s percentage: value: 50 abort: httpStatus: 503 percentage: value: 10 circuitBreaker: simpleCb: maxConnections: 100 httpMaxRequests: 100 httpMaxPendingRequests: 10 sleepWindow: 5s httpDetectionInterval: 10s Canary Deployments Canary deployments with Istio offer a powerful strategy for releasing new features or updates to a subset of users or traffic while minimizing the risk of impacting the entire system. With Istio's traffic management capabilities, you can easily implement canary deployments by directing a fraction of the traffic to the new version or feature. Istio's VirtualService resource allows you to define routing rules based on percentages, HTTP headers, or other criteria to selectively route traffic. By gradually increasing the traffic to the canary version, you can monitor its performance and gather feedback before rolling it out to the entire user base. Istio also provides powerful observability features, such as distributed tracing and metrics collection, allowing you to closely monitor the canary deployment and make data-driven decisions. In case of any issues or anomalies, you can quickly roll back to the stable version or implement other remediation strategies, minimizing the impact on users. Canary deployments with Istio provide a controlled and gradual approach to releasing new features, ensuring that changes are thoroughly tested and validated before impacting the entire system, thus improving the overall reliability and stability of your applications. To implement canary deployments with Istio, we can use the VirtualService resource to define routing rules and gradually shift traffic to the canary version. Code Sample YAML apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: my-service spec: hosts: - my-service http: - route: - destination: host: my-service subset: stable weight: 90 - destination: host: my-service subset: canary weight: 10 In the code above, we configure the VirtualService to route 90% of the traffic to the stable version of the service (subset: stable) and 10% of the traffic to the canary version (subset: canary). The weight field specifies the distribution of traffic between the subsets. By applying this configuration, you can gradually increase the traffic to the canary version and monitor its behavior and performance. Istio's observability features, such as distributed tracing and metrics collection, can provide insights into the canary deployment's behavior and impact. If any issues or anomalies are detected, you can quickly roll back to the stable version by adjusting the traffic weights or implementing other remediation strategies. By leveraging Istio's traffic management capabilities, you can safely release new features or updates, gather feedback, and mitigate risks before fully rolling them out to your user base. Autoscaling Istio seamlessly integrates with Kubernetes' Horizontal Pod Autoscaler (HPA) to enable automated scaling of microservices based on various metrics, such as CPU or memory usage. By configuring Istio's metrics collection and setting up the HPA, you can ensure that your microservices scale dynamically in response to increased traffic or resource demands. Istio's metrics collection capabilities allow you to gather detailed insights into the performance and resource utilization of your microservices. These metrics can then be used by the HPA to make informed scaling decisions. The HPA continuously monitors the metrics and adjusts the number of replicas for a given microservice based on predefined scaling rules and thresholds. When the defined thresholds are crossed, the HPA automatically scales up or down the number of pods, ensuring that the microservices can handle the current workload efficiently. This automated scaling approach eliminates the need for manual intervention and enables your microservices to adapt to fluctuating traffic patterns or resource demands in real time. By leveraging Istio's integration with Kubernetes' HPA, you can achieve optimal resource utilization, improve performance, and ensure the availability and scalability of your microservices. Code Sample YAML apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: my-service-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-service minReplicas: 1 maxReplicas: 10 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 In the example above, the HPA is configured to scale the my-service deployment based on CPU usage. The HPA will maintain an average CPU utilization of 50% across all pods. By applying this configuration, Istio will collect metrics from your microservices, and the HPA will automatically adjust the number of replicas based on the defined scaling rules and thresholds. With this integration, your microservices can dynamically scale up or down based on traffic patterns and resource demands, ensuring optimal utilization of resources and improved performance. It’s important to note that the Istio integration with Kubernetes' HPA may require additional configuration and tuning based on your specific requirements and monitoring setup. Implementing Fault Injection and Chaos Testing With Istio Chaos fault injection with Istio is a powerful technique that allows you to test the resilience and robustness of your microservices architecture. Istio provides built-in features for injecting faults and failures into your system, simulating real-world scenarios, and evaluating how well your system can handle them. With Istio's Fault Injection feature, you can introduce delays, errors, aborts, or latency spikes to specific requests or services. By configuring VirtualServices and DestinationRules, you can selectively apply fault injection based on criteria such as HTTP headers or paths. By combining fault injection with observability features like distributed tracing and metrics collection, you can closely monitor the impact of injected faults on different services in real time. Chaos fault injection with Istio helps you identify weaknesses, validate error handling mechanisms, and build confidence in the resilience of your microservices architecture, ensuring the reliability and stability of your applications in production environments. Securing External Traffic Using Istio's Ingress Gateway Securing external traffic using Istio's Ingress Gateway is crucial for protecting your microservices architecture from unauthorized access and potential security threats. Istio's Ingress Gateway acts as the entry point for external traffic, providing a centralized and secure way to manage inbound connections. By configuring Istio's Ingress Gateway, you can enforce authentication, authorization, and encryption protocols to ensure that only authenticated and authorized traffic can access your microservices. Istio supports various authentication mechanisms such as JSON Web Tokens (JWT), mutual TLS (mTLS), and OAuth, allowing you to choose the most suitable method for your application's security requirements. Additionally, Istio's Ingress Gateway enables you to define fine-grained access control policies based on source IP, user identity, or other attributes, ensuring that only authorized clients can reach specific microservices. By leveraging Istio's powerful traffic management capabilities, you can also enforce secure communication between microservices within your architecture, preventing unauthorized access or eavesdropping. Overall, Istio's Ingress Gateway provides a robust and flexible solution for securing external traffic, protecting your microservices, and ensuring the integrity and confidentiality of your data and communications. Code Sample YAML apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: my-gateway spec: selector: istio: ingressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - "*" In this example, we define a Gateway named my-gateway that listens on port 80 and accepts HTTP traffic from any host. The Gateway's selector is set to istio: ingressgateway, which ensures that it will be used as the Ingress Gateway for external traffic. Best Practices for Managing and Operating Istio in Production Environments When managing and operating Istio in production environments, there are several best practices to follow. First, it is essential to carefully plan and test your Istio deployment before production rollout, ensuring compatibility with your specific application requirements and infrastructure. Properly monitor and observe your Istio deployment using Istio's built-in observability features, including distributed tracing, metrics, and logging. Regularly review and update Istio configurations to align with your evolving application needs and security requirements. Implement traffic management cautiously, starting with conservative traffic routing rules and gradually introducing more advanced features like traffic splitting and canary deployments. Take advantage of Istio's traffic control capabilities to implement circuit breaking, retries, and timeout policies to enhance the resilience of your microservices. Regularly update and patch your Istio installation to leverage the latest bug fixes, security patches, and feature enhancements. Lastly, establish a robust backup and disaster recovery strategy to mitigate potential risks and ensure business continuity. By adhering to these best practices, you can effectively manage and operate Istio in production environments, ensuring the reliability, security, and performance of your microservices architecture. Conclusion In the evolving landscape of service-to-service communication, Istio, as a service mesh, has surfaced as an integral component, offering a robust and flexible solution for managing complex communication between microservices in a distributed architecture. Istio's capabilities extend beyond merely facilitating communication to providing comprehensive traffic management, enabling sophisticated routing rules, retries, failovers, and fault injections. It also addresses security, a critical aspect in the microservices world, by implementing it at the infrastructure level, thereby reducing the burden on application code. Furthermore, Istio enhances observability in the system, allowing organizations to effectively monitor and troubleshoot their services. Despite the steep learning curve associated with Istio, the multitude of benefits it offers makes it a worthy investment for organizations. The control and flexibility it provides over microservices are unparalleled. With the growing adoption of microservices, the role of service meshes like Istio is becoming increasingly pivotal, ensuring reliable, secure operation of services, and providing the scalability required in today's dynamic business environment. In conclusion, Istio holds a significant position in the service mesh realm, offering a comprehensive solution for managing microservices at scale. It represents the ongoing evolution in service-to-service communication, driven by the need for more efficient, secure, and manageable solutions. The future of Istio and service mesh, in general, appears promising, with continuous research and development efforts aimed at strengthening and broadening their capabilities. References "What is a service mesh?" (Red Hat) "Istio - Connect, secure, control, and observe services." (Istio) "What is Istio?" (IBM Cloud) "Understanding the Basics of Service Mesh" (Container Journal)
I blogged about Java stream debugging in the past, but I skipped an important method that's worthy of a post of its own: peek. This blog post delves into the practicalities of using peek() to debug Java streams, complete with code samples and common pitfalls. Understanding Java Streams Java Streams represent a significant shift in how Java developers work with collections and data processing, introducing a functional approach to handling sequences of elements. Streams facilitate declarative processing of collections, enabling operations such as filter, map, reduce, and more in a fluent style. This not only makes the code more readable but also more concise compared to traditional iterative approaches. A Simple Stream Example To illustrate, consider the task of filtering a list of names to only include those that start with the letter "J" and then transforming each name into uppercase. Using the traditional approach, this might involve a loop and some "if" statements. However, with streams, this can be accomplished in a few lines: List<String> names = Arrays.asList("John", "Jacob", "Edward", "Emily"); // Convert list to stream List<String> filteredNames = names.stream() // Filter names that start with "J" .filter(name -> name.startsWith("J")) // Convert each name to uppercase .map(String::toUpperCase) // Collect results into a new list .collect(Collectors.toList()); System.out.println(filteredNames); Output: [JOHN, JACOB] This example demonstrates the power of Java streams: by chaining operations together, we can achieve complex data transformations and filtering with minimal, readable code. It showcases the declarative nature of streams, where we describe what we want to achieve rather than detailing the steps to get there. What Is the peek() Method? At its core, peek() is a method provided by the Stream interface, allowing developers a glance into the elements of a stream without disrupting the flow of its operations. The signature of peek() is as follows: Stream<T> peek(Consumer<? super T> action) It accepts a Consumer functional interface, which means it performs an action on each element of the stream without altering them. The most common use case for peek() is logging the elements of a stream to understand the state of data at various points in the stream pipeline. To understand peek, let's look at a sample similar to the previous one: List<String> collected = Stream.of("apple", "banana", "cherry") .filter(s -> s.startsWith("a")) .collect(Collectors.toList()); System.out.println(collected); This code filters a list of strings, keeping only the ones that start with "a". While it's straightforward, understanding what happens during the filter operation is not visible. Debugging With peek() Now, let's incorporate peek() to gain visibility into the stream: List<String> collected = Stream.of("apple", "banana", "cherry") .peek(System.out::println) // Logs all elements .filter(s -> s.startsWith("a")) .peek(System.out::println) // Logs filtered elements .collect(Collectors.toList()); System.out.println(collected); By adding peek() both before and after the filter operation, we can see which elements are processed and how the filter impacts the stream. This visibility is invaluable for debugging, especially when the logic within the stream operations becomes complex. We can't step over stream operations with the debugger, but peek() provides a glance into the code that is normally obscured from us. Uncovering Common Bugs With peek() Filtering Issues Consider a scenario where a filter condition is not working as expected: List<String> collected = Stream.of("apple", "banana", "cherry", "Avocado") .filter(s -> s.startsWith("a")) .collect(Collectors.toList()); System.out.println(collected); Expected output might be ["apple"], but let's say we also wanted "Avocado" due to a misunderstanding of the startsWith method's behavior. Since "Avocado" is spelled with an upper case "A" this code will return false: Avocado".startsWith("a"). Using peek(), we can observe the elements that pass the filter: List<String> debugged = Stream.of("apple", "banana", "cherry", "Avocado") .peek(System.out::println) .filter(s -> s.startsWith("a")) .peek(System.out::println) .collect(Collectors.toList()); System.out.println(debugged); Large Data Sets In scenarios involving large datasets, directly printing every element in the stream to the console for debugging can quickly become impractical. It can clutter the console and make it hard to spot the relevant information. Instead, we can use peek() in a more sophisticated way to selectively collect and analyze data without causing side effects that could alter the behavior of the stream. Consider a scenario where we're processing a large dataset of transactions, and we want to debug issues related to transactions exceeding a certain threshold: class Transaction { private String id; private double amount; // Constructor, getters, and setters omitted for brevity } List<Transaction> transactions = // Imagine a large list of transactions // A placeholder for debugging information List<Transaction> highValueTransactions = new ArrayList<>(); List<Transaction> processedTransactions = transactions.stream() // Filter transactions above a threshold .filter(t -> t.getAmount() > 5000) .peek(t -> { if (t.getAmount() > 10000) { // Collect only high-value transactions for debugging highValueTransactions.add(t); } }) .collect(Collectors.toList()); // Now, we can analyze high-value transactions separately, without overloading the console System.out.println("High-value transactions count: " + highValueTransactions.size()); In this approach, peek() is used to inspect elements within the stream conditionally. High-value transactions that meet a specific criterion (e.g., amount > 10,000) are collected into a separate list for further analysis. This technique allows for targeted debugging without printing every element to the console, thereby avoiding performance degradation and clutter. Addressing Side Effects Streams shouldn't have side effects. In fact, such side effects would break the stream debugger in IntelliJ which I have discussed in the past. It's crucial to note that while collecting data for debugging within peek() avoids cluttering the console, it does introduce a side effect to the stream operation, which goes against the recommended use of streams. Streams are designed to be side-effect-free to ensure predictability and reliability, especially in parallel operations. Therefore, while the above example demonstrates a practical use of peek() for debugging, it's important to use such techniques judiciously. Ideally, this debugging strategy should be temporary and removed once the debugging session is completed to maintain the integrity of the stream's functional paradigm. Limitations and Pitfalls While peek() is undeniably a useful tool for debugging Java streams, it comes with its own set of limitations and pitfalls that developers should be aware of. Understanding these can help avoid common traps and ensure that peek() is used effectively and appropriately. Potential for Misuse in Production Code One of the primary risks associated with peek() is its potential for misuse in production code. Because peek() is intended for debugging purposes, using it to alter state or perform operations that affect the outcome of the stream can lead to unpredictable behavior. This is especially true in parallel stream operations, where the order of element processing is not guaranteed. Misusing peek() in such contexts can introduce hard-to-find bugs and undermine the declarative nature of stream processing. Performance Overhead Another consideration is the performance impact of using peek(). While it might seem innocuous, peek() can introduce a significant overhead, particularly in large or complex streams. This is because every action within peek() is executed for each element in the stream, potentially slowing down the entire pipeline. When used excessively or with complex operations, peek() can degrade performance, making it crucial to use this method judiciously and remove any peek() calls from production code after debugging is complete. Side Effects and Functional Purity As highlighted in the enhanced debugging example, peek() can be used to collect data for debugging purposes, but this introduces side effects to what should ideally be a side-effect-free operation. The functional programming paradigm, which streams are a part of, emphasizes purity and immutability. Operations should not alter state outside their scope. By using peek() to modify external state (even for debugging), you're temporarily stepping away from these principles. While this can be acceptable for short-term debugging, it's important to ensure that such uses of peek() do not find their way into production code, as they can compromise the predictability and reliability of your application. The Right Tool for the Job Finally, it's essential to recognize that peek() is not always the right tool for every debugging scenario. In some cases, other techniques such as logging within the operations themselves, using breakpoints and inspecting variables in an IDE, or writing unit tests to assert the behavior of stream operations might be more appropriate and effective. Developers should consider peek() as one tool in a broader debugging toolkit, employing it when it makes sense and opting for other strategies when they offer a clearer or more efficient path to identifying and resolving issues. Navigating the Pitfalls To navigate these pitfalls effectively: Reserve peek() strictly for temporary debugging purposes. If you have a linter as part of your CI tools, it might make sense to add a rule that blocks code from invoking peek(). Always remove peek() calls from your code before committing it to your codebase, especially for production deployments. Be mindful of performance implications and the potential introduction of side effects. Consider alternative debugging techniques that might be more suited to your specific needs or the particular issue you're investigating. By understanding and respecting these limitations and pitfalls, developers can leverage peek() to enhance their debugging practices without falling into common traps or inadvertently introducing problems into their codebases. Final Thoughts The peek() method offers a simple yet effective way to gain insights into Java stream operations, making it a valuable tool for debugging complex stream pipelines. By understanding how to use peek() effectively, developers can avoid common pitfalls and ensure their stream operations perform as intended. As with any powerful tool, the key is to use it wisely and in moderation. The true value of peek() is in debugging massive data sets, these elements are very hard to analyze even with dedicated tools. By using peek() we can dig into the said data set and understand the source of the issue programmatically.
In this blog, you will learn how to implement Retrieval Augmented Generation (RAG) using Weaviate, LangChain4j, and LocalAI. This implementation allows you to ask questions about your documents using natural language. Enjoy! 1. Introduction In the previous post, Weaviate was used as a vector database in order to perform a semantic search. The source documents used are two Wikipedia documents. The discography and list of songs recorded by Bruce Springsteen are the documents used. The interesting part of these documents is that they contain facts and are mainly in a table format. Parts of these documents are converted to Markdown in order to have a better representation. The Markdown files are embedded in Collections in Weaviate. The result was amazing: all questions asked, resulted in the correct answer to the question. That is, the correct segment was returned. You still needed to extract the answer yourself, but this was quite easy. However, can this be solved by providing the Weaviate search results to an LLM (Large Language Model) by creating the right prompt? Will the LLM be able to extract the correct answers to the questions? The setup is visualized in the graph below: The documents are embedded and stored in Weaviate; The question is embedded and a semantic search is performed using Weaviate; Weaviate returns the semantic search results; The result is added to a prompt and fed to LocalAI which runs an LLM using LangChain4j; The LLM returns the answer to the question. Weaviate also supports RAG, so why bother using LocalAI and LangChain4j? Unfortunately, Weaviate does not support integration with LocalAI and only cloud LLMs can be used. If your documents contain sensitive information or information you do not want to send to a cloud-based LLM, you need to run a local LLM and this can be done using LocalAI and LangChain4j. If you want to run the examples in this blog, you need to read the previous blog. The sources used in this blog can be found on GitHub. 2. Prerequisites The prerequisites for this blog are: Basic knowledge of embedding and vector stores; Basic Java knowledge, Java 21 is used; Basic knowledge of Docker; Basic knowledge of LangChain4j; You need Weaviate and the documents need to be embedded, see the previous blog on how to do so; You need LocalAI if you want to run the examples, see a previous blog on how you can make use of LocalAI. Version 2.2.0 is used for this blog. If you want to learn more about RAG, read this blog. 3. Create the Setup Before getting started, there is some setup to do. 3.1 Setup LocalAI LocalAI must be running and configured. How to do so is explained in the blog Running LLM’s Locally: A Step-by-Step Guide. 3.2 Setup Weaviate Weaviate must be started. The only difference with the Weaviate blog is that you will run it on port 8081 instead of port 8080. This is because LocalAI is already running on port 8080. Start the compose file from the root of the repository. Shell $ docker compose -f docker/compose-embed-8081.yaml Run class EmbedMarkdown in order to embed the documents (change the port to 8081!). Three collections are created: CompilationAlbum: a list of all compilation albums of Bruce Springsteen; Song: a list of all songs by Bruce Springsteen; StudioAlbum: a list of all studio albums of Bruce Springsteen. 4. Implement RAG 4.1 Semantic Search The first part of the implementation is based on the semantic search implementation of class SearchCollectionNearText. It is assumed here, that you know in which collection (argument className) to search for. In the previous post, you noticed that strictly spoken, you do not need to know which collection to search for. However, at this moment, it makes the implementation a bit easier and the result remains identical. The code will take the question and with the help of NearTextArgument, the question will be embedded. The GraphQL API of Weaviate is used to perform the search. Java private static void askQuestion(String className, Field[] fields, String question, String extraInstruction) { Config config = new Config("http", "localhost:8081"); WeaviateClient client = new WeaviateClient(config); Field additional = Field.builder() .name("_additional") .fields(Field.builder().name("certainty").build(), // only supported if distance==cosine Field.builder().name("distance").build() // always supported ).build(); Field[] allFields = Arrays.copyOf(fields, fields.length + 1); allFields[fields.length] = additional; // Embed the question NearTextArgument nearText = NearTextArgument.builder() .concepts(new String[]{question}) .build(); Result<GraphQLResponse> result = client.graphQL().get() .withClassName(className) .withFields(allFields) .withNearText(nearText) .withLimit(1) .run(); if (result.hasErrors()) { System.out.println(result.getError()); return; } ... 4.2 Create Prompt The result of the semantic search needs to be fed to the LLM including the question itself. A prompt is created which will instruct the LLM to answer the question using the result of the semantic search. Also, the option to add extra instructions is implemented. Later on, you will see what to do with that. Java private static String createPrompt(String question, String inputData, String extraInstruction) { return "Answer the following question: " + question + "\n" + extraInstruction + "\n" + "Use the following data to answer the question: " + inputData; } 4.3 Use LLM The last thing to do is to feed the prompt to the LLM and print the question and answer to the console. Java private static void askQuestion(String className, Field[] fields, String question, String extraInstruction) { ... ChatLanguageModel model = LocalAiChatModel.builder() .baseUrl("http://localhost:8080") .modelName("lunademo") .temperature(0.0) .build(); String answer = model.generate(createPrompt(question, result.getResult().getData().toString(), extraInstruction)); System.out.println(question); System.out.println(answer); } 4.4 Questions The questions to be asked are the same as in the previous posts. They will invoke the code above. Java public static void main(String[] args) { askQuestion(Song.NAME, Song.getFields(), "on which album was \"adam raised a cain\" originally released?", ""); askQuestion(StudioAlbum.NAME, StudioAlbum.getFields(), "what is the highest chart position of \"Greetings from Asbury Park, N.J.\" in the US?", ""); askQuestion(CompilationAlbum.NAME, CompilationAlbum.getFields(), "what is the highest chart position of the album \"tracks\" in canada?", ""); askQuestion(Song.NAME, Song.getFields(), "in which year was \"Highway Patrolman\" released?", ""); askQuestion(Song.NAME, Song.getFields(), "who produced \"all or nothin' at all?\"", ""); } The complete source code can be viewed here. 5. Results Run the code and the result is the following: On which album was “Adam Raised a Cain” originally released?The album “Darkness on the Edge of Town” was originally released in 1978, and the song “Adam Raised a Cain” was included on that album. What is the highest chart position of “Greetings from Asbury Park, N.J.” in the US?The highest chart position of “Greetings from Asbury Park, N.J.” in the US is 60. What is the highest chart position of the album “Tracks” in Canada?Based on the provided data, the highest chart position of the album “Tracks” in Canada is -. This is because the data does not include any Canadian chart positions for this album. In which year was “Highway Patrolman” released?The song “Highway Patrolman” was released in 1982. Who produced “all or nothin’ at all?”The song “All or Nothin’ at All” was produced by Bruce Springsteen, Roy Bittan, Jon Landau, and Chuck Plotkin. All answers to the questions are correct. The most important job has been done in the previous post, where embedding the documents in the correct way, resulted in finding the correct segments. An LLM is able to extract the answer to the question when it is fed with the correct data. 6. Caveats During the implementation, I ran into some strange behavior which is quite important to know when you are starting to implement your use case. 6.1 Format of Weaviate Results The Weaviate response contains a GraphQLResponse object, something like the following: JSON GraphQLResponse( data={ Get={ Songs=[ {_additional={certainty=0.7534831166267395, distance=0.49303377}, originalRelease=Darkness on the Edge of Town, producers=Jon Landau Bruce Springsteen Steven Van Zandt (assistant), song="Adam Raised a Cain", writers=Bruce Springsteen, year=1978} ] } }, errors=null) In the code, the data part is used to add to the prompt. Java String answer = model.generate(createPrompt(question, result.getResult().getData().toString(), extraInstruction)); What happens when you add the response as-is to the prompt? Java String answer = model.generate(createPrompt(question, result.getResult().toString(), extraInstruction)); Running the code returns the following wrong answer for question 3 and some unnecessary additional information for question 4. The other questions are answered correctly. What is the highest chart position of the album “Tracks” in Canada?Based on the provided data, the highest chart position of the album “Tracks” in Canada is 50. In which year was “Highway Patrolman” released?Based on the provided GraphQLResponse, “Highway Patrolman” was released in 1982.who produced “all or nothin’ at all?” 6.2 Format of Prompt The code contains functionality to add extra instructions to the prompt. As you have probably noticed, this functionality is not used. Let’s see what happens when you remove this from the prompt. The createPrompt method becomes the following (I did not remove everything so that only a minor code change is needed). Java private static String createPrompt(String question, String inputData, String extraInstruction) { return "Answer the following question: " + question + "\n" + "Use the following data to answer the question: " + inputData; } Running the code adds some extra information to the answer to question 3 which is not entirely correct. It is correct that the album has chart positions for the United States, United Kingdom, Germany, and Sweden. It is not correct that the album reached the top 10 in the UK and US charts. All other questions are answered correctly. What is the highest chart position of the album “Tracks” in Canada?Based on the provided data, the highest chart position of the album “Tracks” in Canada is not specified. The data only includes chart positions for other countries such as the United States, United Kingdom, Germany, and Sweden. However, the album did reach the top 10 in the UK and US charts. It remains a bit brittle when using an LLM. You cannot always trust the answer it is given. Changing the prompt accordingly seems to be possible to minimize the hallucinations of an LLM. It is therefore important that you collect feedback from your users in order to identify when an LLM seems to hallucinate. This way, you will be able to improve the responses to the users. An interesting blog is written by Fiddler which addresses this kind of issue. 7. Conclusion In this blog, you learned how to implement RAG using Weaviate, LangChain4j, and LocalAI. The results are quite amazing. Embedding documents the right way, filtering the results, and feeding them to an LLM is a very powerful combination that can be used in many use cases.
The relentless advancement of artificial intelligence (AI) technology reshapes our world, with Large Language Models (LLMs) spearheading this transformation. The emergence of the LLM-4 architecture signifies a pivotal moment in AI development, heralding new capabilities in language processing that challenge the boundaries between human and machine intelligence. This article provides a comprehensive exploration of LLM-4 architectures, detailing their innovations, applications, and broader implications for society and technology. Unveiling LLM-4 Architectures LLM-4 architectures represent the cutting edge in the evolution of large language models, building upon their predecessors' foundations to achieve new levels of performance and versatility. These models excel in interpreting and generating human language, driven by enhancements in their design and training methodologies. The core innovation of LLM-4 models lies in their advanced neural networks, particularly transformer-based structures, which allow for efficient and effective processing of large data sequences. Unlike traditional models that process data sequentially, transformers handle data in parallel, significantly enhancing learning speed and comprehension. To illustrate, consider the Python implementation of a transformer encoder layer below. This code reflects the intricate mechanisms that enable LLM-4 models to learn and adapt with remarkable proficiency: Python import torch import torch.nn as nn class TransformerEncoderLayer(nn.Module): def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1): super(TransformerEncoderLayer, self).__init__() self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout) self.linear1 = nn.Linear(d_model, dim_feedforward) self.dropout = nn.Dropout(dropout) self.linear2 = nn.Linear(dim_feedforward, d_model) self.norm1 = nn.LayerNorm(d_model) self.norm2 = nn.LayerNorm(d_model) self.dropout1 = nn.Dropout(dropout) self.dropout2 = nn.Dropout(dropout) def forward(self, src): src2 = self.self_attn(src, src, src)[0] src = src + self.dropout1(src2) src = self.norm1(src) src2 = self.linear2(self.dropout(self.linear1(src))) src = src + self.dropout2(src2) src = self.norm2(src) return src This encoder layer serves as a fundamental building block for the transformer architecture, facilitating deep learning processes that underpin the intelligence of LLM-4 models. Broadening Horizons: Applications of LLM-4 The versatility of LLM-4 architectures opens a plethora of applications across various sectors. In natural language processing, these models enhance translation, summarization, and content generation, bridging communication gaps and fostering global collaboration. Beyond these traditional uses, LLM-4 models are instrumental in creating interactive AI agents capable of nuanced conversation and making strides in customer service, therapy, education, and entertainment. Moreover, LLM-4 architectures extend their utility to the realm of coding, offering predictive text generation and debugging assistance, thus revolutionizing software development practices. Their ability to process and generate complex language structures also finds applications in legal analysis, financial forecasting, and research, where they can synthesize vast amounts of information into coherent, actionable insights. Navigating the Future: Implications of LLM-4 The ascent of LLM-4 architectures raises critical considerations regarding their impact on society. As these models blur the line between human and machine-generated content, they prompt discussions on authenticity, intellectual property, and the ethics of AI. Furthermore, their potential to automate complex tasks necessitates a reevaluation of workforce dynamics, emphasizing the need for policies that address job displacement and skill evolution. The development of LLM-4 architectures also underscores the importance of robust AI governance. Ensuring transparency, accountability, and fairness in these models is paramount to harnessing their benefits while mitigating associated risks. As we chart the course for future AI advancements, the lessons learned from LLM-4 development will be instrumental in guiding responsible innovation. Conclusion The emergence of LLM-4 architectures marks a watershed moment in AI development, signifying profound advancements in machine intelligence. These models not only enhance our technological capabilities but also challenge us to contemplate their broader implications. As we delve deeper into the potential of LLM-4 architectures, it is imperative to foster an ecosystem that promotes ethical use, ongoing learning, and societal well-being, ensuring that AI continues to serve as a force for positive transformation.
User-defined functions (UDFs) are a very useful feature supported in SQL++ (UDF documentation). Couchbase 7.6 introduces improvements that allow for more debuggability and visibility into UDF execution. This blog will explore two new features in Couchbase 7.6 in the world of UDFs: Profiling for SQL++ statements executed in JavaScript UDFs EXPLAIN FUNCTION to access query plans of SQL++ statements within UDFs The examples in this blog require the travel-sample dataset to be installed. Documentation to install sample buckets Profiling SQL++ Executed in JavaScript UDFs Query profiling is a debuggability feature that SQL++ offers. When profiling is enabled for a statement’s execution, the result of the request includes a detailed execution tree with timing and metrics of each step of the statement’s execution. In addition to the profiling information being returned in the results of the statement, it can also be accessed for the request in the system:active_requests and system:completed_requests system keyspaces. To dive deeper into request profiling, see request profiling in SQL++. In Couchbase 7.0, profiling was included for subqueries. This included profiling subqueries that were within Inline UDFs. However, in versions before Couchbase 7.6, profiling was not extended to SQL++ statements within JavaScript UDFs. In earlier versions, to profile statements within a JavaScript UDF, the user would be required to open up the function’s definition, individually run each statement within the UDF, and collect their profiles. This additional step will no longer be needed in 7.6.0! Now, when profiling is enabled, if the statement contains JavaScript UDF execution, profiles for all SQL++ statements executed in the UDF will also be collected. This UDF-related profiling information will be available in the request output, system:active_requests and system:completed_requests system keyspaces as well. Example 1 Create a JavaScript UDF “js1” in a global library “lib1” via the REST endpoint or via the UI. JavaScript function js1() { var query = SELECT * FROM default:`travel-sample`.inventory.airline LIMIT 1; var res = []; for (const row of query) { res.push(row); } query.close() return res; } Create the corresponding SQL++ function. SQL CREATE FUNCTION js1() LANGUAGE JAVASCRIPT AS "js1" AT "lib1"; Execute the UDF with profiling enabled. SQL EXECUTE FUNCTION js1(); The response to the statement above will contain the following: In the profile section of the returned response, the executionTimings subsection contains a field ~udfStatements. ~udfStatements: An array of profiling information that contains an entry for every SQL++ statement within the JavaScript UDF Every entry within the ~udfStatements section contains: executionTimings: This is the execution tree for the statement. It has metrics and timing information for every step of the statement’s execution. statement: The statement string function: This is the name of the function where the statement was executed and is helpful to identify the UDF that executed the statement when there are nested UDF executions. JavaScript { "requestID": "2c5576b5-f01d-445f-a35b-2213c606f394", "signature": null, "results": [ [ { "airline": { "callsign": "MILE-AIR", "country": "United States", "iata": "Q5", "icao": "MLA", "id": 10, "name": "40-Mile Air", "type": "airline" } } ] ], "status": "success", "metrics": { "elapsedTime": "20.757583ms", "executionTime": "20.636792ms", "resultCount": 1, "resultSize": 310, "serviceLoad": 2 }, "profile": { "phaseTimes": { "authorize": "12.835µs", "fetch": "374.667µs", "instantiate": "27.75µs", "parse": "251.708µs", "plan": "9.125µs", "primaryScan": "813.249µs", "primaryScan.GSI": "813.249µs", "project": "5.541µs", "run": "27.925833ms", "stream": "26.375µs" }, "phaseCounts": { "fetch": 1, "primaryScan": 1, "primaryScan.GSI": 1 }, "phaseOperators": { "authorize": 2, "fetch": 1, "primaryScan": 1, "primaryScan.GSI": 1, "project": 1, "stream": 1 }, "cpuTime": "468.626µs", "requestTime": "2023-12-04T20:30:00.369+05:30", "servicingHost": "127.0.0.1:8091", "executionTimings": { "#operator": "Authorize", "#planPreparedTime": "2023-12-04T20:30:00.369+05:30", "#stats": { "#phaseSwitches": 4, "execTime": "1.918µs", "servTime": "1.125µs" }, "privileges": { "List": [] }, "~child": { "#operator": "Sequence", "#stats": { "#phaseSwitches": 2, "execTime": "2.208µs" }, "~children": [ { "#operator": "ExecuteFunction", "#stats": { "#itemsOut": 1, "#phaseSwitches": 4, "execTime": "22.375µs", "kernTime": "20.271708ms" }, "identity": { "name": "js1", "namespace": "default", "type": "global" } }, { "#operator": "Stream", "#stats": { "#itemsIn": 1, "#itemsOut": 1, "#phaseSwitches": 2, "execTime": "26.375µs" }, "serializable": true } ] }, "~udfStatements": [ { "executionTimings": { "#operator": "Authorize", "#stats": { "#phaseSwitches": 4, "execTime": "2.626µs", "servTime": "7.166µs" }, "privileges": { "List": [ { "Priv": 7, "Props": 0, "Target": "default:travel-sample.inventory.airline" } ] }, "~child": { "#operator": "Sequence", "#stats": { "#phaseSwitches": 2, "execTime": "4.375µs" }, "~children": [ { "#operator": "PrimaryScan3", "#stats": { "#itemsIn": 1, "#itemsOut": 1, "#phaseSwitches": 7, "execTime": "22.082µs", "kernTime": "1.584µs", "servTime": "791.167µs" }, "bucket": "travel-sample", "index": "def_inventory_airline_primary", "index_projection": { "primary_key": true }, "keyspace": "airline", "limit": "1", "namespace": "default", "optimizer_estimates": { "cardinality": 187, "cost": 45.28617059639748, "fr_cost": 12.1780009122802, "size": 12 }, "scope": "inventory", "using": "gsi" }, { "#operator": "Fetch", "#stats": { "#itemsIn": 1, "#itemsOut": 1, "#phaseSwitches": 10, "execTime": "18.376µs", "kernTime": "797.542µs", "servTime": "356.291µs" }, "bucket": "travel-sample", "keyspace": "airline", "namespace": "default", "optimizer_estimates": { "cardinality": 187, "cost": 192.01699202888378, "fr_cost": 24.89848658838975, "size": 204 }, "scope": "inventory" }, { "#operator": "InitialProject", "#stats": { "#itemsIn": 1, "#itemsOut": 1, "#phaseSwitches": 7, "execTime": "5.541µs", "kernTime": "1.1795ms" }, "discard_original": true, "optimizer_estimates": { "cardinality": 187, "cost": 194.6878862611588, "fr_cost": 24.912769445246838, "size": 204 }, "preserve_order": true, "result_terms": [ { "expr": "self", "star": true } ] }, { "#operator": "Limit", "#stats": { "#itemsIn": 1, "#itemsOut": 1, "#phaseSwitches": 4, "execTime": "6.25µs", "kernTime": "333ns" }, "expr": "1", "optimizer_estimates": { "cardinality": 1, "cost": 24.927052302103924, "fr_cost": 24.927052302103924, "size": 204 } }, { "#operator": "Receive", "#stats": { "#phaseSwitches": 3, "execTime": "10.324833ms", "kernTime": "792ns", "state": "running" } } ] } }, "statement": "SELECT * FROM default:`travel-sample`.inventory.airline LIMIT 1;", "function": "default:js1" } ], "~versions": [ "7.6.0-N1QL", "7.6.0-1847-enterprise" ] } } } Query Plans With EXPLAIN FUNCTION SQL++ offers another wonderful capability to access the plan of a statement with the EXPLAIN statement. However, the EXPLAIN statement does not extend to plans of statements within UDFs, neither inline nor JavaScript UDFs. In earlier versions, to analyze the query plans for SQL++ within a UDF, it would require the user to open the function’s definition and individually run an EXPLAIN on all the statements within the UDF. These extra steps will be minimized in Couchbase 7.6 with the introduction of a new statement: EXPLAIN FUNCTION. This statement does exactly what EXPLAIN does, but for SQL++ statements within a UDF. Let’s explore how to use the EXPLAIN FUNCTION statement! Syntax explain_function ::= 'EXPLAIN' 'FUNCTION' function function refers to the name of the function. For more detailed information on syntax, please check out the documentation. Prerequisites To execute EXPLAIN FUNCTION, the user requires the correct RBAC permissions. To run EXPLAIN FUNCTION on a UDF, the user must have sufficient RBAC permissions to execute the function. The user must also have the necessary RBAC permissions to execute the SQL++ statements within the UDF function body as well. For more information, refer to the documentation regarding roles supported in Couchbase. Inline UDF EXPLAIN FUNCTION on an inline UDF will return the query plans of all the subqueries within its definition (see inline function documentation). Example 2: EXPLAIN FUNCTION on an Inline Function Create an inline UDF and run EXPLAIN FUNCTION on it. SQL CREATE FUNCTION inline1() { ( SELECT * FROM default:`travel-sample`.inventory.airport WHERE city = "Zachar Bay" ) }; SQL EXPLAIN FUNCTION inline1(); The results of the above statement will contain: function: The name of the function on which EXPLAIN FUNCTION was run plans: An array of plan information that contains an entry for every subquery within the inline UDF JavaScript { "function": "default:inline1", "plans": [ { "cardinality": 1.1176470588235294, "cost": 25.117642854609013, "plan": { "#operator": "Sequence", "~children": [ { "#operator": "IndexScan3", "bucket": "travel-sample", "index": "def_inventory_airport_city", "index_id": "2605c88c115dd3a2", "index_projection": { "primary_key": true }, "keyspace": "airport", "namespace": "default", "optimizer_estimates": { "cardinality": 1.1176470588235294, "cost": 12.200561852726496, "fr_cost": 12.179450078755286, "size": 12 }, "scope": "inventory", "spans": [ { "exact": true, "range": [ { "high": "\\"Zachar Bay\\"", "inclusion": 3, "index_key": "`city`", "low": "\\"Zachar Bay\\"" } ] } ], "using": "gsi" }, { "#operator": "Fetch", "bucket": "travel-sample", "keyspace": "airport", "namespace": "default", "optimizer_estimates": { "cardinality": 1.1176470588235294, "cost": 25.082370508382763, "fr_cost": 24.96843677065826, "size": 249 }, "scope": "inventory" }, { "#operator": "Parallel", "~child": { "#operator": "Sequence", "~children": [ { "#operator": "Filter", "condition": "((`airport`.`city`) = \\"Zachar Bay\\")", "optimizer_estimates": { "cardinality": 1.1176470588235294, "cost": 25.100006681495888, "fr_cost": 24.98421650449632, "size": 249 } }, { "#operator": "InitialProject", "discard_original": true, "optimizer_estimates": { "cardinality": 1.1176470588235294, "cost": 25.117642854609013, "fr_cost": 24.99999623833438, "size": 249 }, "result_terms": [ { "expr": "self", "star": true } ] } ] } } ] }, "statement": "select self.* from `default`:`travel-sample`.`inventory`.`airport` where ((`airport`.`city`) = \\"Zachar Bay\\")" } ] } JavaScript UDF SQL++ statements within JavaScript UDFs can be of two types as listed below. EXPLAIN FUNCTION works differently based on the way the SQL++ statement is called. Refer to the documentation to learn more about calling SQL++ in JavaScript functions. 1. Embedded SQL++ Embedded SQL++ is “embedded” in the function body and its detection is handled by the JavaScript transpiler. EXPLAIN FUNCTION can return query plans for embedded SQL++ statements. 2. SQL++ Executed by the N1QL() Function Call SQL++ can also be executed by passing a statement in the form of a string as an argument to the N1QL() function. When parsing the function for potential SQL++ statements to run the EXPLAIN on, it is difficult to get the dynamic string in the function argument. This can only be reliably resolved at runtime. With this reasoning, EXPLAIN FUNCTION does not return the query plans for SQL++ statements executed via N1QL() calls, but instead, returns the line numbers where the N1QL() function calls have been made. This line number is calculated from the beginning of the function definition. The user can then map the line numbers in the actual function definition and investigate further. Example 3: EXPLAIN FUNCTION on an External JavaScript Function Create a JavaScript UDF “js2” in a global library “lib1” via the REST endpoint or via the UI. JavaScript function js2() { // SQL++ executed by a N1QL() function call var query1 = N1QL("UPDATE default:`travel-sample` SET test = 1 LIMIT 1"); // Embedded SQL++ var query2 = SELECT * FROM default:`travel-sample` LIMIT 1; var res = []; for (const row of query2) { res.push(row); } query2.close() return res; } Create the corresponding SQL++ function. SQL CREATE FUNCTION js2() LANGUAGE JAVASCRIPT AS "js2" AT "lib1"; Run EXPLAIN FUNCTION on the SQL++ function. SQL EXPLAIN FUNCTION js2; The results of the statement above will contain: function: The name of the function on which EXPLAIN FUNCTION was run line_numbers: An array of line numbers calculated from the beginning of the JavaScript function definition where there are N1QL() function calls plans: An array of plan information that contains an entry for every embedded SQL++ statement within the JavaScript UDF JavaScript { "function": "default:js2", "line_numbers": [ 4 ], "plans": [ { "cardinality": 1, "cost": 25.51560885530435, "plan": { "#operator": "Authorize", "privileges": { "List": [ { "Target": "default:travel-sample", "Priv": 7, "Props": 0 } ] }, "~child": { "#operator": "Sequence", "~children": [ { "#operator": "Sequence", "~children": [ { "#operator": "Sequence", "~children": [ { "#operator": "PrimaryScan3", "index": "def_primary", "index_projection": { "primary_key": true }, "keyspace": "travel-sample", "limit": "1", "namespace": "default", "optimizer_estimates": { "cardinality": 31591, "cost": 5402.279801258844, "fr_cost": 12.170627071041082, "size": 11 }, "using": "gsi" }, { "#operator": "Fetch", "keyspace": "travel-sample", "namespace": "default", "optimizer_estimates": { "cardinality": 31591, "cost": 46269.39474997121, "fr_cost": 25.46387878667884, "size": 669 } }, { "#operator": "Parallel", "~child": { "#operator": "Sequence", "~children": [ { "#operator": "InitialProject", "discard_original": true, "optimizer_estimates": { "cardinality": 31591, "cost": 47086.49704894546, "fr_cost": 25.489743820991595, "size": 669 }, "preserve_order": true, "result_terms": [ { "expr": "self", "star": true } ] } ] } } ] }, { "#operator": "Limit", "expr": "1", "optimizer_estimates": { "cardinality": 1, "cost": 25.51560885530435, "fr_cost": 25.51560885530435, "size": 669 } } ] }, { "#operator": "Stream", "optimizer_estimates": { "cardinality": 1, "cost": 25.51560885530435, "fr_cost": 25.51560885530435, "size": 669 }, "serializable": true } ] } }, "statement": "SELECT * FROM default:`travel-sample` LIMIT 1 ;" } ] } Constraints If the N1QL() function has been aliased in a JavaScript function definition, EXPLAIN FUNCTION will not be able to return the line numbers where this aliased function was called.Example of such a function definition: JavaScript function js3() { var alias = N1QL; var q = alias("SELECT 1"); } If the UDF contains nested UDF executions, EXPLAIN FUNCTION does not support generating the query plans of SQL++ statements within these nested UDFs. Summary Couchbase 7.6 introduces new features to debug UDFs which will help users peek into UDF execution easily. Helpful References 1. Javascript UDFs: A guide to JavaScript UDFs Creating an external UDF 2. EXPLAIN statement
Biology insists — and common sense says — that I've started to become that old fogey I used to laugh at in my younger days. ...THIRD YORKSHIREMAN:Well, of course, we had it tough. We used to 'ave to get up out of shoebox at twelve o'clock at night and lick road clean wit' tongue. We had two bits of cold gravel, worked twenty-four hours a day at mill for sixpence every four years, and when we got home our Dad would slice us in two wit' bread knife. FOURTH YORKSHIREMAN:Right. I had to get up in the morning at ten o'clock at night half an hour before I went to bed, drink a cup of sulphuric acid, work twenty-nine hours a day down mill, and pay mill owner for permission to come to work, and when we got home, our Dad and our mother would kill us and dance about on our graves singing Hallelujah. FIRST YORKSHIREMAN:And you try and tell the young people of today that ... they won't believe it. - Monty Python, Four Yorkshiremen Now that I'm now that old, grizzled software veteran whom I feared in earlier times, I've reflected on how the job has changed — definitely for the better — and how engineers today (me included) are so incredibly lucky to be working with the tools available today. Image source: "Coding w/ Gedit" by Matrixizationized, licensed under CC BY 2.0 Those older days? Not much to get excited about. Text Editors Unbeknownst to modern-day software engineers, prehistoric software engineering did not have IDEs to help: no Visual Studio, IntelliJ, VSCode, Eclipse, Atom, nothing. No autocomplete. No syntax checking. No code navigation. No integrated debugging. Nada. Instead, you wrote code in (OMG) text editors like vi or emacs or even Windows Notepad (or edlin, when desperate), enhanced by other tools (who's run lint from the command line recently?). And similar to debating IDEs today, then we debated text editors. Engineers may be able to customize those that were configurable, but in the end, it's a damn text editor. Wrap your head around it. Tabs vs. Spaces How many characters to indent has been vigorously debated since the dawn of structured programming - no Fortran's fixed positions, thank you very much - but have you ever debated the pros/cons of indenting with tabs vs. spaces? A senior engineer I worked with was adamant that tabs sped up compilation time due to fewer bytes, and insisted (demanded) that we do the same, IKYN. No supporting data was provided, but s/he who speaks the loudest wins. Hungarian Notation Hungarian Notation is (originally) a C/C++ coding convention to assist data type identification through naming, allowing engineers to infer the underlying data types without digging through code, e.g. szName is a null-terminated string, and usCount an unsigned short integer. Method names became overly convoluted as the data types for the return value and each parameter is baked in; e.g., uiszCountUsersByDomain accepts a null-terminated string and returns an unsigned integer. When the function accepted more than a trivial number of parameters, its name became unreadable and meaningless, so I typically only included the return type. However cryptic, Hungarian Notation was very useful in pre-IDE days and I used it extensively. With shame, I admit to initially applying it to Java but quickly learned the error of my ways. Paper-Based Code Reviews Perhaps difficult to believe, but code reviews predate pull requests and your collaboration tools of choice: physical, in-person, paper-based code reviews. Each engineer printed out the code to review and marked it up: questions, concerns, comments, etc. The code occurred in real-time with all people present in the same room, nothing virtual. The author took hard-written notes — remember, no laptops, notebooks, tablets — and returned to their desk to make the agreed-upon changes. Finito. Single Display Monitor Image source: "Big old monitor" by Harry Wood, licensed under CC BY-SA 2.0 Remember these old clunkers front-and-center on your desk? And let's not mention the incredibly unacceptable screen resolutions. Or how you were restricted to using a single monitor due to hardware limitations, software limitations, hardware cost, physical space, electric consumption, or whatever? Ugh! In my world, there is no such thing as too much screen real estate: multiple monitors, large screen sizes, higher resolution, virtual desktops - I want more, more, more! At Dell, I had four monitors (19x12, meh) and four virtual desktops for 16 total screens. My home work environment has two 27" 4K monitors plus my MacBook screen, far superior to what's available on the rare trips to the office. Even travel is tough because I'm limited to my MacBook's screen (though I do hook up to the hotel's TV when possible). Boy, life is hard! Primitive Networks Image source: "modem and phone" by BryanAlexande, licensed under CC BY 2.0 Before the adoption of TCP/IP as the de facto networking standard and the universal availability of the Internet, inter-computer communications were difficult. Businesses sometimes deployed site- or company-specific LANs, but rarely to external companies (only as-needed and at great expense). Dialup modems were all the rage at home until DSL became available in the late 1990s. Do you understand how exciting 9600 baud could be? No, you don't! The first distributed application I worked with had the app on the remote system automatically dial a modem to connect to the central system, send the data to be analyzed, download the generated results, and disconnect. Slow and effective, but it worked. . . until someone in Finland flipped a modem toggle and now the remote system can't connect. Weeks of checking phone lines, reviewing log files, and reinstalling applications until checking the modem. Oops. And Not To Be Forgotten What else was there? 8.3 file names. Floppy-based software installs with hand-entered license codes. Regularly occurring blue screens of death (much more occasional now). Primitive or non-existent security. Microsoft Radio to keep you entertained while waiting for paid support. No open source software: everything is purchased or written from scratch. I'm sure there's more. [Fortunately, I never wrote COBOL programs with punchcards, though a friend did during her training at Anderson Consulting. And make sure you don't drop your stack!] Oh, you kids have it so easy. . . and with that statement, I've finally become who my grandparents warned me about. Damn.
Knowledge graphs are a giant web of information where elements and ideas are linked to show how they are related in the real world. This is beyond databases that just store information. Knowledge graphs also store the connections between information. This makes knowledge graphs very useful in various fields. Here are a few examples: Search engines: Search engines use knowledge graphs to understand the relationships between search terms and real-world entities. A search for "French food" might not just surface recipes, but also information about French wine regions or famous French chefs, thanks to the connections embodied in the knowledge graph. Virtual assistants: Virtual assistants like Siri or Alexa rely on knowledge graphs to understand your requests and provide helpful responses. By knowing that "Eiffel Tower" is a landmark and "Paris" is a city, the assistant can answer your question about the Eiffel Tower's location. Machine learning applications: Machine learning algorithms can leverage knowledge graphs to improve their understanding of the world. A recommendation system, for example, can use a knowledge graph to connect movies with actors, directors, and genres. This allows to recommend similar movies based on past preferences. Large Language Models (LLMs): LLMs can benefit from knowledge graphs by accessing and processing all the information and connections that they have stored. This helps LLMs to generate more comprehensive and informative responses to our questions. Fraud detection: Knowledge graphs can be used to identify fraudulent activity by analyzing connections between entities. For example, a graph might flag a transaction as suspicious if it involves a new account linked to a known fraudulent IP address. Knowledge Graph Basics In a library, books may not just be shelved by category, but also cross-referenced. A book on Paris might be near French history books, but also connected to travel guides and works by Parisian authors. This web of connections is the essence of a knowledge graph. The basic building blocks of a knowledge graph contain: Nodes: These are the fundamental entities in the graph. They can be anything you can describe: physical objects (like the Eiffel Tower), abstract concepts (like democracy), events (like the French Revolution), or even people (like Marie Curie). Edges: These are the connections between nodes. They show how entities relate to each other. Edges are often labeled to specify the nature of the connection. Going back to our Paris example, the edge between "Paris" and "France" might have the label "capital of." Other labels could be "inhabitant of" (between Paris and Marie Curie) or "influenced by" (between French Revolution and democracy). Labels: These are crucial for understanding the edges. They provide context and meaning to the connections between nodes. Properties: Nodes and edges can have properties, which are additional attributes or metadata associated with them. For example, a person node might have properties such as "name," "age," "gender," etc., while an edge representing the relationship "is married to" might have properties like "start date" and "end date." Ontologies: These are blueprints for the knowledge graph. They define the types of entities allowed in the graph, the possible relationships between them, and the labels used for those relationships. In a library, again, there can be a specific classification system for books, defining sections, subsections, and how different categories of books can relate. An ontology sets the rules for how information is organised within the knowledge graph. Schema: Based on the ontology, a schema defines the types of entities, relationships, and properties allowed in the graph. It provides structure and consistency to the data, making it easier to query and analyze. Superpowers of a Knowledge Graph This web of relationships unlocks a unique power: machines can reason and infer new information based on what they "know" in the graph. Here are two examples below. Reasoning and Inference: The "Aha Moment" for Machines Assume a knowledge graph that stores information like "Paris is the capital of France" and "France is in Europe." While the graph might not explicitly state "Paris is in Europe," the connections between these entities allow a machine to reason towards to that conclusion. This "aha moment" is the essence of reasoning with knowledge graphs. Machines can analyze these connections and infer new information that isn't explicitly stated, expanding their understanding of the world. Example A travel recommendation system uses a knowledge graph to connect cities with tourist attractions and nearby landmarks. If a user expresses interest in visiting the Eiffel Tower, the system can reason using the knowledge graph and recommend exploring Paris, even if the user didn't specifically mention the city. Interoperability: Sharing Knowledge Like a Universal Library Knowledge graphs aren't isolated islands of information. They can be built using standardized formats, allowing different systems to understand and exchange information stored within their graphs, like a universal filing system for libraries. Each library can curate its own collection (specific knowledge graph), but they can all leverage the information from other libraries because they follow the same organization principles (standardized formats). Example A product recommendation engine in an online store uses a knowledge graph. This graph might connect products with their features, brands, and similar items. The store could then share this knowledge graph with a partner company that provides product reviews. The review company, with its own knowledge graph for user sentiment analysis, could then analyze reviews in the context of the product information from the store's knowledge graph. This can lead to more insightful recommendations for customers. A Sample of Important Use Cases Knowledge graphs may provide a powerful framework for systematically generating test cases. This can be done by leveraging the structured representation of software components, their interactions, and domain-specific knowledge. By analyzing the graph, testers can identify critical paths, handle complexity, incorporate constraints, and automate the generation process, improving the quality and coverage of the testing effort. Let's explore some important use cases. Modeling Software Components and Interactions Knowledge graphs can represent components of a software system, such as modules, classes, functions, or APIs, as nodes in the graph. Edges between these nodes may represent the interactions or dependencies between the components. By analyzing these interactions, testers can identify potential test scenarios and paths through the system. Incorporating Domain Knowledge Knowledge graphs can integrate domain-specific knowledge, such as industry standards, best practices, or regulatory requirements, into the test case generation process. By incorporating domain-specific nodes and edges into the graph, testers can ensure that test cases align with domain-specific considerations and constraints. Versioning and Change Management Knowledge graphs can also support versioning and change management by tracking the history of requirements and test cases over time. Testers can view the evolution of requirements and their associated test cases, including when changes were made and by whom. This historical context is valuable for understanding the rationale behind changes and ensuring traceability across different iterations of the software. Cross-Referencing Dependencies Requirements often have dependencies on each other, and test cases may also have dependencies on multiple requirements. Knowledge graphs can capture these dependencies as edges between nodes, enabling testers to visualize and understand the interconnectedness of requirements and test cases. This can help in identifying potential conflicts or gaps in the testing coverage. Identifying Patterns and Trends Knowledge graphs may enable testers to identify patterns and trends in defect occurrences, such as recurring issues, common failure scenarios, or correlations between specific code changes and defects. By analyzing the graph, testers can gain insights into the root causes of defects and prioritize their investigation efforts accordingly. OpenSource Knowledge Graphs Some open-source knowledge graphs offer a glimpse into how these systems are structured and function. Examples include: Wikidata: A collaborative, editable knowledge base operated by the Wikimedia Foundation DBpedia: A knowledge graph extracted from Wikipedia YAGO: A knowledge graph from Wikipedia for web search KBpedia: KBpedia is an open-source knowledge graph that integrates seven leading public knowledge bases, including Wikipedia, Wikidata, schema.org, DBpedia, GeoNames, OpenCyc, and standard UNSPSC products and services. It provides a comprehensive structure for promoting data interoperability and knowledge-based artificial intelligence (KBAI). KBpedia’s upper ontology (KKO) includes more than 58,000 reference concepts, mapped linkages to about 40 million entities (mostly from Wikidata), and 5,000 relations and properties. It’s a flexible and computable knowledge graph suitable for various machine learning tasks. Logseq: A knowledge graph tool that combines note-taking, outlining, and wiki functionality; It allows users to create interconnected notes and organize information in a graph-like structure. Athens: A knowledge graph tool that integrates with other note-taking apps like Roam Research; It allows users to create linked notes and build a network of ideas. GraphGPT: While not a standalone knowledge graph, GraphGPT is a language model fine-tuned for generating graph-based responses. It can be used to create educational content related to knowledge graphs. GitJournal: A knowledge graph tool that integrates with Git repositories; It allows users to create and manage notes using Git version control. RecBole: A recommendation library that leverages knowledge graphs for personalized recommendations; It can be useful for educational scenarios related to recommendation systems. DeepKE: A toolkit for knowledge embedding that can be used to embed entities and relations from knowledge graphs into vector representations; It’s helpful for educational purposes related to graph-based machine learning. These resources provide a valuable learning ground for understanding the fundamentals of knowledge graphs and their potential applications. Knowledge Graphs in the Industry There are multiple cases in the industry where companies benefit from knowledge graphs. The tech giant Google utilizes knowledge graphs extensively. Their knowledge graph powers search results by understanding the relationships between entities, providing more relevant information to users. Amazon leverages knowledge graphs to enhance its recommendation systems. By analyzing user behavior and product attributes, they create personalized recommendations for customers. Walmart uses knowledge graphs to optimize supply chain management. By modeling relationships between products, suppliers, and logistics, they improve inventory management and distribution. The ride-sharing company Lyft, employs knowledge graphs to enhance route optimization and improve driver-passenger matching. By understanding geographical relationships, they optimize travel times and reduce wait times. Airbnb’s knowledge graph helps match hosts and guests based on preferences, location, and availability. It enhances the user experience by suggesting relevant listings. Let's dive into the details of two specific cases: Allianz and eBay. Allianz: Streamlining Regression Testing with Knowledge Graphs German insurance giant Allianz implemented a knowledge graph system to streamline regression testing for their core insurance platform. Here's how it worked: Knowledge Graph Construction Allianz built a knowledge graph that captured information about the insurance platform's functionalities, user roles, data entities (policies, claims, customers), and the relationships between them. Test Case Automation The knowledge graph was leveraged to automate the generation of basic regression test cases. The rich network of information within the graph allowed the system to identify different testing scenarios and create corresponding test cases. This significantly reduced the manual effort required for regression testing. Improved Test Maintenance The knowledge graph's ability to represent changes in the system proved valuable. When updates were made to the insurance platform, the knowledge graph was easily updated to reflect these changes. This ensured that the automatically generated regression tests remained relevant and continued to cover the latest functionalities. The results for Allianz were positive. They reported a significant reduction in regression testing time and a corresponding increase in test coverage. The knowledge graph also simplified test maintenance, allowing testers to focus on more complex scenarios. eBay: Enhancing Test Case Design With Knowledge Graphs E-commerce giant eBay experimented with knowledge graphs to improve the design and management of test cases for their marketplace platform. Here's a breakdown of their approach: Mapping User Journeys eBay used a knowledge graph to model user journeys on the platform. This included entities like buyers, sellers, products, search functionalities, and checkout processes. Relationships between these entities were carefully mapped, providing a holistic view of user interactions. Identifying Test Coverage Gaps By visualizing user journeys within the knowledge graph, eBay could easily identify areas where existing test cases were lacking. For example, the graph might reveal that there were no tests for a specific type of user interaction or a particular edge case scenario. Optimizing Test Suite Design With these gaps identified, eBay could then design new test cases to ensure comprehensive coverage of user journeys. The knowledge graph facilitated a more systematic approach to test case design, ensuring functionalities were thoroughly tested. While specific details about the outcomes are limited, eBay's experiment demonstrates the potential of knowledge graphs to improve the efficiency and effectiveness of test case design for complex software systems. Technological Challenges There are open issues in building and maintaining these powerful tools. From gathering and cleaning vast amounts of data to ensuring the knowledge graph stays up-to-date, there are significant challenges to overcome. Let's explore a sample of challenges in detail. 1. Data Acquisition and Cleaning Knowledge Gathering Building a comprehensive knowledge graph requires gathering information from diverse sources. This can be a time-consuming and resource-intensive task, especially for complex domains. Data Quality The accuracy and consistency of information feeding into the knowledge graph are crucial. Cleaning and filtering data to eliminate errors, inconsistencies, and duplicates can be a significant challenge. 2. Knowledge Graph Construction and Maintenance Schema Design Defining the structure of the knowledge graph, including the types of entities, relationships, and properties, requires careful planning. This schema should be flexible enough to accommodate new information while maintaining consistency. Knowledge Graph Population Populating the graph with accurate and up-to-date information can be an ongoing process. As the world changes, the knowledge graph needs to be updated to reflect these changes. 3. Integration and Interoperability Data Integration Knowledge graphs often need to integrate information from various sources, which can have different formats and structures. Reconciling these differences and ensuring seamless data flow can be challenging. Interoperability For knowledge graphs to truly unlock their potential, they need to be able to communicate and exchange information with other knowledge graphs. Standardized formats and protocols are needed to facilitate this interoperability. 4. Reasoning and Inference Reasoning Capabilities While knowledge graphs have the potential to reason and infer new information based on existing connections, developing robust reasoning algorithms is an ongoing area of research. Explainability When a knowledge graph makes an inference, it's crucial to understand the reasoning behind it. Ensuring transparency and explainability in the reasoning process is important for building trust in the system. 5. Scalability and Performance Large Knowledge Graphs As knowledge graphs grow in size and complexity, managing their storage, processing, and querying can become challenging. Scalable solutions are needed to handle massive amounts of information efficiently. Query Performance Ensuring fast and efficient retrieval of information from the knowledge graph is essential for real-world applications. Optimizing query processing techniques is an ongoing challenge. Wrapping Up Knowledge graphs represent a paradigm shift in software engineering and testing. By moving beyond traditional test case management approaches, knowledge graphs offer a more holistic and interconnected view of the software system. This structured representation of information unlocks possibilities for automation, optimization, and more robust and efficient software development lifecycles. As the technology matures and the challenges are addressed, knowledge graphs are a promising candidate to become a cornerstone of modern software engineering practices.
NCache Java Edition with distributed cache technique is a powerful tool that helps Java applications run faster, handle more users, and be more reliable. In today's world, where people expect apps to work quickly and without any problems, knowing how to use NCache Java Edition is very important. It's a key piece of technology for both developers and businesses who want to make sure their apps can give users fast access to data and a smooth experience. This makes NCache Java Edition an important part of making great apps. This article is made especially for beginners to make the ideas and steps of adding NCache to your Java applications clear and easy to understand. It doesn't matter if you've been developing for years or if you're new to caching, this article will help you get a good start with NCache Java Edition. Let’s start with a step-by-step process to set up a development workstation for NCache with the Java setup. NCache Server Installation: Java Edition NCache has different deployment options. The classification is listed below: On-premises Cloud Using Docker/Kubernetes You can check all the deployment options and the package available for the deployment here. NCache recommends at least SO-16 (16GB RAM, 8v CPU) to get optimum performance in a production environment, for a higher transaction load we should go with SO-32, SO-64, or SO-128. NCache Server Deployment With Docker Image NCache provides different images (alachisoft/ncache - Docker Image | Docker Hub) for Windows and Linux platform Java edition. Let’s see how to deploy the NCache server using the latest Linux Docker image. Use the below Docker command to pull the latest image: docker pull alachisoft/ncache:latest-java Now we successfully pulled the Docker image. Run the Docker image using the Docker command below: For a development workstation: docker run --name ncache -itd -p 8251:8251 -p 9800:9800 -p 8300:8300 -p 8301:8301 alachisoft/ncache:latest-java Use the actual host configuration for the production NCache server: docker run –name ncache -itd –network host alachisoft/ncache:latest-java The above command will run the NCache server and listen to port 8251. Now, launch NCache Management Center using the browser (localhost:8251). You will get a modal popup to register your license key as shown below: Click on Start Free Trial to activate the free trial with the license key, using the form below. You can register your license key using this registration page form or the Docker command to register the license key as given below: docker exec -it ncache /opt/ncache/bin/tools/register-ncacheevaluation -firstname [registered first name] -lastname [registered last name] -company [registered company name] -email [registered e-mail id] -key [key] Now, open the NCache Management Center from the browser http://localhost:8251/. NCache Cache Cluster Let’s install one more image in a different instance, with proper network configuration. Use the document below for the network configuration with NCache docker image deployment: Create NCache Containers for Windows Server I deployed one image in the 10.0.0.4 instance and another in the 10.0.0.5 instance. I just hopped into 10.0.0.4 NCache Management Center and removed the default cluster cache created during the installation. Let’s create a new clustered cache using the NCache Management Center Wizard. Click on New from the Clustered Cache page as shown in the figure below: It’s a 7-step process to create a clustered cache with the NCache Management Center interface, which we will go through one by one. Step 1: In-Memory Store In this step, you can define the in-memory store type, the name of the clustered cache, and the serialization type. In my case, I named the clustered cache as demoCache and the serialization as JSON. Step 2: Caching Topology Define the caching topology in the screen; in my case, I just went with default options. Step 3: Cache Partitions and Size In this screen, we can define the cache partition size. In my case, I just went with the default value. With this option, it will skip step 4. Also, I added two server nodes: 10.0.0.4 and 10.0.0.5. Step 5: Cluster TCP Parameters Define the Cluster Port, Port Range, and Batch Interval values. In my case, I went with the default values. Step 6: Encryption and Compression Settings You can enable the encryption and compression settings in this step. I just went with default values. Step 7: Advanced Options You can enable eviction and also check other advanced options. In my case, I checked to start the cache on the finish. Finally, click on Finish. Once the process is complete, it will create and start the clustered cache with two nodes. (10.0.0.4 and 10.0.0.5). Now the cluster is formed. Start the Cache You can use the start option from the NCache Management Center to start the clustered cache, as shown in the below figure. You can also use the command below to start the server: start-cache –name demoCache Run a Stress Test Click Test-Stress and select the duration to run the stress test. This is one of my favorite features in NCache Management Center where you can initiate a stress test with ease just by a button click. You can also use the commands below to start the server. For example, to initiate a Test-Stress for the demoCache cluster with default settings: test-stress –cachename demoCache Click on Monitor to check the metrics. You can monitor the number of requests processed by each node. Click on Statistics to get the complete statistics of the clustered caches. SNMP Counter to Monitor NCache Simple Network Management Protocol (SNMP) is a key system used for keeping an eye on and managing different network devices and their activities. It's a part of the Internet Protocol Suite and helps in sharing important information about the network's health and operations between devices like routers, switches, servers, and printers. This allows network managers to change settings, track how well the network is doing, and get alerts on any issues. SNMP is widely used and important for keeping networks running smoothly and safely. It's a vital part of managing and fixing networks. NCache has made SNMP monitoring easier by now allowing the publication of counters through a single port. Before, a separate port was needed for each cache. Make sure the NCache service and cache(s) to monitor are up and running. Configure NCache Service The Alachisoft.NCache.Service.dll.config file, located in the %NCHOME%\bin\service folder, provides the ability to activate or deactivate the monitoring of cache counters via SNMP by modifying particular options. These options are marked by specific tags. Update the value for the tags below: <add key="NCacheServer.EnableSnmpMonitoring" value="true"/> <add key="NCacheServer.SnmpListenersInfoPort" value="8256"/> <add key="NCacheServer.EnableMetricsPublishing" value="true"/> Change the NCacheServer.EnableSnmpMonitoring tag to true to turn on or off the SNMP monitoring of NCache cache counters. Initially, this tag is off (false). Change the NCacheServer.SnmpListenersInfoPort tag to true to set the port for SNMP to listen on. The default port is 8256, but you can adjust it according to your needs. Change the NCacheServer.EnableMetricsPublishing tag to true if you want to start or stop sending metrics to the NCache Service. Remember to reboot the NCache Service once you've made the necessary adjustments to the service configuration files. SNMP Monitoring NCache has made available a single MIB file called alachisoft.mib that keeps track of various counters which can be checked using SNMP. This file tells you about the ports used for different types of caches and client activities. You can find this file at %NCHOME%\bin\resources. To look at these counters, you can use a program called MIB Browser Free Tool to go through the MIB file. Use port 8256 to connect with NCache, and open the SNMP Table from View to check all the attributes of the NCache as shown in the figure below: To check specific attribute details in the SNMP Table, first pick the attributes you want to see. For a sample, I have selected cacheName, cacheSize, cacheCount, fetchesPerSec, requestsPerSec, additionPerSec. Then, click on View from the menu at the top before you choose the SNMP Table. You'll then see the values of the counter in the table as shown in the figure below. Summary This article provides a beginner-friendly guide on how to get started with NCache Java Edition, covering essential steps such as installing the NCache server, deploying it using a Docker image, starting the cache, conducting a stress test to evaluate its performance, and monitoring its operation through JMX counters. It helps you get started with enhancing your Java application's speed and reliability by implementing distributed caching with NCache.
As the landscape of enterprise technology evolves, the marriage of Artificial Intelligence (AI) with SAP ABAP (Advanced Business Application Programming) is reshaping the way businesses approach software development within the SAP ecosystem. This article delves into the groundbreaking integration of AI with SAP ABAP programming, exploring how this fusion is revolutionizing SAP development processes. SAP ABAP and Its Legacy The Foundation of SAP Development SAP ABAP has long been the backbone of SAP development, providing a powerful and versatile language for customizing SAP applications. ABAP's capabilities have driven the customization of SAP systems to meet specific business requirements. Challenges in Traditional Development Traditional SAP development in ABAP involves manual coding, testing, and debugging processes. As business complexities increase, this traditional approach faces challenges in terms of speed, efficiency, and adaptability to dynamic market demands. AI Integration in SAP ABAP Programming Automating Development Processes AI integration in SAP ABAP programming introduces automation to development processes. AI algorithms can analyze patterns in existing code, identify common errors, and even propose optimized solutions, reducing manual coding efforts and minimizing the risk of bugs. Enhancing Code Quality AI-driven tools can assess code quality, adherence to best practices, and compliance with coding standards. This ensures that the developed applications not only meet functional requirements but also align with industry standards, leading to more robust and maintainable codebases. Now, I asked the AI tool to write a simple program to extract data from MARA, and no doubt it produced nice code. To test more I asked it to write a program to extract data by combining two tables — mara (material master data) and marc (material master company code) — and it produced the code below. There is no doubt that this is very good code. But when I asked the AI to write an extract data from the MSGEG table with multiple conditions query, which is a bit complex, the prompt I gave to the AI was: "ABAP program to extract data from mseg table where mblnr in 10 to 100 and mjahr in 2022 to 2024 and bwart eq 1234 and werks Eq 1234". I received the code below which is not good. Intelligent Development Assistance AI-Powered Code Assistants Integrating AI with SAP ABAP introduces intelligent code assistants. These assistants can provide real-time suggestions, auto-completion, and context-aware recommendations as developers write code, streamlining the coding process and improving overall developer productivity. Predictive Debugging AI can predict potential issues in the code during the development phase, enabling developers to proactively address issues before they escalate. This predictive debugging capability saves time, enhances code reliability, and facilitates a smoother development lifecycle. Accelerating Innovation Rapid Prototyping and Innovation The fusion of AI with SAP ABAP accelerates innovation by facilitating rapid prototyping. Developers can leverage AI tools to quickly build and test prototypes, enabling organizations to experiment with new functionalities and features before full-scale implementation. Agility in Customization AI-infused SAP ABAP programming enables organizations to respond more rapidly to changing business requirements. Developers can adapt and customize applications swiftly, ensuring that SAP systems remain agile and aligned with evolving business needs. Challenges and Considerations Skill Upgradation The integration of AI with SAP ABAP necessitates skill upgradation among development teams. Organizations need to invest in training programs to equip their developers with the knowledge and expertise required to leverage AI tools effectively. Ethical AI Practices As AI becomes integral to SAP development, organizations must adhere to ethical AI practices. Ensuring fairness, transparency, and accountability in AI-driven decisions is crucial for responsible development. Conclusion The integration of AI with SAP ABAP programming marks a transformative leap in SAP development. By infusing intelligence into the coding process, organizations can achieve unparalleled efficiency, code quality, and innovation in their SAP applications. As this synergy between AI and SAP ABAP continues to evolve but it takes some time to get the AI matured to write complex ABAP program logic, it promises to redefine the future of SAP development, empowering organizations to stay agile in the face of technological advancements.
I recently read an article about the worst kind of programmer. I agree with the basic idea, but I wanted to add my thoughts on it. I have seen, over time, that developers seem invested in learning new things for the sake of new things, rather than getting better at existing approaches. Programming is like everything else — new is not always better. I have a Honda CRV that is not as easy to use as some cars I used to own before touch interfaces became popular. The touch screen sometimes acts like I'm pressing various places on the screen when I'm not, making beeping noises and flipping screens randomly. I have to stop and turn the car off and on to stop it. It has a config screen with every option disabled. It has bizarre logic about locking and unlocking the doors, that I have never fully figured out. I often wonder if devs who make car software have a driver's license. If I tried asking 100 programmers the following question, chances are very few of them, if any, could answer it without a web search: Bob just completed programming school, and heard about MVC, but is unsure how to tell which code should be modeled, which code should be viewed, and which code should be controlled. How would you explain the MVC division of code to Bob? It's not a genius question, it's really very basic stuff. Here are some other good questions about other very basic stuff: 1. Why Did Developers Decide in REST That POST Is Created and Put Is Updated? The HTTP RFCs have always stated that PUT is created or updated to a resource on the server such that a GET on that resource returns what was PUT, and that POST is basically a grab bag of whatever does not fit into other verbs. The RFCs used to say that a POST URL is indicative of an operation, now they just say POST is whatever you say it is. Developers often talk about the REST usage of POST and PUT like Jesus Christ himself dictated this usage, like there is no argument about it. I have never seen any legitimate reason why PUT cannot be created or updated as the RFC says, and POST can be for non-CRUD stuff. Any real, complex system that is driven by customer demand for features is highly likely to have some operations that are not CRUD — integrations with other systems, calculations, searches (eg, a filter box that shows matches as you type, find results for a search based on input fields), and so on. By reserving POST for these kinds of other operations, you can immediately identify anything that isn't CRUD. Otherwise, you wind up with two usages of POST — mostly for create, but here and there for other stuff. 2. Why Do Java Developers Insist on Spring and JPA for Absolutely Every Java Project Without Question? Arguably, a microservice project should be, well, you know, micro. Micro is defined as an adjective that means extremely small. When Spring and JPA take up over 200MB of memory and take 10 seconds to fire up a near-empty project that barely writes one row to a table, I'm not seeing the micro here. Call me crazy, but maybe micro should be applied to the whole approach, not just the line count: the amount of memory, the amount of handwritten code, the amount of time a new hire takes to understand how the code works, etc. You don't have to be a freak about it, trying 10 languages to see which uses the least amount of RAM, just be reasonable about it. In this case, Spring and JPA were designed for monolithic development, where you might have problems like the following: A constructor is referred to 100 times in the code. Adding a new field requires modifying all 100 constructor calls to provide the new field, but only one of those calls actually uses the new field. So dependency injection is useful. There are thousands of tables, with tens of thousands of queries, that need to be supported in multiple databases (eg, Oracle and MSSQL), with use cases like multi-tenancy and/or sharding. There comes a point where it is just too much to do some other way, and JPA is very helpful. 3. Why Does Every Web App Require Heavy Amounts of JS Code? When I started in this business, we used JSP (Java Server Pages), which is a type of SSR (Server Side Rendering). Basically, an HTML templating system that can fill in the slots with values that usually come from a database. It means when users click on a button, the whole page reloads, which these days is fast enough for it to be a brief sort of blink. The bank I have used since about 2009 still uses some sort of SSR. As a customer, I don't care it's a bit blinky. It responds in about a second after each click, and I'm only going to do maybe 12-page loads in a session before logging out. I can't find any complaint on the web about it. I saw a project "upgrade" from JSP to Angular. They had a lot of uncommented JSP code that nobody really knew how it worked, which became Angular code nobody really knew how it worked. Some people would add new business logic to Angular, some would add it to Java code, and nobody leading the project thought it was a good idea to make a decision about this. Nobody ever explained why this upgrade was of any benefit, or what it would do. The new features being added afterward were no more or less complex than what was there before, so continuing to use JSP would not have posed any problems. It appeared to be an upgrade for the sake of an upgrade. 4. Why Is Everything New Automatically So Much Better Than Older Approaches? What is wrong with the tools used 10 or 15 years ago? After all, everything else works this way. Sure, we have cars with touch screens now, but they still use gas, tires, cloth or leather seats, a glove box, a steering wheel, glass, etc. The parts you touch daily to drive are basically the same as decades ago, with a few exceptions like the touch screen and electric engines. Why can't we just use a simple way of mapping SQL tables to objects, like a code generator? Why can't we still use HTML templating systems for a line of business apps that are mostly CRUD? Why can't we use approaches that are only as complex as required for the system at hand? I haven't seen any real improvements in newer languages or tooling that are significantly better in real-world usage, with a few exceptions like using containers. 5. Do You Think Other Industries Work This Way? I can tell you right now if engineers built stuff like programmers do, I would never get in a car, walk under a bridge, or board an airplane. If doctors worked that way, I'd be mortally afraid every visit. So why do we do things this way? Is this really the best we can do? I worked with a guy who asked shortly after being hired "Why the f do we have a mono repo?". When I asked what was wrong with a monorepo, he was unable to give any answer, but convinced management how this has to change pronto, apparently convinced with almightly passion all microservice projects must be structured as separate repos per service. Not sure if it was him or someone else, but somehow it was determined that each project must be deployed in its own container. These decisions were detrimental to the project in the following ways: One project was a definition of all objects to be sent over the wire. If service A object is updated to require a new field, there is no compile error anywhere to show the need to update constructor calls. If service B calls A to create objects, and nobody thinks of this, then probably only service A is updated to provide the new required field, and a subtle hard-to-find bug exists, that might take a while for anyone to even notice. Your average corporate dev box can handle maybe 15 containers before flopping over and gasping for air. So we quickly lost local development in one of those unrecoverable ways where the team would never get it back. Every new dev would have to check out dozens of repos. No dependency information between repos was tracked anywhere, making it unknowable which subset of services has to be run to stand up service X to work on that one service. Combined with the inability to run all repos locally yields two equally sucktastic solutions to working on service X: Use trial and error to figure out which subset stands up X and run it locally Deploy every code change to a dev server When Alex talks about programmers using hugely complex solutions of the sort he describes, it sounds to me like devs who basically jerk off to everything new and cool. This is very common in this business, every team has people like that in it. That isn't necessarily a big problem by itself, but when combined with the inability/unwillingness to ensure other devs are fully capable of maintaining the system, and possibly the arrogance of "everything I say is best", and/or "only I can maintain this system," that's the killer combination that does far more harm than good.
Being a Backend Developer Today Feels Harder Than 20 Years Ago
March 26, 2024 by
How Scrum Teams Fail Stakeholders
March 26, 2024 by CORE
March 28, 2024 by CORE
Advanced-Data Processing With AWS Glue
March 27, 2024 by CORE
Explainable AI: Making the Black Box Transparent
May 16, 2023 by CORE
Advanced-Data Processing With AWS Glue
March 27, 2024 by CORE
Low Code vs. Traditional Development: A Comprehensive Comparison
May 16, 2023 by
March 28, 2024 by CORE
Advanced-Data Processing With AWS Glue
March 27, 2024 by CORE
Advanced-Data Processing With AWS Glue
March 27, 2024 by CORE
Low Code vs. Traditional Development: A Comprehensive Comparison
May 16, 2023 by
Data Streaming for AI in the Financial Services Industry (Part 2)
March 27, 2024 by CORE
Know How To Get Started With LLMs in 2024
March 27, 2024 by
Five IntelliJ Idea Plugins That Will Change the Way You Code
May 15, 2023 by