DZone Spotlight

In the world of infrastructure management, two fundamental approaches govern how resources are deployed and maintained: mutable and immutable infrastructure. These approaches influence how updates are made, how infrastructure evolves, and how consistency is ensured across different environments. Mutable infrastructure refers to systems that can be changed or updated after they’ve been initially deployed. This means that configuration changes, software updates, or patches can be applied directly to existing infrastructure resources without replacing them entirely. For instance, a server can be updated by installing new software, tweaking its settings, or increasing its resources. While the server itself stays the same, its configuration evolves over time. Immutable infrastructure works differently. Once it’s deployed, it can’t be changed or updated. Instead of modifying what’s already there, any updates or changes require replacing the existing infrastructure with a new version that includes the updates. For example, if a new version of an application needs to be deployed, new servers are created with the updated settings, while the old servers are shut down or removed. This approach ensures consistency with each deployment and avoids any unexpected issues from lingering changes. Key Differences Between Mutable and Immutable Infrastructure When comparing mutable and immutable infrastructure, several key differences highlight the strengths and trade-offs of each approach. These differences revolve around how changes are handled, how infrastructure consistency is maintained, and the overall impact on operations. Use Case Mutable Infrastructure Immutable Infrastructure Change Management Allows in-place updates, where changes, updates, or patches can be applied directly to running infrastructure without redeploying. This can be faster and more convenient for making incremental adjustments. Does not allow in-place changes. Instead, any update requires the creation of a new instance or infrastructure. The old instance is terminated after the new one is successfully deployed. Changes happen by replacing the infrastructure entirely. Configuration Drift Prone to configuration drift. Over time, as small changes are applied to systems manually or through different tools, the configuration of the system can deviate from its original state. This makes it harder to maintain consistency. Eliminates configuration drift. Every deployment starts fresh with a new environment, ensuring that the infrastructure always matches the desired state and behaves consistently. Consistency May lead to inconsistencies if manual changes are made, or if different versions of updates are applied across environments. This can result in unexpected behavior, particularly in environments that are long-lived or frequently updated. Highly consistent because each deployment uses a clean state. This ensures that each instance of the infrastructure are same, avoiding unexpected issues from differing configurations. Downtime Can be updated in-place, which often leads to shorter or no downtime. This is crucial for systems that require high availability and cannot afford to be redeployed. May involve temporary downtime during the replacement process, depending on the strategy used (e.g., blue-green or rolling deployments). However, modern techniques like blue-green or canary deployments can mitigate this issue by seamlessly transitioning between old and new infrastructure. Rollback Complexity Rolling back changes can be complex because reverting to a previous configuration may not fully restore the infrastructure to its original state. In most cases, manual intervention may be required. Rollback is simpler. Since new infrastructure is created for every change, rolling back is as easy as redeploying the previous, working version. This reduces the risk of failure when reverting to an earlier state. Security Security patches are applied directly to running systems, which can leave them vulnerable to attack if patches are delayed or improperly applied. Manual patching introduces risks. More secure because each update replaces the entire system with a fresh instance. This ensures that any vulnerabilities from previous configurations are completely removed, and patched versions are consistently applied. Operational Overhead Requires more maintenance and monitoring to ensure that updates are properly applied and systems remain secure over time. This can increase operational overhead, especially when managing large, complex environments. Reduces operational overhead by standardizing updates and deployments. Since there’s no need to manage in-place updates, less effort is required to maintain consistency across environments. Resource Efficiency More resource-efficient in some cases since updates are made to the existing resources without creating new ones. This can save costs and time, especially for updates that don’t require full system redeployment. Requires the creation of new resources for every update, which can lead to increased costs and resource usage, particularly for environments that need frequent updates or scaling. However, automation tools can help manage this process more efficiently. Use Cases Suited for environments where rapid, in-place updates are needed without full redeployment (e.g., databases, development and testing environments, or legacy systems). It’s also practical for environments where costs need to be minimized by reusing existing infrastructure. Best for environments where consistency, security, and reliability are critical, such as production environments, containerized applications, and microservices. Immutable infrastructure is ideal for organizations that prioritize automation, scaling, and continuous delivery. Infrastructure Lifespan Often associated with long-lived infrastructure where the same resources are maintained and updated over time. These systems may evolve as they stay active, leading to drift over time. Typically associated with short-lived infrastructure. Resources are frequently replaced with newer versions, reducing the need for ongoing maintenance of the same system. When to Choose Mutable Infrastructure While immutable infrastructure is often preferred in modern environments due to its consistency and reliability, there are still several scenarios where choosing mutable infrastructure is more practical and beneficial. Mutable infrastructure offers flexibility for certain use cases where in-place changes, cost-effectiveness, or maintaining long-lived systems are essential. Here are some key situations where you should consider using mutable infrastructure: Dynamic or evolving environments - In scenarios where the infrastructure needs to frequently adapt to changes, such as in development or testing environments, mutable infrastructure is advantageous. Developers may need to quickly update configurations, modify code, or patch resources without spinning up new instances. Cost-sensitive environments - Immutable infrastructure can increase operational costs, particularly in cloud environments where spinning up new instances for every update incurs additional expenses. For cost-sensitive environments, such as startups or organizations with tight budgets, mutable infrastructure can be more economical. Stateful applications - Applications or services that maintain state (such as databases, file systems, or session-oriented services) benefit from mutable infrastructure. In these cases, tearing down and replacing infrastructure can lead to data loss or significant complexity in preserving state. Legacy systems - Legacy systems often rely on mutable infrastructure due to their architecture and the fact that they weren’t designed with modern immutable practices in mind. Rewriting or migrating legacy applications to immutable infrastructure may be impractical, expensive, or risky, making mutable infrastructure the better choice. Applications with infrequent updates - For environments or applications where updates are infrequent and the risk of configuration drift is low, mutable infrastructure can be a simple and effective solution. If the system doesn’t require constant adjustments or scaling, maintaining long-lived infrastructure may be sufficient. Systems that require minimal downtime - Some critical systems cannot afford the downtime that might occur during the replacement of infrastructure, especially in high-availability environments. Mutable infrastructure allows for in-place updates, which can minimize or eliminate downtime altogether. Systems with complex interdependencies - In environments where services or applications have complex interdependencies, managing immutable infrastructure might become difficult. When numerous components rely on each other, applying changes in place ensures those connections remain intact without needing to redeploy the entire infrastructure. When to Choose Immutable Infrastructure Immutable infrastructure has become a popular approach in today's IT world, especially in cloud-native and DevOps environments. It helps prevent configuration drift and guarantees consistent, reliable deployments. However, it's not always the best fit for every situation. There are cases where choosing immutable infrastructure is highly advantageous, particularly when consistency, security, and scalability are crucial priorities. Here are key situations where choosing immutable infrastructure is the better choice: Production environments - Immutable infrastructure works exceptionally well in production environments where stability and reliability are key. By avoiding in-place changes, it keeps the production environment consistent and minimizes the risk of unexpected errors from manual updates or configuration drift. This makes it a solid choice when maintaining a reliable system is essential. Security-conscious environments - In environments where security is a top priority, immutable infrastructure is a more secure option. Since no manual changes are applied after the infrastructure is deployed, there’s less risk of introducing vulnerabilities through untracked or insecure changes. Microservices architectures - Microservices architectures are built to be modular, making it easy to replace individual components. In these setups, immutable infrastructure plays a key role by ensuring each service is deployed consistently and independently, helping to avoid the risk of misconfigurations. This approach enhances reliability across the system. CI/CD pipelines and automation - Continuous Integration and Continuous Delivery (CI/CD) pipelines thrive on consistency and automation. Immutable infrastructure complements CI/CD pipelines by ensuring that every deployment is identical and repeatable, reducing the chances of failed builds or broken environments. Disaster recovery and rollbacks - Immutable infrastructure simplifies disaster recovery and rollbacks. Since infrastructure is not modified after deployment, rolling back to a previous version is as simple as redeploying the last known working configuration. This reduces downtime and makes recovery faster and more reliable. Scalability and auto-scaling - In environments where scalability is a key requirement, immutable infrastructure supports automatic scaling by creating new instances as needed, rather than modifying existing ones. This is particularly useful for cloud-native applications or containerized environments where dynamic scaling is a common requirement. Blue-green or canary deployments - Blue-green and canary deployment strategies are ideal candidates for immutable infrastructure. These deployment methods rely on running two environments simultaneously (blue and green), where the new version is tested before it fully replaces the old one. Cloud-native and containerized applications - Cloud-native and containerized applications naturally align with immutable infrastructure because they are designed to be stateless, scalable, and disposable. Infrastructure as Code (IaC) tools like Terraform, combined with container orchestration platforms like Kubernetes, benefit greatly from immutable practices. High-availability systems - High-availability systems that cannot tolerate downtime can benefit from immutable infrastructure. With rolling or blue-green deployments, updates are applied seamlessly, ensuring that there is no disruption to the end user. Hybrid Approaches: Combining Mutable and Immutable Infrastructure While immutable infrastructure offers reliability and consistency, and mutable infrastructure provides flexibility and state retention, many organizations can benefit from a hybrid approach. By combining the best of both models, you can build a more flexible infrastructure that adapts to the specific needs of different components in your environment. This balanced approach allows you to address varying requirements more effectively. A hybrid approach typically involves using immutable infrastructure for stateless services, where consistency and repeatability are essential, and mutable infrastructure for stateful services or legacy systems, where preserving data and flexibility is more important. Terraform, as a powerful Infrastructure as Code (IaC) tool, can manage both models simultaneously, giving you the flexibility to implement a hybrid approach effectively. Use Immutable Infrastructure for Stateless Components Stateless services and applications that don’t rely on maintaining internal state across sessions are ideal for immutable infrastructure. These services can be replaced or scaled without the need for in-place updates. Example: With Terraform, you can manage web server deployments behind a load balancer seamlessly. Each time the application is updated, new instances are deployed while the old ones are removed. This ensures that the web servers are always running the latest version of the application, without any risk of configuration drift. Use Mutable Infrastructure for Stateful Components Stateful components, such as databases, file systems, or applications that retain session data, require mutable infrastructure to preserve the data across updates. Replacing these components in an immutable model would involve complex data migration and could risk data loss. Example: Manage a relational database like PostgreSQL using Terraform, where you need to update storage capacity or apply security patches without replacing the database instance. This ensures that the data remains intact while the infrastructure is modified. Automate Infrastructure Management With Terraform Terraform’s ability to define infrastructure as code allows you to automate both mutable and immutable deployments. With a hybrid approach, you can use Terraform to manage both types of infrastructure side by side, ensuring consistency where needed and flexibility where required. Example: Use Terraform to define both immutable and mutable resources in a single deployment plan. For example, use immutable infrastructure for auto-scaling application servers, while managing stateful databases with mutable infrastructure for in-place updates. Implement Hybrid Scaling Strategies In environments with both stateless and stateful services, scaling strategies can benefit from a hybrid approach. Stateless services can be scaled horizontally using immutable infrastructure, while stateful services may require vertical scaling or more complex approaches. Example: Use Terraform to manage auto-scaling groups for stateless web servers while adjusting database resources (such as memory or CPU) for stateful services through mutable infrastructure. This ensures that both types of services can scale based on their unique needs. Legacy System Modernization Many organizations have legacy systems that are critical to their operations but are not easily moved to an immutable infrastructure model. In these cases, a hybrid approach allows organizations to maintain these legacy systems with mutable infrastructure while using immutable infrastructure for newer, cloud-native components. Example: Use Terraform to manage infrastructure for both legacy and modern applications. The legacy system (e.g., an on-premises ERP) can be maintained with mutable infrastructure, while cloud-native microservices are deployed with immutable infrastructure in the cloud. Simplifying Disaster Recovery With Hybrid Approaches A hybrid approach can simplify disaster recovery by leveraging the benefits of both models. Immutable infrastructure can be used for services that need quick rollbacks, while mutable infrastructure can handle systems that need to retain their state after recovery. Example: In a hybrid cloud environment managed by Terraform, stateless services (like a front-end application) can be redeployed quickly with immutable infrastructure, while stateful services (like a database) can be restored from backups using mutable infrastructure. Security and Compliance Considerations Security-sensitive applications benefit from immutable infrastructure due to the reduction in configuration drift and manual changes. However, some services, especially those involving sensitive data (like customer databases), may require mutable infrastructure for security patching and retention of critical state information. Also it is important to note that the evidence is updated when immutable infrastructure are used. Example: Use immutable infrastructure for API gateways and front-end services to ensure they are always deployed with the latest security patches. Simultaneously, manage the back-end databases using mutable infrastructure, allowing for in-place security patches without affecting stored data. Example: Terraform can be used to manage testing environments with immutable infrastructure, ensuring that each test is performed on a fresh, consistent environment. Meanwhile, long-lived production databases are managed with mutable infrastructure, allowing for updates and scaling without disruption. Conclusion A hybrid approach to infrastructure combines the best of both mutable and immutable models, offering flexibility for stateful and legacy systems while ensuring consistency and scalability for stateless services. By using Terraform to manage this blend, organizations can optimize their infrastructure for dynamic needs, balancing reliability, cost-efficiency, and operational flexibility. This approach allows for tailored strategies that meet the unique demands of various applications and services.

By Josephine Eskaline Joyce CORE

One of the finest ways to make the code easy to read, maintain, and improve can be done by writing clean code. Clean code helps to reduce errors, improves the code quality of the project, and makes other developers and future teams understand and work with the code. The well-organized code reduces mistakes and maintenance a lot easier. 1. Use Meaningful Names Clean code should have meaningful names that describe the purpose of variables, functions, and classes. The name itself should convey what the code does. So, anyone who is reading it can understand its purpose without referring to any additional documentation. Example Bad naming: Java int d; // What does 'd' represent? Good naming: Java int daysSinceLastUpdate; // Clear, self-explanatory name Bad method name: Java public void process() { // What exactly is this method processing? } Good method name: Java public void processPayment() { // Now it's clear that this method processes a payment. } 2. Write Small Functions Functions should be small and focused on doing just one thing. A good rule of thumb is the Single Responsibility Principle (SRP) which is each function should have only one reason to change. Example Bad example (a large function doing multiple things): Java public void processOrder(Order order) { validateOrder(order); calculateShipping(order); processPayment(order); sendConfirmationEmail(order); } Good example (functions doing only one thing): Java public void processOrder(Order order) { validateOrder(order); processPayment(order); confirmOrder(order); } private void validateOrder(Order order) { // Validation logic } private void processPayment(Order order) { // Payment processing logic } private void confirmOrder(Order order) { // Send confirmation email } Each function now handles only one responsibility, making it easier to read, maintain, and test. 3. Avoid Duplication Duplication is one of the biggest problems in messy code. Avoid copying and pasting code. Instead, look for common patterns and extract them into reusable methods or classes. Example Bad example (duplicated code): Java double getAreaOfRectangle(double width, double height) { return width * height; } double getAreaOfTriangle(double base, double height) { return (base * height) / 2; } Good example (eliminated duplication): Java double getArea(Shape shape) { return shape.area(); } abstract class Shape { abstract double area(); } class Rectangle extends Shape { double width, height; @Override double area() { return width * height; } } class Triangle extends Shape { double base, height; @Override double area() { return (base * height) / 2; } } Now, the logic for calculating the area is encapsulated in each shape class, eliminating the need for duplicating similar logic. Java public void updateOrderStatus(Order order) { if (order.isPaid()) { order.setStatus("shipped"); sendEmailToCustomer(order); // Side effect: sends an email when the status is changed } } 4. Eliminate Side Effects Functions should avoid change of state outside their scope which causes unintended side effects. When functions have side effects, they become harder to understand and can act in unpredictable behavior. Example Bad example (function with side effects): Java public void updateOrderStatus(Order order) { if (order.isPaid()) { order.setStatus("shipped"); sendEmailToCustomer(order); // Side effect: sends an email when the status is changed } } Good example (no side effects): Java public void updateOrderStatus(Order order) { if (order.isPaid()) { order.setStatus("shipped"); } } public void sendOrderShippedEmail(Order order) { if (order.getStatus().equals("shipped")) { sendEmailToCustomer(order); } } In the above case, the function's main job is to update the status. Sending an email is another task that is handled in a separate method. 5. Keep Code Expressive Keeping the code expressive includes structure, method names, and overall design which should be easy to understand to the reader what the code is doing. Comments are rarely used if the code is clear. Example Bad example (unclear code): Java if (employee.type == 1) { // Manager pay = employee.salary * 1.2; } else if (employee.type == 2) { // Developer pay = employee.salary * 1.1; } else if (employee.type == 3) { // Intern pay = employee.salary * 0.8; } Good example (expressive code): Java if (employee.isManager()) { pay = employee.calculateManagerPay(); } else if (employee.isDeveloper()) { pay = employee.calculateDeveloperPay(); } else if (employee.isIntern()) { pay = employee.calculateInternPay(); } In the above good example, method names are made clear to express the intent. It makes user to read easier and understand the code without any comments. Conclusion Here are some of the key principles for writing clean code: use of meaningful names, use small functions, avoid repeating code, remove side effects, and write clear and expressive code. Following these practices makes the code easier to understand and fix.

Organizations are fully adopting Artificial Intelligence (AI) and proving that AI is valuable. Enterprises are looking for valuable AI use cases that abound in their industry and functional areas to reap more benefits. Organizations are responding to opportunities and threats, gain improvements in sales, and lower costs. Organizations are recognizing the special requirements of AI workloads and enabling them with purpose-built infrastructure that supports the consolidated demands of multiple teams across the organization. Organizations adopting a shift-left paradigm by planning for good governance early in the AI process will minimize AI efforts for data movement to accelerate model development. In an era of rapidly evolving AI, data scientists should be flexible in choosing platforms that provide flexibility, collaboration, and governance to maximize adoption and productivity. Let's dive into the workflow automation and pipeline orchestration world. Recently, two prominent terms have appeared in the artificial intelligence and machine learning world: MLOps and LLMOps. What Is MLOps? MLOps (Machine Learning Operations) is a set of practices and technology to standardize and streamline the process of construction and deployment of machine learning systems. It covers the entire lifecycle of a machine learning application from data collection to model management. MLOps provides a provision for huge workloads to accelerate time-to-value. MLOps principles are architected based on the DevOps principles to manage applications built-in ML (Machine Learning). The ML model is created by applying an algorithm to a mass of training data, which will affect the behavior of the model in different environments. Machine learning is not just code, its workflows include the three key assets Code, Model, and Data. Figure 1: ML solution is comprised of Data, Code, and Model These assets in the development environment will have the least restrictive access controls and less quality guarantee, while those in production will be the highest quality and tightly controlled. The data is coming from the real world in production where you cannot control its change, and this raises several challenges that need to be resolved. For example: Slow, shattered, and inconsistent deployment Lack of reproducibility Performance reduction (training-serving skew) To resolve these types of issues, there are combined practices from DevOps, data engineering, and practices unique to machine learning. Figure 2: MLOps is the intersection of Machine Learning, DevOps, and Data Engineering - LLMOps rooted in MLOps Hence, MLOps is a set of practices that combines machine learning, DevOps, and data engineering, which aims to deploy and maintain ML systems in production reliably and efficiently. What Is LLMOps? The recent rise of Generative AI with its most common form of large language models (LLMs) prompted us to consider how MLOps processes should be adapted to this new class of AI-powered applications. LLMOps (Large Language Models Operations) is a specialized subset of MLOps (Machine Learning Operations) tailored for the efficient development and deployment of large language models. LLMOps ensures that model quality remains high and that data quality is maintained throughout data science projects by providing infrastructure and tools. Use a consolidated MLOps and LLMOps platform to enable close interaction between data science and IT DevOps to increase productivity and deploy a greater number of models into production faster. MLOps and LLMOps will both bring Agility to AI Innovation to the project. LLMOps tools include MLOps tools and platforms, LLMs that offer LLMOps capabilities, and other tools that can help with fine-tuning, testing, and monitoring. Explore more on LLMOps tools. Differentiate Tasks Between MLOps and LLMOps MLOps and LLMOps have two different processes and techniques in their primary tasks. Table 1 shows a few key tasks and a comparison between the two methodologies: Task MLOps LLMOps Primary focus Developing and deploying machine-learning models Specifically focused on LLMs Model adaptation If employed, it typically focuses on transfer learning and retraining. Centers on fine-tuning pre-trained models like GPT with efficient methods and enhancing model performance through prompt engineering and retrieval augmented generation (RAG) Model evaluation Evaluation relies on well-defined performance metrics. Evaluating text quality and response accuracy often requires human feedback due to the complexity of language understanding (e.g., using techniques like RLHF) Model management Teams typically manage their models, including versioning and metadata. Models are often externally hosted and accessed via APIs. Deployment Deploy models through pipelines, typically involving feature stores and containerization. Models are part of chains and agents, supported by specialized tools like vector databases. Monitoring Monitor model performance for data drift and model degradation, often using automated monitoring tools. Expands traditional monitoring to include prompt-response efficacy, context relevance, hallucination detection, and security against prompt injection threats Table 1: Key tasks of MLOPs and LLMOps methodologies Adapting any implications into MLOps required minimal changes to existing tools and processes. Moreover, many aspects do not change: The separation of development, staging, and production remains the same. The version control tool and the model registry in the catalog remain the primary channels for promoting pipelines and models toward production. The data architecture for managing data remains valid and essential for efficiency. Existing CI/CD infrastructure should not require changes. The modular structure of MLOps remains the same, with pipelines for model training, model inference, etc., A summary of key properties of LLMs and the implications for MLOps are listed in Table 2. KEY PROPERTIES OF LLMS IMPLICATIONS FOR MLOPS LLMs are available in many forms: Proprietary models behind paid APIs Pre-training models fine-tuned models Projects often develop incrementally, starting from existing, third-party, or open-source models and ending with custom fine-tuned models. This has an impact on the development process. Prompt Engineering: Many LLMs take queries and instructions as input in the form of natural language. Those queries can contain carefully engineered “prompts” to elicit the desired responses. Designing text templates for querying LLMs is often an important part of developing new LLM pipelines. Many LLM pipelines will use existing LLMs or LLM serving endpoints; the ML logic developed for those pipelines may focus on prompt templates, agents, or “chains” instead of the model itself. The ML artifacts packaged and promoted to production may frequently be these pipelines, rather than models. Context-based prompt engineering: Many LLMs can be given prompts with examples and context, or additional information to help answer the query. When augmenting LLM queries with context, it is valuable to use previously uncommon tooling such as vector databases to search for relevant context. Model Size: LLMs are very large deep-learning models, often ranging from gigabytes to hundreds of gigabytes. Many LLMs may require GPUs for real-time model serving. Since larger models require more computation and are thus more expensive to serve, techniques for reducing model size and computation may be required. Model evaluation: LLMs are hard to evaluate via traditional ML metrics since there is often no single “right” answer. Since human feedback is essential for evaluating and testing LLMs, it must be incorporated more directly into the MLOps process, both for testing and monitoring and for future fine-tuning. Table 2: Key properties of LLMs and implications for MLOps Semantics of Development, Staging, and Production An ML solution comprises data, code, and models. These assets are developed, tested, and moved to production through deployments. For each of these stages, we also need to operate within an execution environment. Each of the data, code, models, and execution environments is ideally divided into development, staging, and production. Data: Some organizations label data as either development, staging, or production, depending on which environment it originated in. Code: Machine learning project code is often stored in a version control repository, with most organizations using branches corresponding to the lifecycle phases of development, staging, or production. Model: The model and code lifecycle phases often operate asynchronously and model lifecycles do not correspond one-to-one with code lifecycles. Hence it makes sense for model management to have its model registry to manage model artifacts directly. The loose coupling of model artifacts and code provides flexibility to update production models without code changes, streamlining the deployment process in many cases. Semantics: Semantics indicates that when it comes to MLOps, there should always be an operational separation between development, staging, and production environments. More importantly, observe that data, code, and model, which we call Assets, in development will have the least restrictive access controls and quality guarantee, while those in production will be the highest quality and tightly controlled. Deployment Patterns Two major patterns can be used to manage model deployment. The training code (Figure 3, deploy pattern code) which can produce the model is promoted toward the production environment after the code is developed in the dev and tested in staging environments using a subset of data. Figure 3: Deploy pattern code The packaged model (Figure 4, deploy pattern model) is promoted through different environments, and finally to production. Model training is executed in the dev environment. The produced model artifact is then moved to the staging environment for model validation checks, before deployment of the model to the production environment. This approach requires two separate paths, one for deploying ancillary code such as inference and monitoring code and the other “deploy code” path where the code for these components is tested in staging and then deployed to production. This pattern is typically used when deploying a one-off model, or when model training is expensive and read-access to production data from the development environment is possible. Figure 4: Deploy pattern model The choice of process will also depend on the business use case, maturity of the machine learning infrastructure, compliance and security guidelines, resources available, and what is most likely to succeed for that particular use case. Therefore, it is a good idea to use standardized project templates and strict workflows. Your decisions around packaging ML logic as version-controlled code vs. registered models will help inform your decision about choosing between the deploy models, deploy code, and hybrid architectures. With LLMs, it is common to package machine-learning logic in new forms. These may include: MLflow can be used to package LLMs and LLM pipelines for deployment. Built-in model flavors include: PyTorch and TensorFlow Hugging Face Transformers (relatedly, see Hugging Face Transformers’ MLflowCallback) LangChain OpenAI API MLflow can package the LLM pipelines via the MLflow Pyfunc capability, which can store arbitrary Python code. Figure 5 is a machine learning operations architecture and process that uses Azure Databricks. Figure 5: MLOps Architecture (Image source, Azure Databricks) Key Components of LLM-Powered Applications The field of LLMOps is quickly evolving. Here are key components and considerations to bear in mind. Some, but not necessarily all of the following approaches make up a single LLM-based application. Any of these approaches can be taken to leverage your data with LLMs. Prompt engineering is the practice of adjusting the text prompts given to an LLM to extract more accurate or relevant responses from the model. It is very important to craft effective and specialized prompt templates to guide LLM behavior and mitigate risks such as model hallucination and data leakage. This approach is fast, cost-effective, with no training required, and less control than fine-tuning. Retrieval Augmented Generation (RAG), combining an LLM with external knowledge retrieval, requires an external knowledge base or database (e.g., vector database) with moderate training time (e.g., computing embeddings). The primary use case of this approach is dynamically updated context and enhanced accuracy but it significantly increases prompt length and inference computation. RAG LLMs use two systems to obtain external data: Vector databases: Vector databases help find relevant documents using similarity searches. They can either work independently or be part of the LLM application. Feature stores: These are systems or platforms to manage and store structured data features used in machine learning and AI applications. They provide organized and accessible data for training and inference processes in machine learning models like LLMs. Fine-tuning LLMs: Fine-tuning is the process of adapting a pre-trained LLM on a comparatively smaller dataset that is specific to an individual domain or task. During the fine-tuning process, only a small number of weights are updated, allowing it to learn new behaviors and specialize in certain tasks. The advantage of this approach is granular control, and high specialization but it requires labeled data and comes with a computational cost. The term “fine-tuning” can refer to several concepts, with the two most common forms being: Supervised instruction fine-tuning: This approach involves continuing training of a pre-trained LLM on a dataset of input-output training examples - typically conducted with thousands of training examples. Instruction fine-tuning is effective for question-answering applications, enabling the model to learn new specialized tasks such as information retrieval or text generation. The same approach is often used to tune a model for a single specific task (e.g. summarizing medical research articles), where the desired task is represented as an instruction in the training examples. Continued pre-training: This fine-tuning method does not rely on input and output examples but instead uses domain-specific unstructured text to continue the same pre-training process (e.g. next token prediction, masked language modeling). This approach is effective when the model needs to learn new vocabulary or a language it has not encountered before. Pre-training a model from scratch refers to the process of training a language model on a large corpus of data (e.g. text, code) without using any prior knowledge or weights from an existing model. This is in contrast to fine-tuning, where an already pre-trained model is further adapted to a specific task or dataset. The output of full pre-training is a base model that can be directly used or further fine-tuned for downstream tasks. The advantage of this approach is maximum control, tailored for specific needs, but it is extremely resource-intensive, and it requires longer training from days to weeks. A good rule of thumb is to start with the simplest approach possible, such as prompt engineering with a third-party LLM API, to establish a baseline. Once this baseline is in place, you can incrementally integrate more sophisticated strategies like RAG or fine-tuning to refine and optimize performance. The use of standard MLOps tools such as MLflow is equally crucial in LLM applications to track performance over different approach iterations. Quick, on-the-fly model guidance. Model Evaluation Challenges Evaluating LLMs is a challenging and evolving domain, primarily because LLMs often demonstrate uneven capabilities across different tasks. LLMs can be sensitive to prompt variations, demonstrating high proficiency in one task but faltering with slight deviations in prompts. Since most LLMs output natural language, it is very difficult to evaluate the outputs via traditional Natural Language Processing metrics. For domain-specific fine-tuned LLMs, popular generic benchmarks may not capture their nuanced capabilities. Such models are tailored for specialized tasks, making traditional metrics less relevant. It is often the case that LLM performance is being evaluated in domains where text is scarce or there is a reliance on subject matter expert knowledge. In such scenarios, evaluating LLM output can be costly and time-consuming. Some prominent benchmarks used to evaluate LLM performance include: BIG-bench (Beyond the Imitation Game Benchmark): A dynamic benchmarking framework, currently hosting over 200 tasks, with a focus on adapting to future LLM capabilities Elluether AI LM Evaluation Harness: A holistic framework that assesses models on over 200 tasks, merging evaluations like BIG-bench and MMLU, promoting reproducibility and comparability Mosaic Model Gauntlet: An aggregated evaluation approach, categorizing model competency into six broad domains (shown below) rather than distilling it into a single monolithic metric LLMOps Reference Architecture A well-defined LLMOps architecture is essential for managing machine learning workflows and operationalizing models in production environments. Here is an illustration of the production architecture with key adjustments to the reference architecture from traditional MLOps, and below is the reference production architecture for LLM-based applications: RAG workflow using a third-party API: Figure 6: RAG workflow using a third-party API (Image Source: Databricks) RAG workflow using a self-hosted fine-tuned model and an existing base model from the model hub that is then fine-tuned in production: Figure 7: RAG workflow using a self-hosted fine-tuned model (Image Source: Databricks) LLMOps: Pros and Cons Pros Minimal changes to base model: Most of the LLM applications often make use of existing, pre-trained models, and an internal or external model hub becomes a valuable part of the infrastructure. It is easy and requires simple changes to adopt it. Easy to model and deploy: The complexities of model construction, testing, and fine-tuning are overcome in LLMOps, enabling quicker development cycles. Also, deploying, monitoring, and enhancing models is made hassle-free. You can leverage expansive language models directly as the engine for your AI applications. Advanced language models: By utilizing advanced models like the pre-trained Hugging Face model (e.g., meta-llama/Llama-2-7b, google/gemma-7b) or one from OpenAI (e.g., GPT-3.5-turbo or GPT-4). LLMOps enables you to harness the power of billions or trillions of parameters, delivering natural and coherent text generation across various language tasks. Cons Human feedback: Human feedback in monitoring and evaluation loops may be used in traditional ML but becomes essential in most LLM applications. Human feedback should be managed like other data, ideally incorporated into monitoring based on near real-time streaming. Limitations and quotas: LLMOps comes with constraints such as token limits, request quotas, response times, and output length, affecting its operational scope. Risky and complex integration: The LLM pipeline will make external API calls, from the model serving endpoint to internal or third-party LLM APIs. This adds complexity, potential latency, and another layer of credential management. Also, integrating large language models as APIs requires technical skills and understanding. Scripting and tool utilization have become integral components, adding to the complexity. Conclusion Automation of workload is variable and intensive and will help in filling the gap between the data science team and the IT operations team. Planning for good governance early in the AI process will minimize AI efforts for data movement to accelerate model development. The emergence of LLMOps highlights the rapid advancement and specialized needs of the field of Generative AI and LLMOps is still rooted in the foundational principles of MLOps. In this article, we have looked at key components, practices, tools, and reference architecture with examples such as: Major similarities and differences between MLOPs and LLOPs Major deployment patterns to migrate data, code, and model Schematics of Ops such as development, staging, and production environments Major approaches to building LLM applications such as prompt engineering, RAGs, fine-tuned, and pre-trained models, and key comparisons LLM serving and observability, including tools and practices for monitoring LLM performance The end-to-end architecture integrates all components across dev, staging, and production environments. CI/CD pipelines automate deployment upon branch merges.

With the rapid development of Internet technology, server-side architectures have become increasingly complex. It is now difficult to rely solely on the personal experience of developers or testers to cover all possible business scenarios. Therefore, real online traffic is crucial for server-side testing. TCPCopy [1] is an open-source traffic replay tool that has been widely adopted by large enterprises. While many use TCPCopy for testing in their projects, they may not fully understand its underlying principles. This article provides a brief introduction to how TCPCopy works, with the hope of assisting readers. Architecture The architecture of TCPCopy has undergone several upgrades, and this article introduces the latest 1.0 version. As shown in the diagram below, TCPCopy consists of two components: tcpcopy and intercept. tcpcopy runs on the online server, capturing live TCP request packets, modifying the TCP/IP header information, and sending them to the test server, effectively "tricking" the test server. intercept runs on an auxiliary server, handling tasks such as relaying response information back to tcpcopy. Figure 1: Overview of the TCPCopy Architecture The simplified interaction process is as follows: tcpcopy captures packets on the online server. tcpcopy modifies the IP and TCP headers, spoofing the source IP and port, and sends the packet to the test server. The spoofed IP address is determined by the -x and -c parameters set at startup. The test server receives the request and returns a response packet with the destination IP and port set to the spoofed IP and port from tcpcopy. The response packet is routed to the intercept server, where intercept captures and parses the IP and TCP headers, typically returning only empty response data to tcpcopy. tcpcopy receives and processes the returned data. Technical Principles TCPCopy operates in two modes: online and offline. The online mode is primarily used for real-time capturing of live request packets, while the offline mode reads request packets from pcap-format files. Despite the difference in working modes, the core principles remain the same. This section provides a detailed explanation of TCPCopy's core principles from several perspectives. 1. Packet Capturing and Sending The core functions of tcpcopy can be summarized as "capturing" and "sending" packets. Let's begin with packet capturing. How do you capture real traffic from the server? Many people may feel confused when first encountering this question. In fact, Linux operating systems already provide the necessary functionality, and a solid understanding of advanced Linux network programming is all that's needed. The initialization of packet capturing and sending in tcpcopy is handled in the tcpcopy/src/communication/tc_socket.c file. Next, we will introduce the two methods tcpcopy uses for packet capturing and packet sending. Raw Socket A raw socket can receive packets from the network interface card on the local machine. This is particularly useful for monitoring and analyzing network traffic. The code for initializing raw socket packet capturing in tcpcopy is shown below, and this method supports capturing packets at both the data link layer and the IP layer. int tc_raw_socket_in_init(int type) { int fd, recv_buf_opt, ret; socklen_t opt_len; if (type == COPY_FROM_LINK_LAYER) { /* Copy ip datagram from Link layer */ fd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_IP)); } else { /* Copy ip datagram from IP layer */ #if (TC_UDP) fd = socket(AF_INET, SOCK_RAW, IPPROTO_UDP); #else fd = socket(AF_INET, SOCK_RAW, IPPROTO_TCP); #endif } if (fd == -1) { tc_log_info(LOG_ERR, errno, "Create raw socket to input failed"); fprintf(stderr, "Create raw socket to input failed:%s\n", strerror(errno)); return TC_INVALID_SOCK; } recv_buf_opt = 67108864; opt_len = sizeof(int); ret = setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &recv_buf_opt, opt_len); if (ret == -1) { tc_log_info(LOG_ERR, errno, "Set raw socket(%d)'s recv buffer failed"); tc_socket_close(fd); return TC_INVALID_SOCK; } return fd; } The code for initializing the raw socket for sending packets is shown below. First, it creates a raw socket at the IP layer and informs the protocol stack not to append an IP header to the IP layer. int tc_raw_socket_out_init(void) { int fd, n; n = 1; /* * On Linux when setting the protocol as IPPROTO_RAW, * then by default the kernel sets the IP_HDRINCL option and * thus does not prepend its own IP header. */ fd = socket(AF_INET, SOCK_RAW, IPPROTO_RAW); if (fd == -1) { tc_log_info(LOG_ERR, errno, "Create raw socket to output failed"); fprintf(stderr, "Create raw socket to output failed: %s\n", strerror(errno)); return TC_INVALID_SOCK; } /* * Tell the IP layer not to prepend its own header. * It does not need setting for linux, but *BSD needs */ if (setsockopt(fd, IPPROTO_IP, IP_HDRINCL, &n, sizeof(n)) < 0) { tc_socket_close(fd); tc_log_info(LOG_ERR, errno, "Set raw socket(%d) option \"IP_HDRINCL\" failed", fd); return TC_INVALID_SOCK; } return fd; } Construct the complete packet and send it to the target server. dst_addr is filled with the target IP address. The IP header is populated with the source and destination IP addresses. The TCP header is filled with the source port, destination port, and other relevant information. Pcap Pcap is an application programming interface (API) provided by the operating system for capturing network traffic, with its name derived from "packet capture." On Linux systems, pcap is implemented via libpcap, and most packet capture tools, such as tcpdump, use libpcap for capturing traffic. Below is the code for initializing packet capture with pcap. int tc_pcap_socket_in_init(pcap_t **pd, char *device, int snap_len, int buf_size, char *pcap_filter) { int fd; char ebuf[PCAP_ERRBUF_SIZE]; struct bpf_program fp; bpf_u_int32 net, netmask; if (device == NULL) { return TC_INVALID_SOCK; } tc_log_info(LOG_NOTICE, 0, "pcap open,device:%s", device); *ebuf = '\0'; if (tc_pcap_open(pd, device, snap_len, buf_size) == TC_ERR) { return TC_INVALID_SOCK; } if (pcap_lookupnet(device, &net, &netmask, ebuf) < 0) { tc_log_info(LOG_WARN, 0, "lookupnet:%s", ebuf); return TC_INVALID_SOCK; } if (pcap_compile(*pd, &fp, pcap_filter, 0, netmask) == -1) { tc_log_info(LOG_ERR, 0, "couldn't parse filter %s: %s", pcap_filter, pcap_geterr(*pd)); return TC_INVALID_SOCK; } if (pcap_setfilter(*pd, &fp) == -1) { tc_log_info(LOG_ERR, 0, "couldn't install filter %s: %s", pcap_filter, pcap_geterr(*pd)); pcap_freecode(&fp); return TC_INVALID_SOCK; } pcap_freecode(&fp); if (pcap_get_selectable_fd(*pd) == -1) { tc_log_info(LOG_ERR, 0, "pcap_get_selectable_fd fails"); return TC_INVALID_SOCK; } if (pcap_setnonblock(*pd, 1, ebuf) == -1) { tc_log_info(LOG_ERR, 0, "pcap_setnonblock failed: %s", ebuf); return TC_INVALID_SOCK; } fd = pcap_get_selectable_fd(*pd); return fd; } The code for initializing packet sending with pcap is as follows: int tc_pcap_snd_init(char *if_name, int mtu) { char pcap_errbuf[PCAP_ERRBUF_SIZE]; pcap_errbuf[0] = '\0'; pcap = pcap_open_live(if_name, mtu + sizeof(struct ethernet_hdr), 0, 0, pcap_errbuf); if (pcap_errbuf[0] != '\0') { tc_log_info(LOG_ERR, errno, "pcap open %s, failed:%s", if_name, pcap_errbuf); fprintf(stderr, "pcap open %s, failed: %s, err:%s\n", if_name, pcap_errbuf, strerror(errno)); return TC_ERR; } return TC_OK; } Raw Socket vs. Pcap Since tcpcopy offers two methods, which one is better? When capturing packets, we are primarily concerned with the specific packets we need. If the capture configuration is not set correctly, the system kernel might capture too many irrelevant packets, leading to packet loss, especially under high traffic pressure. After extensive testing, it has been found that when using the pcap interface to capture request packets, the packet loss rate in live environments is generally higher than when using raw sockets. Therefore, tcpcopy defaults to using raw sockets for packet capture, although the pcap interface can also be used (with the --enable-pcap option), which is mainly suited for high-end pfring captures and captures after switch mirroring. For packet sending, tcpcopy uses the raw socket output interface by default, but it can also send packets via pcap_inject (using the --enable-dlinject option). The choice of which method to use can be determined based on performance testing in your actual environment. 2. TCP Protocol Stack We know that the TCP protocol is stateful. Although the packet-sending mechanism was explained earlier, without establishing an actual TCP connection, the sent packets cannot be truly received by the testing service. In everyday network programming, we typically use the TCP socket interfaces provided by the operating system, which abstract away much of the complexity of TCP states. However, in tcpcopy, since we need to modify the source IP and destination IP of the packets to deceive the testing service, the APIs provided by the operating system are no longer sufficient. As a result, tcpcopy implements a simulated TCP state machine, representing the most complex and challenging aspect of its codebase. The relevant code, located in tcpcopy/src/tcpcopy/tc_session.c, handles crucial tasks such as simulating TCP interactions, managing network latency, and emulating upper-layer interactions. Figure 2: Classic TCP state machine overview In tcpcopy, a session is defined to maintain information for different connections. Different captured packets are processed accordingly: SYN packet: Represents a new connection request. tcpcopy assigns a source IP, modifies the destination IP and port, then sends the packet to the test server. At the same time, it creates a new session to store all states of this connection. ACK packet: Pure ACK packet: To reduce the number of sent packets, tcpcopy generally doesn't send pure ACKs. ACK packet with payload (indicating a specific request): It finds the corresponding session and sends the packet to the test server. If the session is still waiting for the response to the previous request, it delays sending. RST packet: If the current session is waiting for the test server's response, the RST packet is not sent. Otherwise, it's sent. FIN packet: If the current session is waiting for the test server's response, it waits; otherwise, the FIN packet is sent. 3. Routing After tcpcopy sends the request packets, their journey may not be entirely smooth: The IP of the request packet is forged and not the actual IP of the machine running tcpcopy. If some machines have rpfilter (reverse path filtering) enabled, it will check whether the source IP address is trustworthy. If the source IP is untrustworthy, the packet will be discarded at the IP layer. If the test server receives the request packet, the response packet will be sent to the forged IP address. To ensure these response packets don't mistakenly go back to the client with the forged IP, proper routing configuration is necessary. If the routing isn't set up correctly, the response packet won't be captured by intercept, leading to incomplete data exchange. After intercept captures the response packet, it extracts the response packet, and discards the actual data, returning only the response headers and other necessary information to tcpcopy. When necessary, it also merges the return information to reduce the impact on the network of the machine running tcpcopy. 4. Intercept For those new to tcpcopy, it might be puzzling — why is intercept necessary if we already have tcpcopy? While intercept may seem redundant, it actually plays a crucial role. You can think of intercept as the server-side counterpart of tcpcopy, with its name itself explaining its function: an "interceptor." But what exactly does intercept need to intercept? The answer is the response packet from the test service. If intercept were not used, the response packets from the test server would be sent directly to tcpcopy. Since tcpcopy is deployed in a live environment, this means the response packets would be sent directly to the production server, significantly increasing its network load and potentially affecting the normal operation of the live service. With intercept, by spoofing the source IP, the test service is led to "believe" that these spoofed IP clients are accessing it. Intercept also performs aggregation and optimization of the response packet information, further ensuring that the live environment at the network level is not impacted by the test environment. intercept is an independent process that, by default, captures packets using the pcap method. During startup, the -F parameter needs to be passed, for example, "tcp and src port 8080," following libpcap's filter syntax. This means that intercept does not connect directly to the test service but listens on the specified port, capturing the return data packets from the test service and interacting with tcpcopy. 5. Performance tcpcopy uses a single-process, single-thread architecture based on an epoll/select event-driven model, with related code located in the tcpcopy/src/event directory. By default, epoll is used during compilation, though you can switch to select with the --select option. The choice of method can depend on the performance differences observed during testing. Theoretically, epoll performs better when handling a large number of connections. In practical use, tcpcopy's performance is directly tied to the amount of traffic and the number of connections established by intercept. The single-threaded architecture itself is usually not a performance bottleneck (for instance, Nginx and Redis both use single-threaded + epoll models and can handle large amounts of concurrency). Since tcpcopy only establishes connections directly with intercept and does not need to connect to the test machines or occupy port numbers, tcpcopy consumes fewer resources, with the main impact being on network bandwidth consumption. static tc_event_actions_t tc_event_actions = { #ifdef TC_HAVE_EPOLL tc_epoll_create, tc_epoll_destroy, tc_epoll_add_event, tc_epoll_del_event, tc_epoll_polling #else tc_select_create, tc_select_destroy, tc_select_add_event, tc_select_del_event, tc_select_polling #endif }; Conclusion TCPCopy is an excellent open-source project. However, due to the author's limitations, this article only covers the core technical principles of TCPCopy, leaving many details untouched [2]. Nevertheless, I hope this introduction provides some inspiration to those interested in TCPCopy and traffic replay technologies! References [1] GitHub: session-replay-tools/tcpcopy [2] Mobile test development A brief analysis of the principle of TCPCopy, a real-time traffic playback tool

As a solutions architect with over two decades of experience in relational database systems, I recently started exploring MariaDB's new Vector Edition to see if it could address some of the AI data challenges we're facing. A quick look seemed pretty convincing, especially with how it could bring AI magic right into a regular database setup. However, I wanted to test it with a simple use case to see how it performs in practice. In this article, I will share my hands-on experience and observations about MariaDB's vector capabilities by running a simple use case. Specifically, I will be loading sample customer reviews into MariaDB and performing fast similarity searches to find related reviews. Environment Setup Python 3.10 or higher Docker Desktop My experiment started with setting up a Docker container using MariaDB's latest release (11.6) which includes vector capabilities. Shell # Pull the latest release docker pull quay.io/mariadb-foundation/mariadb-devel:11.6-vector-preview # Update password docker run -d --name mariadb_vector -e MYSQL_ROOT_PASSWORD=<replace_password> quay.io/mariadb-foundation/mariadb-devel:11.6-vector-preview Now, create a table and load it with sample customer reviews that include sentiment scores and embeddings for each review. To generate text embeddings, I am using SentenceTransformer, which lets you use pre-trained models. To be specific, I decided to go with a model called paraphrase-MiniLM-L6-v2 that takes our customer reviews and maps them into a 384-dimensional space. Python import mysql.connector import numpy as np from sentence_transformers import SentenceTransformer model = SentenceTransformer('paraphrase-MiniLM-L6-v2') # I already have a database created with a name vectordb connection = mysql.connector.connect( host="localhost", user="root", password="<password>", # Replace me database="vectordb" ) cursor = connection.cursor() # Create a table to store customer reviews with sentiment score and embeddings. cursor.execute(""" CREATE TABLE IF NOT EXISTS customer_reviews ( id INT PRIMARY KEY AUTO_INCREMENT, product_name INT, customer_review TEXT, customer_sentiment_score FLOAT, customer_review_embedding BLOB, INDEX vector_idx (customer_review_embedding) USING HNSW ) ENGINE=ColumnStore; """) # Sample reviews reviews = [ (1, "This product exceeded my expectations. Highly recommended!", 0.9), (1, "Decent quality, but pricey.", 0.6), (2, "Terrible experience. The product does not work.", 0.1), (2, "Average product, ok ok", 0.5), (3, "Absolutely love it! Best purchase I have made this year.", 1.0) ] # Load sample reviews into vector DB for product_id, review_text, sentiment_score in reviews: embedding = model.encode(review_text) cursor.execute( "INSERT INTO customer_reviews (product_id, review_text, sentiment_score, review_embedding) VALUES (%s, %s, %s, %s)", (product_id, review_text, sentiment_score, embedding.tobytes())) connection.commit() connection.close() Now, let's leverage MariaDB's vector capabilities to find similar reviews. This is more like asking "What other customers said similar to this review?". In the below example, I am going to find the top 2 reviews that are similar to a customer review that says "I am super satisfied!". To do this, I am using one of the vector functions (VEC_Distance_Euclidean) available in the latest release. Python # Convert the target customer review into vector target_review_embedding = model.encode("I am super satisfied!") # Find top 2 similar reviews using MariaDB's VEC_Distance_Euclidean function cursor.execute(""" SELECT review_text, sentiment_score, VEC_Distance_Euclidean(review_embedding, %s) AS similarity FROM customer_reviews ORDER BY similarity LIMIT %s """, (target_review_embedding.tobytes(), 2)) similar_reviews = cursor.fetchall() Observations It is easy to set up and we can combine both structured data (like product IDs and sentiment scores), unstructured data (review text), and their vector representations in a single table. I like its ability to use SQL syntax alongside vector operations which makes it easy for teams that are already familiar with relational databases. Here is the full list of vector functions supported in this release. The HNSW index improved the performance of the similarity search query for larger datasets that I tried so far. Conclusion Overall, I am impressed! MariaDB's Vector Edition is going to simplify certain AI-driven architectures. It bridges the gap between the traditional database world and the evolving demands of AI tools. In the coming months, I look forward to seeing how this technology matures and how the community adopts it in real-world applications.

The time for rapid technology development and cloud computing is perhaps the most sensitive time when security issues are of great importance. It is here that security will have to be injected into a process right from the beginning — be it software development or cloud infrastructure deployment. Two concepts that are very influential in doing so are CSPM and DevSecOps. Don't worry if these terms seem complicated — all they really mean is the inclusion of security within how companies build and manage their cloud environments and software pipelines. So, let's break down what CSPM and DevSecOps are, how they fit together, and how they can assist with keeping systems secure in this article. What Is Cloud Security Posture Management? Imagine that there is this huge cloud environment, like a giant digital warehouse, containing data, services, and software. Keeping everything in such a huge environment secure is very difficult. It is here that companies find Cloud Security Posture Management or CSPM. CSPM assists the companies in the following. Track everything: There is now a bird's eye view of an entire cloud infrastructure for companies, which enables them to easily point out something that may be risky, such as misconfiguration or vulnerability. Being compliant: CSPM tools support the idea that everything in the cloud is governed whether it is in line with company policy or with regulatory compliance such as GDPR and HIPAA. Remediate issues in record time: If a problem arises, it will either automatically remediate the issue or suggest remediation. CSPM acts like a thorough security guard in the cloud, ever vigilant and watchful, ensuring everything stays safe and sound. Understanding DevSecOps We'll introduce DevSecOps in simple terms. As the name suggests, we're describing an intersection of three core domains: Dev: The activity of writing and testing software Sec: The protection of software and infrastructure against malicious activities Ops: Ensuring that the software works well and reliably once it goes live Security, before DevSecOps, tended to be an afterthought added simply at the very end of development. This meant that it had delays and would make the system more vulnerable, but with DevSecOps, security is actually integrated all the way through from when you first write a line of code to running the software in production. Key Benefits of DevSecOps Catches issues early: Security checks happen throughout development, catching issues while they are still small problems rather than waiting until they develop into major issues. Delivers fast: Without security, it only tends to the end, so software will come faster and faster. Improves collaboration: Developers, security experts, and operations teams interact with each other more closely to minimize misunderstandings and delays. How Does CSPM Relate to DevSecOps? CSPM tools serve as the security guard for your cloud. When infused into DevSecOps, they ensure that every change in the cloud or during development is made with the best security practices from day one. In a nutshell, here is the integration of CSPM and DevSecOps: Continuous security monitoring: These CSPM tools continuously scan into their cloud environment for risk-readiness. Integration of this into the DevSecOps pipeline ensures security checks occur each and every time new infrastructure is deployed or updated. Automated compliance checks: As more features are added to their cloud infrastructure, CSPM automatically scans whether the concerned infrastructure is compliant with security rules and industry standards in real time. Infrastructure as Code security: DevSecOps teams use tools like Terraform to IaC, or automatically deploy cloud infrastructure. CSPM can scan the IaC templates before anything is live to ensure that configurations are secure from the get-go. The below diagram shows stages of DevSecOps (development, testing, deployment) with continuous CSPM monitoring at each stage. Empowering DevSecOps With CSPM Here's why CSPM is so powerful when added to DevSecOps pipelines: Proactive security: The security solution will be proactive scanning continuously for risks. You don't have to wait till something breaks; you fix issues before they become a problem Speeder compliance: Instead of waiting for time to run checks through, CSPM automates checks to ensure newly deployed software and applications are meeting the security standards at an instance. Higher transparency: The teams of DevSecOps have visibility into all types of cloud assets, their configurations, and the risks. It is such transparency that it makes it easier to manage the cloud environment. Lesser manual patches: Some of the CSPM tools also include an auto-fix feature for most common security issues which saves time and effort for your team. Common Challenges With DevSecOps in Implementing CSPM Even though the benefits are clearly visible, implementing CSPM in DevSecOps pipelines is not very straightforward sometimes. Some of the frequent problems arising in this process are listed below. Complexity of tools: DevSecOps involves a large number of tools for development and deployment purposes. Hence, adding on the CSPM sometimes complicates things if not done very well. Too many alerts: Some of the tools used in CSPM often send too many notifications, which results in "alert fatigue." Thus, the alerts must be fine-tuned in order to make them meaningful. Team collaboration: DevSecOps is truly effective if and only if proper communication between development, security, and operations teams takes place; otherwise, implementing CSPM is going to be pretty challenging. Multi-cloud setups: In most organizations, a multi-cloud environment is implemented. Ensuring consistency in security across multiple clouds might be challenging, but that's exactly what CSPM tools are built for, given the right configurations in place. Infrastructure as Code (IaC) and Pre-Certified Modules The role of CSPM in IaC tools like Terraform is pretty important by scanning the code that expresses the cloud infrastructure. In one practical way, making sure that the deployment is secured can make use of pre-certified modules. Here again, the modules come with baked-in security best practices that enable DevSecOps to build environments from scratch securely. Compliance modules are only deployed here, and they will be continuously monitored. CSPM Tools Here’s a list of CSPM tools: IBM Cloud Security and Compliance Center (SCC) - Provides continuous compliance monitoring, risk management, and policy enforcement for IBM Cloud environments with in-depth audit capabilities Palo Alto Networks Prisma Cloud - Offers multi-cloud security posture management with threat detection, visibility, and automated compliance checks AWS Security Hub - A native AWS service that aggregates security alerts and enables compliance checks across AWS accounts Microsoft Defender for Cloud - Secures workloads across Azure and hybrid cloud environments by assessing security posture and providing real-time threat protection Check Point CloudGuard - Provides posture management, threat intelligence, and automated compliance enforcement for cloud-native applications and multi-cloud environments Aqua Security - Combines CSPM with container and Kubernetes security, offering end-to-end visibility and risk management for cloud infrastructures Wiz - A fast-growing CSPM solution offering deep security insights, prioritizing vulnerabilities and compliance risks across cloud platforms Orca Security - An agentless CSPM tool that provides real-time risk assessment and cloud workload protection for multiple cloud environments CSPM and Beyond In addition to CSPM, there are several other cloud security tools and frameworks designed to ensure the safety, compliance, and efficiency of cloud environments. Here are some of the key tools commonly used alongside or as alternatives to CSPM: Cloud Workload Protection Platform (CWPP) Secures cloud-based workloads, including virtual machines (VMs), containers, and serverless functions Includes vulnerability management, system integrity monitoring, runtime protection, and network segmentation Cloud Access Security Broker (CASB) Acts as a gatekeeper between users and cloud service providers, ensuring secure access to cloud services Provides visibility, compliance, data security, and threat protection for cloud applications. Cloud Infrastructure Entitlement Management (CIEM) Focuses on managing and securing permissions and access to cloud resources Helps with least privilege enforcement, identity governance, and mitigating risks of misconfigurations Cloud-Native Application Protection Platform (CNAPP) Provides a comprehensive suite that integrates CSPM, CWPP, and more to secure applications across development and production Encompasses vulnerability management, runtime security, and compliance for cloud-native applications like containers and Kubernetes Security Information and Event Management (SIEM) Centralized logging and analysis of security events from cloud infrastructure and applications Enables threat detection, incident response, and compliance reporting Runtime Application Self-Protection (RASP) Provides real-time protection for applications while they are running in the cloud Detects and mitigates attacks by monitoring the behavior of an application and blocking malicious activity Security Orchestration, Automation, and Response (SOAR) Automates security operations and workflows to reduce manual effort in threat detection and response. Coordinates multiple security tools to streamline threat management and incident response. Conclusion: The Force of Security From the Start This enables companies to build secure, compliant, and fast cloud environments. Companies are able to move fast while staying ahead of security threats by integrating security throughout every stage of development and cloud management. Tools like CSPM make sure no cloud misconfiguration slips through and with this approach, DevSecOps carries out this process — that of being collaborative and fast. The integration of security is essentially a core part of every decision. If you're into cloud infrastructure, think about what kinds of such tools and practices you might bring into your processes. By putting security into your applications at the beginning, you save time, decrease risks, and give a more solid environment for your applications.

By Josephine Eskaline Joyce CORE

This is another article in the series related to supporting the Postgres JSON functions in a project using the Hibernate framework with version 6. The topic for the article is modification operations on JSON records. As in the previous article, it is worth mentioning that Postgres might now have such comprehensive operations as other NoSQL databases like MongoDB for JSON modification (although, with the proper function constructions, it is possible to achieve the same effect). It still suits most projects that require JSON modification. Plus, with transaction support (not support in a NoSQL database at such a level), it is a pretty good idea to use Postgres with JSON data. Of course, NoSQL databases have other benefits that might suit better projects. There are generally many articles on Postgres' support for JSON. This article focuses on integrating this support with the Hibernate 6 library. In case someone is interested in querying JSON data or text search using Postgres and Hibernate, please see the below links: Postgres Full-Text Search With Hibernate 6 Postgres JSON Functions With Hibernate 6 Test Data For the article, let's assume that our database has a table called the item, which has a column with JSON content, like in the below example: SQL create table item ( id int8 not null, jsonb_content jsonb, primary key (id) ) We also might have some test data: SQL INSERT INTO item (id, jsonb_content) VALUES (1, '{"top_element_with_set_of_values":["TAG1","TAG2","TAG11","TAG12","TAG21","TAG22"]}'); INSERT INTO item (id, jsonb_content) VALUES (2, '{"top_element_with_set_of_values":["TAG3"]}'); -- item without any properties, just an empty json INSERT INTO item (id, jsonb_content) VALUES (6, '{}'); -- int values INSERT INTO item (id, jsonb_content) VALUES (7, '{"integer_value": 132}'); -- double values INSERT INTO item (id, jsonb_content) VALUES (10, '{"double_value": 353.01}'); INSERT INTO item (id, jsonb_content) VALUES (11, '{"double_value": -1137.98}'); -- enum values INSERT INTO item (id, jsonb_content) VALUES (13, '{"enum_value": "SUPER"}'); -- string values INSERT INTO item (id, jsonb_content) VALUES (18, '{"string_value": "the end of records"}'); Native SQL Execution Like in other Java frameworks, with Hibernate, you can run native SQL queries — which is well documented and there are a lot of examples on the internet. That is why in this article, we won't focus on native SQL operation execution. However, there will be examples of what kind of SQL the JPA operations generate. Because Hibernate is a JPA implementation, it makes sense to show how JPA API can modify JSON data in the Postgres database. Modify JSON Object Properties and Not the Entire JSON Object (Path) Setting the whole JSON payload for one column is easy and does not require much explanation. We just set the value for the property in our Entity class, which represents a column with JSON content. It is similar to setting single or multiple properties for JSON for one database row. We just read the table row, deserialize the JSON value to a POJO representing a JSON object, set values for particular properties, and update the database records with the whole payload. However, such an approach might not be practical when we want to modify JSON properties for multiple database rows. Suppose we have to do batch updates of particular JSON properties. Fetching from the database and updating each record might not be an effective method. It would be much better to do such an update with one update statement where we set values for particular JSON properties. Fortunately, Postgres has functions that modify JSON content and can be used in the SQL update statement. Posjsonhelper Hibernate has better support for JSON modification in version 7, including most of the functions and operators mentioned in this article. Still, there are no plans to add such support in version 6. Fortunately, the Posjsonhelper project adds such support for Hibernate in version 6. All the examples below will use the Posjsonhelper library. Check this link to find out how to attach a library to your Java project. You will also have to attach FunctionContributor. All examples use Java entity class that represents the item table, whose definition was mentioned above: Java import jakarta.persistence.Column; import jakarta.persistence.Entity; import jakarta.persistence.Id; import jakarta.persistence.Table; import org.hibernate.annotations.JdbcTypeCode; import org.hibernate.annotations.Type; import org.hibernate.type.SqlTypes; import java.io.Serializable; @Entity @Table(name = "item") public class Item implements Serializable { @Id private Long id; @JdbcTypeCode(SqlTypes.JSON) @Column(name = "jsonb_content") private JsonbContent jsonbContent; public Long getId() { return id; } public void setId(Long id) { this.id = id; } public JsonbContent getJsonbContent() { return jsonbContent; } public void setJsonbContent(JsonbContent jsonbContent) { this.jsonbContent = jsonbContent; } } jsonb_set Function Wrapper The jsonb_set function is probably the most helpful function when modifying JSON data is required. It allows specific properties for JSON objects and specific array elements to be set based on the array index. For example, the below code adds the property "birthday" to the inner property "child". Java // GIVEN Long itemId = 19L; String property = "birthday"; String value = "1970-01-01"; String expectedJson = "{\"child\": {\"pets\" : [\"dog\"], \"birthday\": \"1970-01-01\"}"; // when CriteriaUpdate<Item> criteriaUpdate = entityManager.getCriteriaBuilder().createCriteriaUpdate(Item.class); Root<Item> root = criteriaUpdate.from(Item.class); // Set the property you want to update and the new value criteriaUpdate.set("jsonbContent", new JsonbSetFunction((NodeBuilder) entityManager.getCriteriaBuilder(), root.get("jsonbContent"), new JsonTextArrayBuilder().append("child").append(property).build().toString(), JSONObject.quote(value), hibernateContext)); // Add any conditions to restrict which entities will be updated criteriaUpdate.where(entityManager.getCriteriaBuilder().equal(root.get("id"), itemId)); // Execute the update entityManager.createQuery(criteriaUpdate).executeUpdate(); // then Item item = tested.findById(itemId); assertThat((String) JsonPath.read(item.getJsonbContent(), "$.child." + property)).isEqualTo(value); JSONObject jsonObject = new JSONObject(expectedJson); DocumentContext document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo(jsonObject.toString()); This code would generate such a SQL statement: SQL update item set jsonb_content=jsonb_set(jsonb_content, ?::text[], ?::jsonb) where id=? Hibernate: select i1_0.id, i1_0.jsonb_content from item i1_0 where i1_0.id=? Concatenation Operator Wrapper "||" The wrapper for the concatenation operator (||) concatenates two JSONB values into a new JSONB value. Based on Postgres documentation, the operator behavior is as follows: Concatenating two arrays generates an array containing all the elements of each input. Concatenating two objects generates an object containing the union of their keys, taking the second object's value when there are duplicate keys. All other cases are treated by converting a non-array input into a single-element array, and then proceeding as for two arrays. Does not operate recursively: only the top-level array or object structure is merged. Here is an example of how to use this wrapper in your code: Java // GIVEN Long itemId = 19l; String property = "birthday"; String value = "1970-01-01"; // WHEN CriteriaUpdate<Item> criteriaUpdate = entityManager.getCriteriaBuilder().createCriteriaUpdate(Item.class); Root<Item> root = criteriaUpdate.from(Item.class); JSONObject jsonObject = new JSONObject(); jsonObject.put("child", new JSONObject()); jsonObject.getJSONObject("child").put(property, value); criteriaUpdate.set("jsonbContent", new ConcatenateJsonbOperator((NodeBuilder) entityManager.getCriteriaBuilder(), root.get("jsonbContent"), jsonObject.toString(), hibernateContext)); criteriaUpdate.where(entityManager.getCriteriaBuilder().equal(root.get("id"), itemId)); entityManager.createQuery(criteriaUpdate).executeUpdate(); // THEN Item item = tested.findById(itemId); assertThat((String) JsonPath.read(item.getJsonbContent(), "$.child." + property)).isEqualTo(value); JSONObject expectedJsonObject = new JSONObject().put(property, value); DocumentContext document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$.child")); assertThat(document.jsonString()).isEqualTo(expectedJsonObject.toString()); Code merge a JSON object with the child property with the already stored JSON object in the database. This code generates such a SQL query: SQL update item set jsonb_content=jsonb_content || ?::jsonb where id=? Hibernate: select i1_0.id, i1_0.jsonb_content from item i1_0 where i1_0.id=? Delete the Field or Array Element Based on the Index at the Specified Path "#-" The Posjsonhelper has a wrapper for the delete operation (#-). It deletes the field or array element based on the index at the specified path, where path elements can be either field keys or array indexes. For example, the below code removes from the JSON object property based on the "child.pets" JSON path. Java // GIVEN Item item = tested.findById(19L); JSONObject jsonObject = new JSONObject("{\"child\": {\"pets\" : [\"dog\"]}"); DocumentContext document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo(jsonObject.toString()); // WHEN CriteriaUpdate<Item> criteriaUpdate = entityManager.getCriteriaBuilder().createCriteriaUpdate(Item.class); Root<Item> root = criteriaUpdate.from(Item.class); // Set the property you want to update and the new value criteriaUpdate.set("jsonbContent", new DeleteJsonbBySpecifiedPathOperator((NodeBuilder) entityManager.getCriteriaBuilder(), root.get("jsonbContent"), new JsonTextArrayBuilder().append("child").append("pets").build().toString(), hibernateContext)); // Add any conditions to restrict which entities will be updated criteriaUpdate.where(entityManager.getCriteriaBuilder().equal(root.get("id"), 19L)); // Execute the update entityManager.createQuery(criteriaUpdate).executeUpdate(); // THEN entityManager.refresh(item); jsonObject = new JSONObject("{\"child\": {}"); document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo(jsonObject.toString()); The generated SQL would be: SQL update item set jsonb_content=(jsonb_content #- ?::text[]) where id=? Delete Multiple Array Elements at the Specified Path By default, Postgres (at least in version 16) does not have a built-in function that allows the removal of array elements based on their value. However, it does have the built-in operator, -#, which we mentioned above, that helps to delete array elements based on the index but not their value. For this purpose, the Posjsonhelper can generate a function that must be added to the DDL operation and executed on your database. SQL CREATE OR REPLACE FUNCTION {{schema}.remove_values_from_json_array(input_json jsonb, values_to_remove jsonb) RETURNS jsonb AS $$ DECLARE result jsonb; BEGIN IF jsonb_typeof(values_to_remove) <> 'array' THEN RAISE EXCEPTION 'values_to_remove must be a JSON array'; END IF; result := ( SELECT jsonb_agg(element) FROM jsonb_array_elements(input_json) AS element WHERE NOT (element IN (SELECT jsonb_array_elements(values_to_remove))) ); RETURN COALESCE(result, '[]'::jsonb); END; $$ LANGUAGE plpgsql; One of the wrappers will use this function to allow the deletion of multiple values from the JSON array. This code removes a "mask" and "compass" elements for the "child.inventory" property. Java // GIVEN Item item = tested.findById(24L); DocumentContext document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo("{\"child\":{\"pets\":[\"crab\",\"chameleon\"]},\"inventory\":[\"mask\",\"fins\",\"compass\"]}"); CriteriaUpdate<Item> criteriaUpdate = entityManager.getCriteriaBuilder().createCriteriaUpdate(Item.class); Root<Item> root = criteriaUpdate.from(Item.class); NodeBuilder nodeBuilder = (NodeBuilder) entityManager.getCriteriaBuilder(); JSONArray toRemoveJSONArray = new JSONArray(Arrays.asList("mask", "compass")); RemoveJsonValuesFromJsonArrayFunction deleteOperator = new RemoveJsonValuesFromJsonArrayFunction(nodeBuilder, new JsonBExtractPath(root.get("jsonbContent"), nodeBuilder, Arrays.asList("inventory")), toRemoveJSONArray.toString(), hibernateContext); JsonbSetFunction jsonbSetFunction = new JsonbSetFunction(nodeBuilder, (SqmTypedNode) root.get("jsonbContent"), new JsonTextArrayBuilder().append("inventory").build().toString(), deleteOperator, hibernateContext); // Set the property you want to update and the new value criteriaUpdate.set("jsonbContent", jsonbSetFunction); // Add any conditions to restrict which entities will be updated criteriaUpdate.where(entityManager.getCriteriaBuilder().equal(root.get("id"), 24L)); // WHEN entityManager.createQuery(criteriaUpdate).executeUpdate(); // THEN entityManager.refresh(item); document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo("{\"child\":{\"pets\":[\"crab\",\"chameleon\"]},\"inventory\":[\"fins\"]}"); Here is the SQL generated by the above code: SQL update item set jsonb_content=jsonb_set(jsonb_content, ?::text[], remove_values_from_json_array(jsonb_extract_path(jsonb_content, ?), ?::jsonb)) where id=? Hibernate6JsonUpdateStatementBuilder: How To Combine Multiple Modification Operations With One Update Statement All the above examples demonstrated the execution of a single operation that modifies JSON data. Of course, we can have update statements in our code that use many of the wrappers mentioned in this article together. However, being aware of how those operations and functions will be executed is crucial because it makes the most sense when the result of the first JSON operation is an input for the following JSON modification operations. The output for that operation would be an input for the next operation, and so on, until the last JSON modification operation. To better illustrate that, check the SQL code. SQL update item set jsonb_content= jsonb_set( jsonb_set( jsonb_set( jsonb_set( ( (jsonb_content #- ?::text[]) -- the most nested #- operator #- ?::text[]) , ?::text[], ?::jsonb) -- the most nested jsonb_set operation , ?::text[], ?::jsonb) , ?::text[], ?::jsonb) , ?::text[], ?::jsonb) where id=? This assumes that we have four jsonb_set function executions and two delete operations. The most nested delete operation is a first JSON modification operation because the original value from a column that stores JSON data is passed as a parameter. Although this is the correct approach, and the existing wrapper allows the creation of such an UPDATE statement, it might not be readable from a code perspective. Fortunately, Posjsonhelper has a builder component that makes building such a complex statement easy. The Hibernate6JsonUpdateStatementBuilder type allows the construction of update statements with multiple operations that modify JSON and rely on each other. Below is a code example: Java // GIVEN Item item = tested.findById(23L); DocumentContext document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo("{\"child\":{\"pets\":[\"dog\"]},\"inventory\":[\"mask\",\"fins\"],\"nicknames\":{\"school\":\"bambo\",\"childhood\":\"bob\"}"); CriteriaUpdate<Item> criteriaUpdate = entityManager.getCriteriaBuilder().createCriteriaUpdate(Item.class); Root<Item> root = criteriaUpdate.from(Item.class); Hibernate6JsonUpdateStatementBuilder hibernate6JsonUpdateStatementBuilder = new Hibernate6JsonUpdateStatementBuilder(root.get("jsonbContent"), (NodeBuilder) entityManager.getCriteriaBuilder(), hibernateContext); hibernate6JsonUpdateStatementBuilder.appendJsonbSet(new JsonTextArrayBuilder().append("child").append("birthday").build(), quote("2021-11-23")); hibernate6JsonUpdateStatementBuilder.appendJsonbSet(new JsonTextArrayBuilder().append("child").append("pets").build(), "[\"cat\"]"); hibernate6JsonUpdateStatementBuilder.appendDeleteBySpecificPath(new JsonTextArrayBuilder().append("inventory").append("0").build()); hibernate6JsonUpdateStatementBuilder.appendJsonbSet(new JsonTextArrayBuilder().append("parents").append(0).build(), "{\"type\":\"mom\", \"name\":\"simone\"}"); hibernate6JsonUpdateStatementBuilder.appendJsonbSet(new JsonTextArrayBuilder().append("parents").build(), "[]"); hibernate6JsonUpdateStatementBuilder.appendDeleteBySpecificPath(new JsonTextArrayBuilder().append("nicknames").append("childhood").build()); // Set the property you want to update and the new value criteriaUpdate.set("jsonbContent", hibernate6JsonUpdateStatementBuilder.build()); // Add any conditions to restrict which entities will be updated criteriaUpdate.where(entityManager.getCriteriaBuilder().equal(root.get("id"), 23L)); // WHEN entityManager.createQuery(criteriaUpdate).executeUpdate(); // THEN entityManager.refresh(item); document = JsonPath.parse((Object) JsonPath.read(item.getJsonbContent(), "$")); assertThat(document.jsonString()).isEqualTo("{\"child\":{\"pets\":[\"cat\"],\"birthday\":\"2021-11-23\"},\"parents\":[{\"name\":\"simone\",\"type\":\"mom\"}],\"inventory\":[\"fins\"],\"nicknames\":{\"school\":\"bambo\"}"); The SQL statement that was mentioned previously was generated by this code. To know more about how the builder works, please check the documentation. Conclusion Postgres database has a wide range of possibilities regarding JSON data modification operations. This leads us to consider Postgres a good document storage solution choice. So if our solution does not require higher read performance, better scaling, or sharding (although all those things could be achieved with Postgres database, especially with solutions provided by cloud providers like AWS), then is it worth considering storing your JSON documents in Postgres database — not to mention transaction support with databases like Postgres.

By Szymon Tarnowski CORE

Full stack development is often likened to an intricate balancing act, where developers are expected to juggle multiple responsibilities across the front end, back end, database, and beyond. As the definition of full-stack development continues to evolve, so too does the approach to debugging. Full stack debugging is an essential skill for developers, as it involves tracking issues through multiple layers of an application, often navigating domains where one’s knowledge may only be cursory. In this blog post, I aim to explore the nuances of full-stack debugging, offering practical tips and insights for developers navigating the complex web of modern software development. Notice that this is an introductory post focusing mostly on the front-end debugging aspects. In the following posts, I will dig deeper into the less familiar capabilities of front-end debugging. Full Stack Development: A Shifting Definition The definition of full stack development is as fluid as the technology stacks themselves. Traditionally, full-stack developers were defined as those who could work on both the front end and back end of an application. However, as the industry evolves, this definition has expanded to include aspects of operations (OPS) and configuration. The modern full-stack developer is expected to submit pull requests that cover all parts required to implement a feature — backend, database, frontend, and configuration. While this does not make them an expert in all these areas, it does require them to navigate across domains, often relying on domain experts for guidance. I've heard it said that full-stack developers are: Jack of all trades, master of none. However, the full quote probably better represents the reality: Jack of all trades, master of none, but better than a master of one. The Full Stack Debugging Approach Just as full-stack development involves working across various domains, full-stack debugging requires a similar approach. A symptom of a bug may manifest in the front end, but its root cause could lie deep within the backend or database layers. Full stack debugging is about tracing these issues through the layers and isolating them as quickly as possible. This is no easy task, especially when dealing with complex systems where multiple layers interact in unexpected ways. The key to successful full-stack debugging lies in understanding how to track an issue through each layer of the stack and identifying common pitfalls that developers may encounter. Frontend Debugging: Tools and Techniques It Isn't "Just Console.log" Frontend developers are often stereotyped as relying solely on Console.log for debugging. While this method is simple and effective for basic debugging tasks, it falls short when dealing with the complex challenges of modern web development. The complexity of frontend code has increased significantly, making advanced debugging tools not just useful, but necessary. Yet, despite the availability of powerful debugging tools, many developers continue to shy away from them, clinging to old habits. The Power of Developer Tools Modern web browsers come equipped with robust developer tools that offer a wide range of capabilities for debugging front-end issues. These tools, available in browsers like Chrome and Firefox, allow developers to inspect elements, view and edit HTML and CSS, monitor network activity, and much more. One of the most powerful, yet underutilized, features of these tools is the JavaScript debugger. The debugger allows developers to set breakpoints, step through code, and inspect the state of variables at different points in the execution. However, the complexity of frontend code, particularly when it has been obfuscated for performance reasons, can make debugging a challenging task. We can launch the browser tools on Firefox using this menu: On Chrome we can use this option: I prefer working with Firefox, I find their developer tools more convenient but both browsers have similar capabilities. Both have fantastic debuggers (as you can see with the Firefox debugger below); unfortunately, many developers limit themselves to console printing instead of exploring this powerful tool. Tackling Code Obfuscation Code obfuscation is a common practice in frontend development, employed to protect proprietary code and reduce file sizes for better performance. However, obfuscation also makes the code difficult to read and debug. Fortunately, both Chrome and Firefox developer tools provide a feature to de-obfuscate code, making it more readable and easier to debug. By clicking the curly brackets button in the toolbar, developers can transform a single line of obfuscated code into a well-formed, debuggable file. Another important tool in the fight against obfuscation is the source map. Source maps are files that map obfuscated code back to its original source code, including comments. When generated and properly configured, source maps allow developers to debug the original code instead of the obfuscated version. In Chrome, this feature can be enabled by ensuring that “Enable JavaScript source maps” is checked in the developer tools settings. You can use code like this in the JavaScript file to point at the sourcemap file: //@sourceMappingURL=myfile.js.map For this to work in Chrome we need to ensure that “Enable JavaScript source maps” is checked in the settings. Last I checked it was on by default, but it doesn’t hurt to verify: Debugging Across Layers Isolating Issues Across the Stack In full-stack development, issues often manifest in one layer but originate in another. For example, a frontend error might be caused by a misconfigured backend service or a database query that returns unexpected results. Isolating the root cause of these issues requires a methodical approach, starting from the symptom and working backward through the layers. A common strategy is to reproduce the issue in a controlled environment, such as a local development setup, where each layer of the stack can be tested individually. This helps to narrow down the potential sources of the problem. Once the issue has been isolated to a specific layer, developers can use the appropriate tools and techniques to diagnose and resolve it. The Importance of System-Level Debugging Full stack debugging is not limited to the application code. Often, issues arise from the surrounding environment, such as network configurations, third-party services, or hardware limitations. A classic example of this that we ran into a couple of years ago was a production problem where a WebSocket connection would frequently disconnect. After extensive debugging, Steve discovered that the issue was caused by the CDN provider (CloudFlare) timing out the WebSocket after two minutes — something that could only be identified by debugging the entire system, not just the application code. System-level debugging requires a broad understanding of how different components of the infrastructure interact with each other. It also involves using tools that can monitor and analyze the behavior of the system as a whole, such as network analyzers, logging frameworks, and performance monitoring tools. Embracing Complexity Full stack debugging is inherently complex, as it requires developers to navigate multiple layers of an application, often dealing with unfamiliar technologies and tools. However, this complexity also presents an opportunity for growth. By embracing the challenges of full-stack debugging, developers can expand their knowledge and become more versatile in their roles. One of the key strengths of full-stack development is the ability to collaborate with domain experts. When debugging an issue that spans multiple layers, it is important to leverage the expertise of colleagues who specialize in specific areas. This collaborative approach not only helps to resolve issues more efficiently but also fosters a culture of knowledge sharing and continuous learning within the team. As tools continue to evolve, so too do the tools and techniques available for debugging. Developers should strive to stay up-to-date with the latest advancements in debugging tools and best practices. Whether it’s learning to use new features in browser developer tools or mastering system-level debugging techniques, continuous learning is essential for success in full-stack development. Video Conclusion Full stack debugging is a critical skill for modern developers, we mistakenly think it requires a deep understanding of both the application and its surrounding environment. I disagree: By mastering the tools and techniques discussed in this post/upcoming posts, developers can more effectively diagnose and resolve issues that span multiple layers of the stack. Whether you’re dealing with obfuscated frontend code, misconfigured backend services, or system-level issues, the key to successful debugging lies in a methodical, collaborative approach. You don't need to understand every part of the system, just the ability to eliminate the impossible.

By Shai Almog CORE

Equality in Java Object equality is often a hot topic for assessing concepts and one of the pillars (the other is- hashCode()) of how many of the implementations of Collection Frameworks work. We check equality by providing our own implementation for the method public booleanjava.lang.Object#equals(java.lang.Object other). According to Oracle documentation, the following mandates should be adhered to: It is reflexive: For any non-null reference value x, x.equals(x) should return true. It is symmetric: For any non-null reference values x and y, x.equals(y) should return true if and only if y.equals(x) returns true. It is transitive: For any non-null reference values x, y, and z, if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) should return true. It is consistent: For any non-null reference values x and y, multiple invocations of x.equals(y) consistently return true or consistently return false, provided no information used in equals comparisons on the objects are modified. For any non-null reference value x, x.equals(null) should return false. Please note that there exist a few more related to using it along with hashCode(), but we do not discuss them here for brevity, assuming the readers are already aware of them. Reference Equality or Content Equality? The term "equality" can itself be ambiguous, since we can either talk about reference equality or be interested in content equality. Let us illustrate both with a simple example. However, the reader may choose to skip this section and jump into the main topic of discussion at one's own discretion. Assume a class (POJO), LaptopCharger: Java package com.yourcompany.model; /** * An AC-to-DC converter LaptopCharger * */ public class LaptopCharger { private String manufacturer; private int wattage; // Consumption: Volt times Amp. private float outputCurrent; // output Amp. private float outputVoltage; // output Volt private double price; private String connectorJackType; // E.g. USB-C, pin etc. // Setters and Getters follow here } Note that we did not override any method of java.lang.Object (which, is inherited by any Java class); the default implementations, therefore, apply here. The below code snippet outputs false false: Java LaptopCharger charger_A = new LaptopCharger(65, 3.25f, 19.0f, 100, "usb-c"); LaptopCharger charger_B =new LaptopCharger(65, 3.25f, 19.0f, 100, "usb-c"); boolean refEqulas=charger_A==charger_B; boolean equals=charger_A.equals(charger_B); System.out.println(refEqulas+" "+equals); We see that reference equality is the default return value of the equals method. However, consider that Bob was searching a popular e-commerce site for a charger for his laptop. His laptop requires a 65-watt/19.8Volt type-C charger, but he finds that the one given by his laptop manufacturer is not going to reach him anytime soon. He, therefore, searches for a close alternative. The meaning of equality, in this case, is content equality as shown below: Java @Override public boolean equals(Object obj) { if (null == obj) return false; if (obj == this) return true; if (!(obj instanceof LaptopCharger)) return false; LaptopCharger other = (LaptopCharger) obj; return this.wattage == other.wattage && this.outputCurrent == other.outputCurrent && this.connectorJackType.equals(this.connectorJackType); } The output is: false true. However, the equals method can be overridden if these conditions are met: The code, i.e. LaptopCharger is open to us. This logic is accepted across the business domain. Otherwise, we can use Objects.compare(..) somewhat like the following:(Important note: Unless we are certain about ordering the objects, it may be against the prescribed contract to use Comprator<T> for just checking content equality.) Java Comparator<LaptopCharger> specificationComparator=(x,y)->{ if(x.wattage == y.wattage && x.outputCurrent == y.outputCurrent && x.connectorJackType.equals(y.connectorJackType)) return 0; else return 1; }; int t=Objects.compare(charger_A, charger_B, specificationComparator); System.out.println(t); How Much Equal Is an Object to Another?: Degree of Equality So far, we talked about content equality and it was all in black and white. However, what if we needed to check the degree of equality beyond just false and true? To elaborate on this point, let us assume the following fields: Equality of wattage, outputCurrent, and outputVoltage Equality of charger connectivity : connectorJackType Brand: manufacturer Price of the item Hypothetical business requirements are: If all 4 points above are the same, we consider 100% equality. [2] must be the same. A small variation in output current and voltage may be permissible. (Alert: in real life, this may not be the best practice!) The manufacturer of the charger is not required to be the same as the laptop's but is recommended to be. Price: Customers always hunt for low prices, discounts, and of course, value for money! A small compromise for a few other constraints is granted. Restricting the discussion to Java SE 17, we can address this scenario using third-party libraries like Fuzzy-Matcher, etc. However, would this not just be great if Java itself handled this by using a utility method in java.util.Objects? Note that it does not until this version. I just wish it were a part of Java SE and here itself! Below is a small and coarse prototype to illustrate what would have been good to have: Java /** * @param t1 * @param t2 * @param fuzzyComparator * @return R the result. No type is enforced to provide more flexibility */ public static <T, R> R fuzzyEquals(T t1, T t2, BiFunction<? super T, ? super T, R> fuzzyComparator) { return fuzzyComparator.apply(t1, t2); } The first two parameters are of type T and last one, the comparator itself is a BiFunction<? super T, ? super T, R>. In this example, I did not enforce a return type for the method, leveraging the power of generics and functional programming to provide more flexibility. This eliminates the need for a strict return type such as double as well as a dedicated functional interface like FuzzyComprator which would otherwise have looked somewhat like this: Java @FunctionalInterface public interface Comparator<T>{ // other stuff like static, default methods etc. double compare(T o1, T o2) } Below is a simple illustration using it: Java BiFunction<LaptopCharger, LaptopCharger, OptionalDouble> mySimpleFuzzyCompartor = (x, y) -> { if (x.connectorJackType.equals(y.connectorJackType)) { if (x.wattage == y.wattage && x.outputCurrent == y.outputCurrent && x.manufacturer.equals(y.manufacturer) && x.price == y.price) return OptionalDouble.of(1.0D); // Full match if (x.wattage == y.wattage && x.outputCurrent == y.outputCurrent && x.manufacturer.equals(y.manufacturer)) return OptionalDouble.of(1.0 - (x.price - y.price) / x.price);// Price based match if (x.wattage == y.wattage && x.outputCurrent == y.outputCurrent) return OptionalDouble.of(1.0 - 0.2 - (x.price - y.price) / x.price); // if (x.wattage == y.wattage && Math.abs(x.outputCurrent - y.outputCurrent) < 0.15) return OptionalDouble .of(1.0 - 0.2 - Math.abs((x.outputCurrent - y.outputCurrent) / x.outputCurrent)); return OptionalDouble.empty(); } else { return OptionalDouble.empty(); } }; OptionalDouble fuzzyEquals = fuzzyEquals(charger_A, charger_B, mySimpleFuzzyCompartor); System.out.println(fuzzyEquals); We used OptionalDouble as the return type of the fuzzyEquals. Readers are strongly encouraged to introduce the method, fuzzyEquals, in java.util.Objects and use it and get it benchmarked. Once we have that satisfactory, Collection Frameworks might be made to undergo relevant contract upgradation to strongly support beyond-the-Boolean comparison!

Distributed Application Runtime (Dapr) is a portable and event-driven runtime that commoditizes some of the problems developers face with distributed systems and microservices daily. Imagine there are 3-4 different microservices. As part of communication between these services, developers must think about: Handling timeouts and failures Metrics and traces Controlling and restricting communication between services. These challenges are recurring, but with Dapr's Service-to-Service Invocation building block, they are seamlessly abstracted. Dapr divides such capabilities into components that can be invoked using a building block, aka API. Components Overview Below mentioned are a subset of components that Dapr supports. Component Description Service-to-Service Facilitates communication between microservices: It encapsulates handling failures, observability, and applying policies (responsible for enforcing restrictions on who is allowed to call) Secrets Facilitate communication with cloud secrets and Kubernetes secrets provider stores Workflows With the Workflows component, developers can run long-running workloads distributed across nodes. Publish/Subscribe Similar to the producer/consumer pattern, with this component messages can be produced to a topic and listeners can consume from the subscribed topic. Let's dive into the workflow component. Workflow Component Problem An example of a simple Workflow can be a scheduled job that moves data between data sources. The complexity increases when child workflows must be triggered as part of the parent workflow and the workflow author also becomes responsible for saving, resuming, and maintaining the state and the schema. With the Dapr Workflow component, most of the state management is abstracted out, allowing developers to focus only on the business logic. Key Terms Workflow: Contains a set of tasks that need to be executed Activities: Tasks that need to be executed; For example, in the previous work where data must be moved from source to destination: Activity 1: Reads data from Source Activity 2: Writes to the destination Workflow will compromise both these activities. Benefits Using Workflow Replays we inherently get checkpointing mechanism. For example, in the C# async/await model, Dapr automatically checkpoints at each await call. This allows the system to recover from the most recent I/O operation during a failure, making recovery less costly. Built-in retry strategies for the workflows and activities are customizable to suit specific workflows. Workflow Patterns Pattern 1 The parent workflow parallelly schedules multiple child activities. Pattern 2 In this scenario, the workflow schedules Activity 1 and then passes its output to Activity 2 for further processing. Pattern 3 Here, the parent workflow schedules another child workflow which in turn schedules some activities. Example Let's explore an example using C# and Dapr to schedule workflows that read data from Blob storage. Step 1 Import the Dapr packages into csproj. XML <ItemGroup> # https://www.nuget.org/packages/Dapr.AspNetCore <PackageReference Include="Dapr.AspNetCore" Version="1.14.0" ></PackageReference> # https://www.nuget.org/packages/Dapr.Workflow <PackageReference Include="Dapr.Workflow" Version="1.14.0" ></PackageReference> </ItemGroup> Step 2: Configuring Workflow and Activity Add workflow and activities to the Dapr Workflow extension. "Register Workflow" is used to register workflows. "Register Activity" is used to register activity. C# /// <summary> /// Configure workflow extension. /// </summary> public static class DaprConfigurationExtension { /// <summary> /// Configure Workflow extension. /// </summary> /// <param name="services">services.</param> /// <returns>IServiceCollection.</returns> public static IServiceCollection ConfigureDaprWorkflows(this IServiceCollection services) { services.AddDaprWorkflow(options => { // Note that it's also possible to register a lambda function as the workflow // or activity implementation instead of a class. options.RegisterWorkflow<BlobOrchestrationWorkflow>(); // These are the activities that get invoked by the Dapr workflow(s). options.RegisterActivity<BlobDataFetchActivity>(); }); return services; } } Step 3: Writing the First Workflow The Blob Orchestration Workflow implements Workflow coming from Dapr NuGet with input and output parameters. The input here is the name of the blob, which is a string, and the output is content from the blob, nothing but a list of lines. C# /// <summary> /// Dapr workflow responsible for peforming operations on blob. /// </summary> public class BlobOrchestrationWorkflow : Workflow<string, List<string>> { /// <inheritdoc/> public async override Task<List<string>> RunAsync(WorkflowContext context, string input) { ArgumentNullException.ThrowIfNull(context); ArgumentNullException.ThrowIfNull(input); List<string> identifiers = await context.CallActivityAsync<List<string>>( name: nameof(BlobDataFetchActivity), input: input).ConfigureAwait(false); // state is saved return identifiers; } } Step 4: Writing the First Activity Like Workflow, Activity also takes input and output. In this case, input is the blob name, and output is the list of lines from the blob. C# /// <summary> /// Fetch identifiers from Blob. /// </summary> public class BlobDataFetchActivity : WorkflowActivity<string, List<string>> { private readonly IBlobReadProcessor readProcessor; /// <summary> /// Initializes a new instance of the <see cref="BlobDataFetchActivity"/> class. /// </summary> /// <param name="blobReadProcessor">read blob data.</param> public BlobDataFetchActivity(IBlobReadProcessor blobReadProcessor) { this.readProcessor = blobReadProcessor; } /// <inheritdoc/> public override async Task<List<string>> RunAsync(WorkflowActivityContext context, string input) { return await this.readProcessor.ReadBlobContentAsync<List<string>>(input).ConfigureAwait(false); // state is saved } } Step 5: Scheduling the First Workflow Use the Workflow Client schedule workflows. The "instance id" must be unique to each workflow. Using the same ID can cause indeterministic behavior. Each workflow has an input and an output. For example, if the workflow is going to take a blob name as input and return a list of lines in the blob, the input is a string, and the output is a List<string>. Workflow is tracked using the workflow ID and once it is completed, the "Execute Workflow Async" method completes execution. C# public class DaprService { // Workflow client injected using Dependency Injection. private readonly DaprWorkflowClient daprWorkflowClient; /// <summary> /// Initializes a new instance of the <see cref="QueuedHostedService{T}"></see> class. /// </summary> /// <param name="daprWorkflowClient">Dapr workflow client.</param> public QueuedHostedService(DaprWorkflowClient daprWorkflowClient) { this.daprWorkflowClient = daprWorkflowClient; } /// <summary> /// Execute Dapr workflow. /// </summary> /// <param name="message">string Message.</param> /// <returns>Task.</returns> public async Task ExecuteWorkflowAsync(string message) { string id = Guid.NewGuid().ToString(); // Schedule the Dapr Workflow. await this.daprWorkflowClient.ScheduleNewWorkflowAsync( name: nameof(NetworkRecordIngestionWorkflow), instanceId: id, input: message).ConfigureAwait(false); WorkflowState state = await this.daprWorkflowClient.GetWorkflowStateAsync( instanceId: id, getInputsAndOutputs: true).ConfigureAwait(false); // Track the workflow state until completion. while (!state.IsWorkflowCompleted) { state = await this.daprWorkflowClient.GetWorkflowStateAsync( instanceId: id, getInputsAndOutputs: true).ConfigureAwait(false); } } } Best Practices Each time Dapr encounters an "await," it saves the workflow state. Leveraging this feature is important for ensuring workflows can resume efficiently and cost-effectively after interruptions. In addition to the above, the input and output must be deterministic for the Workflow replay pattern to work correctly. For example, Assume below is the first input to the workflow. The workflow then pulls the data from the blob, saves it to the state, and for some reason crashes. JSON { "blobName": "dapr-blob", "createdOn": "2024-12-11T23:00:00.11212Z" } After a restart, we resend the input with a different "created on" timestamp. Even though we’ve already saved the output for the blob name, the new timestamp qualifies this as a new payload, prompting the output to be recomputed. If the "created on" timestamp was omitted, we could retrieve the state from the state store without making an additional I/O call. JSON { "blobName": "dapr-blob", "createdOn": "2024-12-11T23:01:00.11212Z" } Workflow interaction with data other than the state must happen through Activities only.