DZone Spotlight

Are you a software developer or other tech professional? If you’re reading this, chances are pretty good that the answer is "yes." Long story short — we want DZone to work for you! We're asking that you take our annual community survey so we can better serve you! ^^ You can also enter the drawing for a chance to receive an exclusive DZone Swag Pack! The software development world moves fast, and we want to keep up! Across our community, we found that readers come to DZone for various reasons, including to learn about new development trends and technologies, find answers to help solve problems they have, connect with other peers, publish their content, and expand their personal brand's audience. In order to continue helping the DZone Community reach goals such as these, we need to know more about you, your learning preferences, and your overall experience on dzone.com and with the DZone team. For this year's DZone Community research, our primary goals are to: Learn about developer tech preferences and habits Identify content types and topics that developers want to get more information on Share this data for public consumption! To support our Community research, we're focusing on several primary areas in the survey: You, including your experience, the types of software you work on, and the tools you use How you prefer to learn and what you want to learn more about on dzone.com The ways in which you engage with DZone, your content likes vs. dislikes, and your overall journey on dzone.com As a community-driven site, our relationships with our members and contributors is invaluable, and we want to make sure that we continue to serve our audience to the best of our ability. If you're curious to see the report from the 2023 Community survey, feel free to check it out here! Thank you in advance for your participation!—Your favorite DZone Content and Community team

Kong Gateway is an open-source API gateway that ensures only the right requests get in while managing security, rate limiting, logging, and more. OPA (Open Policy Agent) is an open-source policy engine that takes control of your security and access decisions. Think of it as the mind that decouples policy enforcement from your app, so your services don’t need to stress about enforcing rules. Instead, OPA does the thinking with its Rego language, evaluating policies across APIs, microservices, or even Kubernetes. It’s flexible, and secure, and makes updating policies a breeze. OPA works by evaluating three key things: input (real-time data like requests), data (external info like user roles), and policy (the logic in Rego that decides whether to "allow" or "deny"). Together, these components allow OPA to keep your security game strong while keeping things simple and consistent. What Are We Seeking to Accomplish or Resolve? Oftentimes, the data in OPA is like a steady old friend — static or slowly changing. It’s used alongside the ever-changing input data to make smart decisions. But, imagine a system with a sprawling web of microservices, tons of users, and a massive database like PostgreSQL. This system handles a high volume of transactions every second and needs to keep up its speed and throughput without breaking a sweat. Fine-grained access control in such a system is tricky, but with OPA, you can offload the heavy lifting from your microservices and handle it at the gateway level. By teaming up with Kong API Gateway and OPA, you get both top-notch throughput and precise access control. How do you maintain accurate user data without slowing things down? Constantly hitting that PostgreSQL database to fetch millions of records is both expensive and slow. Achieving both accuracy and speed usually requires compromises between the two. Let’s aim to strike a practical balance by developing a custom plugin (at the gateway level) that frequently loads and locally caches data for OPA to use in evaluating its policies. Demo For the demo, I’ve set up sample data in PostgreSQL, containing user information such as name, email, and role. When a user tries to access a service via a specific URL, OPA evaluates whether the request is permitted. The Rego policy checks the request URL (resource), method, and the user’s role, then returns either true or false based on the rules. If true, the request is allowed to pass through; if false, access is denied. So far, it's a straightforward setup. Let’s dive into the custom plugin. For a clearer understanding of its implementation, please refer to the diagram below. When a request comes through the Kong Proxy, the Kong custom plugin would get triggered. The plugin would fetch the required data and pass it to OPA along with the input/query. This data fetch has two parts to it: one would be to look up Redis to find the required values, and if found, pass it along to OPA; if else, it would further query the Postgres and fetch the data and cache it in Redis before passing it along to OPA. We can revisit this when we run the commands in the next section and observe the logs. OPA makes a decision (based on the policy, input, and data) and if it's allowed, Kong will proceed to send that request to the API. Using this approach, the number of queries to Postgres is significantly reduced, yet the data available for OPA is fairly accurate while preserving the low latency. To start building a custom plugin, we need a handler.lua where the core logic of the plugin is implemented and a schema.lua which, as the name indicates, defines the schema for the plugin’s configuration. If you are starting to learn how to write custom plugins for Kong, please refer to this link for more info. The documentation also explains how to package and install the plugin. Let’s proceed and understand the logic of this plugin. The first step of the demo would be to install OPA, Kong, Postgres, and Redis on your local setup or any cloud setup. Please clone into this repository. Review the docker-compose yaml which has the configurations defined to deploy all four services above. Observe the Kong Env variables to see how the custom plugin is loaded. Run the below commands to deploy the services: Dockerfile docker-compose build docker-compose up Once we verify the containers are up and running, Kong manager and OPA are available on respective endpoints https://localhost:8002 and https://localhost:8181 as shown below: Create a test service, route and add our custom kong plugin to this route by using the below command: Shell curl -X POST http://localhost:8001/config -F config=@config.yaml The OPA policy, defined in authopa.rego file, is published and updated to the OPA service using the below command: Shell curl -X PUT http://localhost:8181/v1/policies/mypolicyId -H "Content-Type: application/json" --data-binary @authopa.rego This sample policy grants access to user requests only if the user is accessing the /demo path with a GET method and has the role of "Moderator". Additional rules can be added as needed to tailor access control based on different criteria. JSON opa_policy = [ { "path": "/demo", "method": "GET", "allowed_roles": ["Moderator"] } ] Now the setup is ready, but before testing, we need some test data to add in Postgres. I added some sample data (name, email, and role) for a few employees as shown below (please refer to the PostgresReadme). Here’s a sample failed and successful request: Now, to test the core functionality of this custom plugin, let’s make two consecutive requests and check the logs for how the data retrieval is happening. Here are the logs: JSON 2024/09/13 14:05:05 [error] 2535#0: *10309 [kong] redis.lua:19 [authopa] No data found in Redis for key: alice@example.com, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] handler.lua:25 [authopa] Fetching roles from PostgreSQL for email: alice@example.com, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] postgres.lua:43 [authopa] Fetched roles: Moderator, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] handler.lua:29 [authopa] Caching user roles in Redis, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] redis.lua:46 [authopa] Data successfully cached in Redis, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] opa.lua:37 [authopa] Is Allowed by OPA: true, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 client 192.168.96.1 closed keepalive connection ------------------------------------------------------------------------------------------------------------------------ 2024/09/13 14:05:07 [info] 2535#0: *10320 [kong] redis.lua:23 [authopa] Redis result: {"roles":["Moderator"],"email":"alice@example.com"}, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "75bf7a4dbe686d0f95e14621b89aba12" 2024/09/13 14:05:07 [info] 2535#0: *10320 [kong] opa.lua:37 [authopa] Is Allowed by OPA: true, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "75bf7a4dbe686d0f95e14621b89aba12" The logs show that for the first request when there’s no data in Redis, the data is being fetched from Postgres and cached in Redis before sending it forward to OPA for evaluation. In the subsequent request, since the data is available in Redis, the response would be much faster. Conclusion In conclusion, by combining Kong Gateway with OPA and implementing the custom plugin with Redis caching, we effectively balance accuracy and speed for access control in high-throughput environments. The plugin minimizes the number of costly Postgres queries by caching user roles in Redis after the initial query. On subsequent requests, the data is retrieved from Redis, significantly reducing latency while maintaining accurate and up-to-date user information for OPA policy evaluations. This approach ensures that fine-grained access control is handled efficiently at the gateway level without sacrificing performance or security, making it an ideal solution for scaling microservices while enforcing precise access policies.

In modern manufacturing, the key to sustaining product reliability, compliance with regulatory standards, and maintaining operational efficiency lies in robust quality control and advanced maintenance management. This study delves into the integration of AI technologies to interpret sensor data from production equipment. By employing predictive modeling techniques such as Stacked LSTM networks, RNNs, Random Forests, Gradient Boosting, SVM, ARIMA, and SARIMA, the goal is to forecast potential equipment failures, implement proactive maintenance solutions, and uphold rigorous quality checks through automated systems. This research showcases how AI-driven insights can be displayed through interactive dashboards and comprehensive reports, aiding in meeting regulatory standards while substantially improving operational dependability in manufacturing. Beyond the immediate benefits of reduced downtime and increased productivity, anomaly detection, especially in rotating machinery, plays a pivotal role in AI-driven predictive maintenance (PdM). The findings highlight how predictive modeling transforms equipment management, fostering innovation and supporting sustainability across various industries by utilizing sensors, cameras, and laser beam sensing technology. Introduction In the manufacturing sector, ensuring quality control and effective maintenance management are crucial to achieving product reliability, adhering to stringent standards, and driving operational efficiency. This paper explores how AI technologies leverage sensor data from industrial equipment, utilizing IoT devices, sensors, and laser beam technology, to monitor performance and predict potential failures. Through the strategic implementation of AI, these technologies facilitate proactive maintenance strategies and enforce rigorous quality control standards using automated inspections and audits. The insights generated from AI models are presented on intuitive dashboards and in detailed reports, which are essential for regulatory compliance. This project uses a variety of advanced predictive modeling methods, including Random Forests, Gradient Boosting, Support Vector Regression (SVR), and deep learning techniques such as Stacked LSTM networks and RNNs. These models, paired with feature engineering techniques like rolling statistics, provide deeper insights into equipment behavior dynamics and enhance traditional time series analyses with ARIMA and SARIMA models. By leveraging predictive analytics, maintenance schedules are optimized, leading to a significant reduction in downtime and mitigating the risk of unforeseen equipment failures. This approach not only yields substantial cost savings but also enhances the operational dependability of industrial systems. In industries where achieving high product reliability and maintaining operational efficiency is paramount, the adoption of AI technologies represents a game-changing development. Literature Review The integration of quality control and maintenance management is critical for ensuring product reliability and efficiency in manufacturing operations. Effective management of these areas not only improves product quality but also drives cost savings and enhances operational reliability. This section reviews the application of AI technologies in these domains, focusing on the use of sensor data from production equipment to enhance maintenance strategies. Machine learning (ML) has emerged as a powerful tool for boosting the accuracy and reliability of predictive maintenance systems. Various ML models, including XGBoost, Gaussian Process Regression, RNNs, SVMs, and Random Forests, have been successfully used to predict equipment efficiency and potential failures across the maintenance cycle. These AI-driven approaches have delivered considerable benefits, such as improved product quality, optimized workforce management, and heightened operational efficiency while ensuring compliance with industry regulations. Figure 1: Advanced Predictive Maintenance System Despite the clear benefits of machine learning (ML) in enhancing Preventive Maintenance (PdM) capabilities, integrating ML techniques into PdM is still in its early stages, which poses challenges. Therefore, there is a pressing need to thoroughly evaluate the effectiveness of these techniques and uncover their potential to further improve PdM functionalities. This paper aims to provide a detailed overview of the current landscape of PdM research, emphasizing major trends, obstacles, and prospective solutions within this area. By conducting this review, our goal is to pinpoint existing research gaps and offer valuable insights that can steer future studies and advancements in Predictive Maintenance Systems. Figure 1 Methodology 1. Types of Industrial Equipment Predictive maintenance is a proactive approach in industrial environments, aimed at forecasting maintenance needs based on the condition and type of equipment. The major categories of industrial machinery benefiting from predictive maintenance include: Heavy machinery and equipment Manufacturing machinery Power generation and distribution systems Material handling equipment Processing equipment Piping and fluid systems Packaging and labeling machines HVAC and environmental control systems Instrumentation and control systems Vehicles and fleet equipment These categories encompass a broad range of industrial applications where predictive maintenance can optimize performance, reduce downtime, and prevent expensive failures. Implementing AI-driven predictive maintenance across these sectors enhances reliability, efficiency, and cost-effectiveness 2. Challenges in Predictive Maintenance Design for Industrial Equipment Designing effective predictive maintenance (PdM) systems presents several critical challenges, including: Variability in equipment types and usage: Industrial operations involve a wide range of machinery, each with unique maintenance needs. Creating tailored maintenance plans is necessary to avoid the risks of over-maintenance or under-maintenance, which can affect operational efficiency. Data availability and reliability: Reliable and accurate data on equipment usage and condition are essential for the success of predictive maintenance. Inconsistent or incomplete data can undermine the accuracy of failure predictions and maintenance schedules. Integration of data from multiple sources: Industrial environments often feature equipment from various manufacturers, each with distinct data formats. The challenge lies in integrating data from these diverse systems into a unified maintenance management platform. Real-time monitoring and decision-making: Implementing real-time data collection and analysis is crucial for predicting equipment failures. The complexity of integrating real-time monitoring with predictive analytics presents a significant technical challenge. Cost of implementation: While the long-term benefits of predictive maintenance are evident, the initial investment required for AI technologies, sensors, and IoT infrastructure can be substantial. Companies must weigh these costs against the potential savings from reduced downtime and maintenance efficiency. 3. AI-Powered Advancements in Predictive Maintenance OR Enhanced Performance and Efficiency of Manufacturing Equipment The application of machine learning and Artificial Neural Networks (ANNs) has propelled Predictive Maintenance (PdM) forward by enhancing maintenance reliability and accuracy. Current research emphasizes the development of adaptive systems that draw from a wide spectrum of sensor data — including temperature, vibration, oil analysis, pressure, noise levels, voltage, and fluid levels — to generate insightful maintenance predictions. For example, in battery health evaluations, parameters such as charge cycles, voltage, and temperature are closely monitored to forecast the need for maintenance. Support Vector Machines (SVM) prove particularly effective in handling complex, high-dimensional data, enabling accurate predictions of equipment health and maintenance schedules. Deep learning models like Recurrent Neural Networks (RNNs) excel at modeling time-dependent behaviors, offering precise predictions of maintenance timelines. Furthermore, Ensemble methods that combine multiple machine learning techniques significantly boost predictive accuracy, ensuring dependable assessments of maintenance needs and schedules. 4. Exploring Diverse Machine Learning Techniques for Preventive Maintenance Effective Preventive Maintenance (PdM) relies on a wide variety of data to predict equipment life expectancy, necessary maintenance schedules, and Remaining Useful Life (RUL). The operational longevity of equipment is deeply affected by its usage and maintenance cycles, making dynamic cycling protocols essential for real-time maintenance predictions and simulating realistic usage scenarios. Key parameters like terminal voltage, pressure, temperature, and vibration are crucial inputs, represented as vector sequences, to fine-tune maintenance schedules and accurately predict RUL while accounting for fluctuations in operational cycles. Historical maintenance records, along with operational data, are fundamental to determining precise maintenance intervals. Machine learning algorithms, combined with experimental data, are employed to create highly reliable PdM models. Degradation indicators — such as rising temperatures, increased pressure, miscalibration, and vibration patterns — are used in current predictive models to detect early anomalies, enabling prompt and proactive maintenance actions. 5. Exploration of Diverse Machine Learning Techniques for Preventive Maintenance of Equipment A wide array of data is essential for accurately predicting equipment maintenance, estimating Remaining Useful Life (RUL), and establishing effective schedules for Preventive Maintenance (PdM). The longevity of machinery is closely tied to its operational cycles and maintenance routines, making dynamic cycling protocols critical for forecasting real-time maintenance requirements and simulating realistic operational scenarios. Parameters such as terminal voltage, pressure, temperature, vibration, and fluid levels — represented as a sequence of vectors—are integral inputs for fine-tuning maintenance schedules and projecting RUL, while accounting for fluctuations in usage patterns. Examining historical maintenance data and operational profiles is key to precisely determining maintenance intervals. By utilizing experimental data alongside machine learning techniques, industries can generate more dependable PdM predictions. Degradation factors such as elevated temperature, pressure variations, calibration discrepancies, and abnormal vibration levels are incorporated into modern machine-learning algorithms to detect early-stage anomalies. This enables timely, proactive maintenance interventions, ensuring that potential failures are identified before they escalate into costly breakdowns. Comprehensive Machine Learning Algorithms for Predictive Maintenance (PdM) Table 1 XGBoost: XGBoost is an ensemble learning algorithm renowned for its effectiveness in predicting maintenance requirements based on work schedules. Its key strengths include efficient second-order gradient descent optimization, integrated regularization, and the ability to handle sparse data effectively [5]. Gaussian Process Regression (GPR): Gaussian Process Regression (GPR) applies Bayesian, non-parametric methods to estimate maintenance needs by extracting significant features from equipment profiles. It builds a probabilistic model that offers predictions alongside confidence assessments, typically evaluated using metrics such as R2 and MAE [8]. Artificial Neural Networks (ANNs): Artificial Neural Networks (ANNs) are highly effective in predicting maintenance schedules and other equipment conditions for future maintenance. They excel at modeling complex, nonlinear relationships and are adept at handling noisy real-world data. The empirical equation for ANNs incorporates weights (Wij), input vectors (xj), and biases (bi) [12]. Support Vector Machine (SVM): SVM can predict maintenance schedules by extracting health performance features from equipment profiles. It excels in handling high-dimensional data and modeling complex, nonlinear relationships. The empirical equation for SVM includes weights (W), kernel functions (K), support vectors (xi, x), and biases (B) [8] Linear Regression (LR): LR is known for its simplicity and ease of interpretation when forecasting maintenance schedules using essential health performance features derived from equipment behavior. The straightforward equation incorporates coefficients (b0, b1, b2, ...) for input features (x1, x2, x3, ...) [8]. Random Forest (RF): RF is an ensemble learning method that effectively predicts equipment maintenance schedules by combining multiple decision trees. It excels in handling high-dimensional data, mitigates overfitting, and offers valuable insights into feature importance, which are crucial for Predictive Maintenance (PdM) [8]. Gradient Boosting Regressor: Gradient Boosting Regressor improves predictive accuracy by sequentially adding decision trees to rectify errors, ensuring robustness against overfitting. This technique is extensively employed to predict equipment health parameters in predictive maintenance (PdM). LSTM & Stacked LSTM: LSTM architectures are effective for time series forecasting, capturing sequential dependencies in equipment data for accurate predictions of maintenance & schedules [19][20]. ARIMA & SARIMA: ARIMA and SARIMA models are traditional time series methods used for predicting equipment parameters, handling non-seasonal and seasonal trends respectively, and contributing to PdM predictive performance. Recurrent Neural Networks (RNNs): RNNs are ideal for sequential data analysis, making them suitable for predicting maintenance schedules based on time-dependent equipment behavior. Comparison With Baseline Models: Table 2 Baseline Models Evaluated Persistence Model (Last Value): Description: This model predicts the maintenance for the next time step using the last observed operational and anomaly trends. Stacked LSTM Models vs. Baseline Models: Observation: Stacked LSTM models significantly outperformed the baseline models, especially in scenarios where maintenance dynamics exhibit complex sequential dependencies. PERFORMANCE COMPARISON Linear Regression vs. Baseline Models: Observation: Linear Regression, while more sophisticated than the Persistence and Moving Average models, did not consistently outperform them across all scenarios. Random Forests and Gradient Boosting vs. Baseline Models: Observation: Both Random Forests and Gradient Boosting consistently outperformed the baseline models across various maintenance prediction scenarios. Quality Control AI technologies are reshaping quality control in manufacturing by automating inspection processes and identifying defects with unmatched precision. Machine learning algorithms analyze various data inputs, such as images, sensor readings, and production metrics, to detect potential defects in real-time, ensuring that only high-quality products reach the market. These AI-powered systems not only improve product consistency but also reduce waste and enhance customer satisfaction, delivering significant improvements across the production chain. Predictive Maintenance Predictive maintenance is a cornerstone of AI applications in manufacturing, leveraging machine learning models to predict equipment failures before they occur. By monitoring equipment performance in real-time and analyzing historical data, predictive maintenance systems identify early warning signs of potential issues, enabling proactive repairs and minimizing unplanned downtime. AI-driven predictive maintenance improves equipment reliability, extends asset lifespan, and optimizes maintenance schedules to reduce costs and enhance operational efficiency. Operational Efficiency AI technologies are optimizing operational efficiency in manufacturing by analyzing production workflows, pinpointing inefficiencies, and improving resource allocation. Machine learning algorithms streamline production schedules, predict demand variations, and optimize supply chain management, ensuring timely deliveries while minimizing inventory costs. These AI-driven enhancements in operational efficiency lead to higher productivity, shorter lead times, and greater flexibility in adapting to changing market demands. Materials and Methods Data Collection For this study, we utilized datasets from multiple pieces of operational equipment. The first dataset comprises 684,233 rows, the second contains 483,450 rows, and the third holds 960,033 rows. Each dataset includes critical parameters such as temperature, vibration, pressure, noise levels, voltage, fluid levels, potential damages, and calibration details. These datasets form the foundation of our research, providing a rich array of data points directly sourced from operational equipment within the manufacturing environment. By incorporating key metrics like temperature, vibration, and pressure, among others, we conducted an in-depth analysis of equipment performance under varied conditions. These datasets are essential to our modeling efforts, advancing our understanding of equipment behavior and improving predictive capabilities related to maintenance requirements and scheduling. Data Preparation and Exploration In preparation for the project, data from multiple manufacturers was carefully curated and integrated using Azure Cloud's streaming services. The process began with comprehensive data cleaning, addressing inconsistencies, and handling missing values. We applied mean imputation for numerical variables and mode imputation for categorical variables due to the small percentage of missing data and its normal distribution. Regression imputation was avoided to prevent overfitting with the dataset size. Outliers were identified using the Z-score method with a threshold of 3, capping extreme values at the 95th percentile to mitigate their impact on model performance while preserving valuable data insights. To further enhance dataset reliability, cubic interpolation was used to smooth the time series data, reducing noise in time-sensitive predictive modeling. After comparing cubic interpolation with spline methods, cubic interpolation proved more stable, avoiding unnecessary oscillations. The integration of all datasets resulted in a unified dataset that reflects diverse operational scenarios. Feature engineering included calculating rolling averages over a 7-day window to capture short-term trends, which significantly improved the model’s ability to predict equipment failures during regular operational cycles. Additional steps involved normalizing and scaling data, addressing missing values and outliers using Databricks, and partitioning the data into training and testing sets for machine learning analysis. These thorough preparations laid the groundwork for generating accurate predictions related to equipment operational parameters, ensuring reliable insights for predictive maintenance. Data Visualization Figure 2 Figure 2 illustrates Time Series, Predictive Performance, Anomaly Detection, Failure Probability Curve, Equipment Health Index, and Trend Analysis graphs of the equipment operational trends. Observations Time Series Plot: Predictive Model Performance: A time series plot shows how a particular variable (such as temperature, vibration, or pressure) changes over time which describes the right-skewed distribution. Overall it helps in identifying trends, seasonality & anomalies that may indicate impending equipment issues. Graphs show the performance of predictive models, such as actual vs. predicted values/the model's accuracy over time, which covers the heatmap of missing values. The pattern was addressed through normalization to ensure accurate modeling. The pattern guides us through cubic interpolation for imputation that improves the data enrichment. This helps in assessing how well the model is predicting maintenance needs. Anomaly Detection: Failure Probability Curve: Anomaly detection plots highlight deviations from normal operating conditions. These anomalies can indicate potential faults or failures in equipment. A failure probability curve estimates the likelihood of equipment failure over time based on predictive models and historical data. It helps in scheduling maintenance proactively. Equipment Health Index: Trend Analysis: An equipment health index graphically represents the overall condition of equipment based on various sensor data and predictive analytics. It provides a quick snapshot of equipment status, detects anomalies over time, and plots the same to understand the critical points that cross the thresholds to call for maintenance Trend analysis graphs show long-term patterns or changes in equipment performance metrics. They help in identifying degradation of failure probabilities that forecast the rising risks and certify the need for required maintenance intervention. Results and Discussion Regression Models Linear Regression: Achieves moderate accuracy with an RMSE of 0.1496, serving as a baseline performance Support Vector Regression (SVR): Shows moderate accuracy with an RMSE of 0.1282, indicating potential for enhancement Tree-Based Models Decision Trees: Exhibits exceptional accuracy with an RMSE of 2.25e-05, adept at capturing subtle health patterns Random Forests: Demonstrates robust performance with an RMSE of 1.74e-05, leveraging its ensemble approach Ensemble Methods Gradient Boosting: Maintains competitive accuracy with an RMSE of 3.73e-05, effective in modeling complex relationships XGB Regressor: Provides consistent precise predictions across evaluations Time Series Models Stacked LSTM models: Excels in capturing time-dependent nuances with an RMSE of 0.000405 ARIMA and SARIMA: Show notable performance with varying RMSE values, contributing to accurate maintenance predictions Classification Model Random Forest Classifier: Achieves impeccable accuracy of 1.0 in predicting maintenance needs, highlighting its reliability. Insights The variety of models applied in this analysis highlights their respective strengths in predicting equipment maintenance needs. While Linear Regression and SVR demonstrate moderate accuracy, more advanced methods such as Decision Trees and Random Forests stand out for their ability to capture complex health patterns with minimal error. Gradient Boosting and XGBoost Regressor excel at modeling intricate relationships, resulting in highly precise predictions. Stacked LSTM models prove particularly effective in managing time-dependent data, which is essential for anticipating maintenance requirements in dynamic operational settings. Additionally, ARIMA and SARIMA models provide significant contributions, illustrating their versatility in different maintenance scenarios. The impeccable accuracy of the Random Forest Classifier reinforces its reliability in predictive maintenance applications, delivering actionable insights for implementing preemptive strategies. These findings emphasize the importance of aligning model selection with the unique characteristics of the data, thereby improving equipment health predictions and optimizing maintenance schedules beyond standard timelines. Conclusion The evaluation of various predictive models for equipment maintenance reveals a diverse spectrum of performance metrics, each offering distinct advantages in enhancing operational efficiency and equipment reliability. Tree-based approaches like Decision Trees and Random Forests exhibit outstanding accuracy, adeptly capturing subtle health indicators with minimal error (RMSE: 1.74e-05 to 2.25e-05). Similarly, ensemble methods such as Gradient Boosting and XGBoost Regressor excel at modeling complex relationships, producing precise maintenance predictions. In contrast, simpler models such as Linear Regression and Support Vector Regression (SVR) demonstrate moderate accuracy, underscoring the importance of selecting models that align with the specific characteristics of the maintenance data. Time-series models like Stacked LSTM excel in detecting temporal patterns, offering critical insights into time-sensitive maintenance requirements. The flawless performance of the Random Forest Classifier in predicting maintenance needs reinforces its dependability in real-world applications. Furthermore, the robust capabilities of ARIMA and SARIMA models in time-series forecasting confirm their effectiveness in forecasting maintenance trends. In conclusion, the varying performance across these models underscores the need for continuous refinement and customization to optimize predictions and maintenance strategies. Leveraging advanced analytics enables organizations to manage maintenance proactively, reduce downtime, and allocate resources more effectively in industrial operations. Future Research Directions 1. Integration of Multimodal Data Future research should explore the integration of multimodal data — such as vibration signals, acoustic data, operational logs, and environmental conditions — to improve predictive accuracy and gain a more comprehensive understanding of equipment health. Developing advanced data fusion techniques and models capable of handling diverse data types will be critical. 2. Explainability and Interpretability of Models Enhancing the transparency of predictive maintenance models is essential. Research should focus on building frameworks that offer clear explanations of model predictions, making them accessible to engineers and decision-makers. This requires incorporating explainability into the model design from the outset. 3. Adaptation To Evolving Systems Research should explore adaptive learning and incremental learning approaches that enable models to evolve and improve as new data becomes available. Future efforts could also focus on developing models that detect and adjust to shifts in system dynamics, ensuring long-term accuracy. 4. Scalability and Real-Time Processing Investigating scalable algorithms and architectures capable of handling large-scale data streams in real-time is an important area for future research. This includes exploring edge computing solutions that facilitate on-site data processing and enable timely decision-making. 5. Integration With Human Expertise Future research should focus on systems that integrate AI-driven insights with human expertise to enhance decision-making processes. This involves designing user-friendly interfaces that promote collaboration between AI systems and maintenance engineers while creating feedback loops where human input can refine and improve AI models.

This post is part of a series comparing different ways to implement asynchronous requests on the client, which is colloquially known as AJAX. I dedicated the previous post to Vue.js. I'll dedicate this one to Alpine.js — not to be confused with Alpine Linux. I'll follow the same structure as previously. Laying out the Work Here's the setup, server- and client-side. Server-Side Here is how I integrate Thymeleaf and Alpine.js in the POM: XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <!--1--> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-thymeleaf</artifactId> <!--1--> </dependency> <dependency> <groupId>org.webjars</groupId> <artifactId>webjars-locator</artifactId> <!--1--> <version>0.52</version> </dependency> <dependency> <groupId>org.webjars.npm</groupId> <artifactId>alpinejs</artifactId> <!--2--> <version>3.14.1</version> </dependency> <dependency> <groupId>org.webjars.npm</groupId> <artifactId>axios</artifactId> <!--1--> <version>1.7.3</version> </dependency> </dependencies> Same as last week with Vue Alpine instead of Vue It's similar to Vue's setup. Client-Side Here's the code on the HTML side: HTML <script th:src="@{/webjars/axios/dist/axios.js}" src="https://cdn.jsdelivr.net/npm/axios@1.7/dist/axios.min.js"></script> <!--1--> <script th:src="@{/webjars/alpinejs/dist/cdn.js}" src="https://cdn.jsdelivr.net/npm/alpinejs@3.14.1/dist/cdn.min.js" defer></script> <!--2--> <script th:src="@{/alpine.js}" src="../static/alpine.js"></script> <!--3--> <script th:inline="javascript"> /*<![CDATA[*/ window.alpineData = { <!--4--> title: /*[[${ title }]]*/ 'A Title', todos: /*[[${ todos }]]*/ [{ 'id': 1, 'label': 'Take out the trash', 'completed': false }] } /*]]>*/ </script> Axios helps make HTTP requests. Alpine itself Our client-side code Set the data As for the POM, it's the same code for Alpine as for Vue. The Alpine Code We want to implement the same features as for Vue. Our First Steps Into Alpine The first step is to bootstrap the framework. We already added the link to our custom alpine.js file above. JavaScript document.addEventListener('alpine:init', () => { //1 Alpine.data('app', () => ({ //2 // The next JavaScript code snippets will be inside the block })) }) Run the block when the alpine:init event is triggered; the triggering event is specific to Alpine. Bootstrap Alpine and configure it to manage the HTML fragment identified by app. We now set the app id on the HTML side. HTML <div id="app"> </div> Until now, it's very similar to Vue.js, a straight one-to-one mapping. Unlike Vue.js, Alpine doesn't seem to have templates. The official UI components are not free. I found an open-source approach, but it's unavailable on WebJars. Basic Interactions Let's implement the check of the complete checkbox. Here's the HTML code: HTML <input type="checkbox" :checked="todo.completed" @click="check(todo.id)"> <!--1--> <input type="checkbox" :checked="todo.completed" @click="check" /> <!--2--> Alpine code Vue code The code is very similar, with the difference that Alpine allows passing parameters. On the JavaScript side, we must define the function, and that's all: JavaScript Alpine.data('app', () => ({ check(id) { axios.patch(`/api/todo/${id}`, {checked: event.target.checked}) } })) Client-Side Model You might wonder where the todo above comes from. The answer is: from the local model. We initialize it in the app or to be more precise, we initialize the list: JavaScript Alpine.data('app', () => ({ title: window.alpineData.title, //1 todos: window.alpineData.todos, //2 })) Initialize the title even if it's read-only. Initialize the todos list; at this point, it's read-only but we are going to update it the next section. Updating the Model In this section, we will implement adding a new Todo. Here's the HTML snippet: HTML <form> <div class="form-group row"> <label for="new-todo-label" class="col-auto col-form-label">New task</label> <div class="col-10"> <input type="text" id="new-todo-label" placeholder="Label" class="form-control" x-model="label" /> <!--1--> </div> <div class="col-auto"> <button type="button" class="btn btn-success" @click="create()">Add</button> <!--2--> </div> </div> </form> The x-model defines a model and binds the label property defined in app. Define the behavior of the button, as in the previous section. The related code is the following: JavaScript Alpine.data('app', () => ({ label: '', //1 create() { axios.post('/api/todo', {label: this.label}).then(response => { //2 this.todos.push(response.data) //3 }).then(() => { this.label = '' //4 }) } })) Define a new label property. Send a POST request with the label value as the JSON payload. Get the response payload and add it to the local model of Todo. Reset the label value. Conclusion Alpine is very similar to Vue, with the notable difference of the lack of templating; components are only available via a price. All other features have an equivalent. I may need to be corrected because the documentation is less extensive. Also, Vue is much more popular than Alpine. The complete source code for this post can be found on GitHub.

By Nicolas Fränkel CORE

In most front-end applications, fetching data from the back-end is a common requirement to provide up-to-date information to the user. Typically, this process is straightforward: know the API endpoint, create a function, call it, process the response, and display it on the page. However, there are scenarios — often rare or specific to certain business needs — where client-server communication must be more sophisticated. In this article, we’ll explore one such case: implementing repetitive API calls (polling) using RxJS. RxJS Refresher Before diving into polling, let’s take a moment to refresh our understanding of RxJS. For those new to it, RxJS (Reactive Extensions for JavaScript) is a library that brings reactive programming concepts into the JavaScript ecosystem. It’s particularly useful for managing asynchronous operations, offering powerful tools like Observables, Observers, Subjects, and Operators. In short, RxJS is perfect for situations where you need to handle complex asynchronous tasks. Polling is one such scenario, making RxJS an ideal candidate for implementing this kind of functionality. Repetitive API Calls What is the idea of repetitive API calls? Well, there are times when a simple one-time API call won’t cut it. For example, waiting for an online queue (e.g., buying concert tickets where the queue updates frequently), checking order status changes at regular intervals without reloading the page, etc. While you could use setInterval to achieve this, it’s not a reliable solution. It can lead to issues like infinite API calls, excessive server load, or unpredictable behavior in your application. Instead, we can use RxJS to implement a more robust, controlled polling mechanism. Code Breakdown Let’s break down the code that does this: TypeScript import { MonoTypeOperatorFunction, timer } from 'rxjs'; import { last, scan, switchMapTo, takeWhile, tap } from 'rxjs/operators'; function attemptsGuardFactory(maxAttempts: number) { return (attemptsCount: number) => { if (attemptsCount > maxAttempts) { throw new Error('Exceeded maxAttempts'); } }; } export function pollWhile<T>( pollInterval: number, isPollingActive: (res: T) => boolean, maxAttempts: number = Infinity, emitOnlyLast: boolean = false, ): MonoTypeOperatorFunction<T> { return (source$) => { const poll$ = timer(0, pollInterval).pipe( scan((attempts) => { attempts += 1; return attempts; }, 0), tap(attemptsGuardFactory(maxAttempts)), switchMapTo(source$), takeWhile(isPollingActive, true), ); return emitOnlyLast ? poll$.pipe(last()) : poll$; }; } Here we can see two functions: attemptsGuardFactory and pollWhile. The first one is a helper function, and the second one is the actual implementation of polling. The pollWhile function returns an RxJS operator and accepts some parameters to modify your polling settings: pollInterval: The interval (in milliseconds) at which the polling will occur isPollingActive: A function that determines if polling should continue maxAttempts: Limits the maximum number of polling attempts emitOnlyLast: Should the function emit a value on each "tick"? If true, only the last value will be emitted. We use the timer(0, pollInterval) function to create an observable that emits at the specified interval. The first value is emitted immediately, and subsequent values are emitted after each pollInterval. The scan operator acts similarly to reduce, accumulating the number of polling attempts. Unlike reduce, though, scan emits intermediate values, which is useful in our case as we want to track polling attempts over time. Next, the tap operator allows us to perform side effects — in this case, we use it to check the number of attempts via attemptsGuardFactory. If we’ve reached the maxAttempts, we throw an error to stop further polling. The switchMapTo operator is key here — it subscribes to the source$ observable and cancels the previous subscription if a new one starts. This ensures that if the polling interval triggers again before the previous request completes, the earlier request is canceled, preventing overlapping calls. The switchMapTo is similar to the switchMap, but it takes an observable instead of a callback. Finally, takeWhile ensures polling continues only while isPollingActive returns true. Depending on the emitOnlyLast flag, the function either returns only the last emitted value or all values. Here’s a simple example of how you can use this polling mechanism in your code: TypeScript // Simulate a mock API call that returns the process status function mockApiCall() { const statuses = ['pending', 'pending', 'pending', 'complete']; let index = 0; return of(null).pipe( // Simulating 1 second delay for the API response delay(1000), map(() => { const status = statuses[index]; index++; return status; }) ); } const isPollingActive = (status: string) => status !== 'complete'; mockApiCall().pipe( pollWhile<string>( 2000, isPollingActive, 5, true ) ).subscribe(() => { // This block will be executed when the status is "complete" or the maximum polling attempts are reached }); It's as simple as that! We can easily use this pipe for every observable we want. But let’s take this a step further. In some cases, like when a server is overloaded, we may want to introduce a delay that grows over time to ease the load. This technique, known as exponential backoff, can be easily added to our RxJS operator. Here’s the modified version of pollWhile: TypeScript import { MonoTypeOperatorFunction, timer } from 'rxjs'; import { expand, last, scan, switchMapTo, takeWhile, tap } from 'rxjs/operators'; export function pollWhile<T>( pollInterval: number, growthFactor: number, isPollingActive: (res: T) => boolean, maxAttempts: number = Infinity, emitOnlyLast: boolean = false, ): MonoTypeOperatorFunction<T> { return (source$) => { const poll$ = timer(0).pipe( scan((attempts) => attempts + 1, 0), tap(attemptsGuardFactory(maxAttempts)), expand((attempts) => timer(pollInterval * Math.pow(growthFactor, attempts))), switchMapTo(source$), takeWhile(isPollingActive, true), ); return emitOnlyLast ? poll$.pipe(last()) : poll$; }; } In this version, we introduce a new parameter, growthFactor, which controls how much the delay grows after each polling attempt. We use the expand operator to multiply the delay by the growth factor, increasing the wait time after each attempt. This is particularly useful in scenarios where you want to avoid overloading a server with too many requests in quick succession. Conclusion As you can see, polling with RxJS offers a robust and flexible way to handle repetitive API calls. We can build a highly customizable polling mechanism that ensures our application remains performant and efficient. In addition to this, we can easily implement the exponential backoff reducing server load in situations where response times are inconsistent or where the server might be overwhelmed. Whether you're dealing with long-running processes, status updates, or any other use case that requires periodic server checks, RxJS offers a powerful toolset for implementing polling. Give it a try in your next project, and take full advantage of reactive programming capabilities!

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. In the past, before CI/CD and Kubernetes came along, deploying software to Kubernetes was a real headache. Developers would build stuff on their own machines, then package it and pass it to the operations team to deploy it on production. This approach would frequently lead to delays, miscommunications, and inconsistencies between environments. Operations teams had to set up the deployments themselves, which increased the risk of human errors and configuration issues. When things went wrong, rollbacks were time consuming and disruptive. Also, without automated feedback and central monitoring, it was tough to keep an eye on how builds and deployments were progressing or to identify production issues. With the advent of CI/CD pipelines combined with Kubernetes, deploying software is smoother. Developers can simply push their code, which triggers builds, tests, and deployments. This enables organizations to ship new features and updates more frequently and reduce the risk of errors in production. This article explains the CI/CD transformation with Kubernetes and provides a step-by-step guide to building a pipeline. Why CI/CD Should Be Combined With Kubernetes CI/CD paired with Kubernetes is a powerful combination that makes the whole software development process smoother. Kubernetes, also known as K8s, is an open-source system for automating the deployment, scaling, and management of containerized applications. CI/CD pipelines, on the other hand, automate how we build, test, and roll out software. When you put them together, you can deploy more often and faster, boost software quality with automatic tests and checks, cut down on the chance of pushing out buggy code, and get more done by automating tasks that used to be done by hand. CI/CD with Kubernetes helps developers and operations teams work better together by giving them a shared space to do their jobs. This teamwork lets companies deliver high-quality applications rapidly and reliably, gaining an edge in today's fast-paced world. Figure 1 lays out the various steps: Figure 1. Push-based CI/CD pipeline with Kubernetes and monitoring tools There are several benefits in using CI/CD with Kubernetes, including: Faster and more frequent application deployments, which help in rolling out new features or critical bug fixes to the users Improved quality by automating testing and incorporating quality checks, which helps in reducing the number of bugs in your applications Reduced risk of deploying broken code to production since CI/CD pipelines can conduct automated tests and roll-back deployments if any problems exist Increased productivity by automating manual tasks, which can free developers' time to focus on important projects Improved collaboration between development and operations teams since CI/CD pipelines provide a shared platform for both teams to work Tech Stack Options There are different options available if you are considering building a CI/CD pipeline with Kubernetes. Some of the popular ones include: Open-source tools such as Jenkins, Argo CD, Tekton, Spinnaker, or GitHub Actions Enterprise tools, including but not limited to, Azure DevOps, GitLab CI/CD, or AWS CodePipeline Deciding whether to choose an open-source or enterprise platform to build efficient and reliable CI/CD pipelines with Kubernetes will depend on your project requirements, team capabilities, and budget. Impact of Platform Engineering on CI/CD With Kubernetes Platform engineering builds and maintains the underlying infrastructure and tools (the "platform") that development teams use to create and deploy applications. When it comes to CI/CD with Kubernetes, platform engineering has a big impact on making the development process better. It does so by hiding the complex parts of the underlying infrastructure and giving developers self-service options. Platform engineers manage and curate tools and technologies that work well with Kubernetes to create a smooth development workflow. They create and maintain CI/CD templates that developers can reuse, allowing them to set up pipelines without thinking about the details of the infrastructure. They also set up rules and best practices for containerization, deployment strategies, and security measures, which help maintain consistency and reliability across different applications. What's more, platform engineers provide ways to observe and monitor applications running in Kubernetes, which let developers find and fix problems and make improvements based on data. By building a strong platform, platform engineering helps dev teams zero in on creating and rolling out features more without getting bogged down by the complexities of the underlying tech. It brings together developers, operations, and security teams, which leads to better teamwork and faster progress in how things are built. How to Build a CI/CD Pipeline With Kubernetes Regardless of the tech stack you select, you will often find similar workflow patterns and steps. In this section, I will focus on building a CI/CD pipeline with Kubernetes using GitHub Actions. Step 1: Setup and prerequisites GitHub account – needed to host your code and manage the CI/CD pipeline using GitHub Actions Kubernetes cluster – create one locally (e.g., MiniKube) or use a managed service from Amazon or Azure kubectl – Kubernetes command line tool to connect to your cluster Container registry – needed for storing Docker images; you can either use a cloud provider's registry (e.g., Amazon ECR, Azure Container Registry, Google Artifact Registry) or set up your own private registry Node.js and npm – install Node.js and npm to run the sample Node.js web application Visual Studio/Visual Studio Code – IDE platform for making code changes and submitting them to a GitHub repository Step 2: Create a Node.js web application Using Visual Studio, create a simple Node.js application with a default template. If you look inside, the server.js in-built generated file will look like this: Shell // server.js 'use strict'; var http = require('http'); var port = process.env.PORT || 1337; http.createServer(function (req, res) { res.writeHead(200, { 'Content-Type': 'text/plain' }); res.end('Hello from kubernetes\n'); }).listen(port); Step 3: Create a package.json file to manage dependencies Inside the project, add a new file Package.json to manage dependencies: Shell // Package.Json { "name": "nodejs-web-app1", "version": "0.0.0", "description": "NodejsWebApp", "main": "server.js", "author": { "name": "Sunny" }, "scripts": { "start": "node server.js", "test": "echo \"Running tests...\" && exit 0" }, "devDependencies": { "eslint": "^8.21.0" }, "eslintConfig": { } } Step 4: Build a container image Create a Dockerfile to define how to build your application's Docker image: Shell // Dockerfile # Use the official Node.js image from the Docker Hub FROM node:14 # Create and change to the app directory WORKDIR /usr/src/app # Copy package.json and package-lock.json COPY package*.json ./ # Install dependencies RUN npm install # Copy the rest of the application code COPY . . # Expose the port the app runs on EXPOSE 3000 # Command to run the application CMD ["node", "app.js"] Step 5: Create a Kubernetes Deployment manifest Create a deployment.yaml file to define how your application will be deployed in Kubernetes: Shell // deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nodejs-deployment spec: replicas: 3 selector: matchLabels: app: nodejs-app template: metadata: labels: app: nodejs-app spec: containers: - name: nodejs-container image: nodejswebapp ports: - containerPort: 3000 env: - name: NODE_ENV value: "production" --- apiVersion: v1 kind: Service metadata: name: nodejs-service spec: selector: app: nodejs-app ports: - protocol: TCP port: 80 targetPort: 3000 type: LoadBalancer Step 6: Push code to GitHub Create a new code repository on GitHub, initialize the repository, commit your code changes, and push it to your GitHub repository: Shell git init git add . git commit -m "Initial commit" git remote add origin "<remote git repo url>" git push -u origin main Step 7: Create a GitHub Actions workflow Inside your GitHub repository, go to the Actions tab. Create a new workflow (e.g., main.yml) in the .github/workflows directory. Inside the GitHub repository settings, create Secrets under actions related to Docker and Kubernetes cluster — these are used in your workflow to authenticate: Shell //main.yml name: CI/CD Pipeline on: push: branches: - main jobs: build: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Node.js uses: actions/setup-node@v2 with: node-version: '14' - name: Install dependencies run: npm install - name: Run tests run: npm test - name: Build Docker image run: docker build -t <your-docker-image> . - name: Log in to Docker Hub uses: docker/login-action@v1 with: username: ${{ secrets.DOCKER_USERNAME } password: ${{ secrets.DOCKER_PASSWORD } - name: Build and push Docker image uses: docker/build-push-action@v2 with: context: . push: true tags: <your-docker-image-tag> deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up kubectl uses: azure/setup-kubectl@v1 with: version: 'latest' - name: Set up Kubeconfig run: echo "${{ secrets.KUBECONFIG }" > $HOME/.kube/config - name: Deploy to Kubernetes run: kubectl apply -f deployment.yaml Step 8: Trigger the pipeline and monitor Modify server.js and push it to the main branch; this triggers the GitHub Actions workflow. Monitor the workflow progress. It installs the dependencies, sets up npm, builds the Docker image and pushes it to the container registry, and deploys the application to Kubernetes. Once the workflow is completed successfully, you can access your application that is running inside the Kubernetes cluster. You can leverage open-source monitoring tools like Prometheus and Grafana for metrics. Deployment Considerations There are a few deployment considerations to keep in mind when developing CI/CD pipelines with Kubernetes to maintain security and make the best use of resources: Scalability Use horizontal pod autoscaling to scale your application's Pods based on how much CPU, memory, or custom metrics are needed. This helps your application work well under varying loads. When using a cloud-based Kubernetes cluster, use the cluster autoscaler to change the number of worker nodes as needed to ensure enough resources are available and no money is wasted on idle resources. Ensure your CI/CD pipeline incorporates pipeline scalability, allowing it to handle varying workloads as per your project needs. Security Scan container images regularly to find security issues. Add tools for image scanning into your CI/CD pipeline to stop deploying insecure code. Implement network policies to limit how Pods and services talk to each other inside a Kubernetes cluster. This cuts down on ways attackers could get in. Set up secrets management using Kubernetes Secrets or external key vaults to secure and manage sensitive info such as API keys and passwords. Use role-based access control to control access to Kubernetes resources and CI/CD pipelines. High availability Through multi-AZ or multi-region deployments, you can set up your Kubernetes cluster in different availability zones or regions to keep it running during outages. Pod disruption budgets help you control how many Pods can be down during planned disruptions (like fixing nodes) or unplanned ones (like when nodes fail). Implement health checks to monitor the health of your pods and automatically restart if any fail to maintain availability. Secrets management Store API keys, certificates, and passwords as Kubernetes Secrets, which are encrypted and added to Pods. You can also consider external secrets management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault if you need dynamic secret generation and auditing. Conclusion Leveraging CI/CD pipelines with Kubernetes has become a must-have approach in today's software development. It revolutionizes the way teams build, test, and deploy apps, leading to more efficiency and reliability. By using automation, teamwork, and the strength of container management, CI/CD with Kubernetes empowers organizations to deliver high-quality software at speed. The growing role of AI and ML will likely have an impact on CI/CD pipelines — such as smarter testing, automated code reviews, and predictive analysis to further enhance the development process. When teams adopt best practices, keep improving their pipelines, and are attentive to new trends, they can get the most out of CI/CD with Kubernetes, thus driving innovation and success. This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report

By Naga Santhosh Reddy Vootukuri CORE

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. A decade ago, Google introduced Kubernetes to simplify the management of containerized applications. Since then, it has fundamentally transformed the software development and operations landscape. Today, Kubernetes has seen numerous enhancements and integrations, becoming the de facto standard for container orchestration. This article explores the journey of Kubernetes over the past 10 years, its impact on the software development lifecycle (SDLC) and developers, and the trends and innovations that will shape its next decade. The Evolution of Kubernetes Kubernetes, often referred to as K8s, had its first commit pushed to GitHub on June 6, 2014. About a year later, on July 21, 2015, Kubernetes V1 was released, featuring 14,000 commits from 400 contributors. Simultaneously, the Linux Foundation announced the formation of the Cloud Native Computing Foundation (CNCF) to advance state-of-the-art technologies for building cloud-native applications and services. After that, Google donated Kubernetes to the CNCF, marking a significant milestone in its development. Kubernetes addressed a critical need in the software industry: managing the lifecycle of containerized applications. Before Kubernetes, developers struggled with orchestrating containers, leading to inefficiencies and complexities in deployment processes. Kubernetes brought advanced container management functionality and quickly gained popularity due to its robust capabilities in automating the deployment, scaling, and operations of containers. While early versions of Kubernetes introduced the foundation for container orchestration, the project has since undergone significant improvements. Major updates have introduced sophisticated features such as StatefulSets for managing stateful applications, advanced networking capabilities, and enhanced security measures. The introduction of Custom Resource Definitions (CRDs) and Operators has further extended its functionality, allowing users to manage complex applications and workflows with greater ease. In addition, the community has grown significantly over the past decade. According to the 2023 Project Journey Report, Kubernetes now has over 74,680 contributors, making it the second-largest open-source project in the world after Linux. Over the years, Kubernetes has seen numerous enhancements and integrations, becoming the de facto standard for container orchestration. The active open source community and the extensive ecosystem of tools and projects have made Kubernetes an essential technology for modern software development. It is now the "primary container orchestration tool for 71% of Fortune 100 companies" (Project Journey Report). Kubernetes' Impact on the SDLC and Developers Kubernetes abstracts away the complexities of container orchestration and allows developers to focus on development rather than worry about application deployment and orchestration. The benefits and key impacts on the SDLC and developer workflows include enhanced development and testing, efficient deployment, operational efficiency, improved security, and support for microservices architecture. Enhanced Development and Testing Kubernetes ensures consistency for applications running across testing, development, and production environments, regardless of whether the infrastructure is on-premises, cloud based, or a hybrid setup. This level of consistency, along with the capability to quickly spin up and tear down environments, significantly accelerates development cycles. By promoting portability, Kubernetes also helps enterprises avoid vendor lock-in and refine their cloud strategies, leading to a more flexible and efficient development process. Efficient Deployment Kubernetes automates numerous aspects of application deployment, such as service discovery, load balancing, scaling, and self-healing. This automation reduces manual effort, minimizes human error, and ensures reliable and repeatable deployments, reducing downtime and deployment failures. Operational Efficiency Kubernetes efficiently manages resources by dynamically allocating them based on the application's needs. It ensures operations remain cost effective while maintaining optimal performance and use of computing resources by scheduling containers based on resource requirements and availability. Security Kubernetes enhances security by providing container isolation and managing permissions. Its built-in security features allow developers to build secure applications without deep security expertise. Such built-in features include role-based access control, which ensures that only authorized users can access specific resources and perform certain actions. It also supports secrets management to securely store and manage sensitive information like passwords and API keys. Microservices Architecture Kubernetes has facilitated the adoption of microservices architecture by enabling developers to deploy, manage, and scale individual microservices independently. Each microservice can be packaged into a separate container, providing isolation and ensuring that dependencies are managed within the container. Kubernetes' service discovery and load balancing features enable communication between microservices, while its support for automated scaling and self-healing ensures high availability and resilience. Predictions for the Next Decade After a decade, it has become clear that Kubernetes is now the standard technology for container orchestration that's used by many enterprises. According to the CNCF Annual Survey 2023, the usage of Kubernetes continues to grow, with significant adoption across different industries and use cases. Its reliability and flexibility make it a preferred choice for mission-critical applications, including databases, CI/CD pipelines, and AI and machine learning (ML) workloads. As a result, there is a growing demand for new features and enhancements, as well as simplifying concepts for users. The community is now prioritizing improvements that not only enhance user experiences but also promote the sustainability of the project. Figure 1 illustrates the anticipated future trends in Kubernetes, and below are the trends and innovations expected to shape Kubernetes' future in more detail. Figure 1. Future trends in Kubernetes AI and Machine Learning Kubernetes is increasingly used to orchestrate AI and ML workloads, supporting the deployment and management of complex ML pipelines. This simplifies the integration and scaling of AI applications across various environments. Innovations such as Kubeflow — an open-source platform designed to optimize the deployment, orchestration, and management of ML workflows on Kubernetes — enable data scientists to focus more on model development and less on infrastructure concerns. According to the recent CNCF open-source project velocity report, Kubeflow appeared on the top 30 CNCF project list for the first time in 2023, highlighting its growing importance in the ecosystem. Addressing the resource-intensive demands of AI introduces new challenges that contributors are focusing on, shaping the future of Kubernetes in the realm of AI and ML. The Developer Experience As Kubernetes evolves, its complexity can create challenges for new users. Hence, improving the user experience is crucial moving forward. Tools like Backstage are revolutionizing how developers work with Kubernetes and speeding up the development process. The CNCF's open-source project velocity report also states that "Backstage is addressing a significant pain point around developer experience." Moreover, the importance of platform engineering is increasingly recognized by companies. This emerging trend is expected to grow, with the goal of reducing the learning curve and making it easier for developers to adopt Kubernetes, thereby accelerating the development process and improving productivity. CI/CD and GitOps Kubernetes is revolutionizing continuous integration and continuous deployment (CI/CD) pipelines through the adoption of GitOps practices. GitOps uses Git repositories as the source of truth for declarative infrastructure and applications, enabling automated deployments. Tools like ArgoCD and Flux are being widely adopted to simplify the deployment process, reduce human error, and ensure consistency across environments. Figure 2 shows the integration between a GitOps operator, such as ArgoCD, and Kubernetes for managing deployments. This trend is expected to grow, making CI/CD pipelines more robust and efficient. Figure 2. Kubernetes GitOps Sustainability and Efficiency Cloud computing's carbon footprint now exceeds the airline industry, making sustainability and operational efficiency crucial in Kubernetes deployments. The Kubernetes community is actively developing features to optimize resource usage, reduce energy consumption, and enhance the overall efficiency of Kubernetes clusters. CNCF projects like KEDA (Kubernetes event-driven autoscaling) and Karpenter (just-in-time nodes for any Kubernetes cluster) are at the forefront of this effort. These tools not only contribute to cost savings but also align with global sustainability goals. Hybrid and Multi-Cloud Deployments According to the CNCF Annual Survey 2023, multi-cloud solutions are now the norm: Multi-cloud solutions (hybrid and other cloud combinations) are used by 56% of organizations. Deploying applications across hybrid and multi-cloud environments is one of Kubernetes' most significant advantages. This flexibility enables organizations to avoid vendor lock-in, optimize costs, and enhance resilience by distributing workloads across multiple cloud providers. Future developments in Kubernetes will focus on improving and simplifying management across different cloud platforms, making hybrid and multi-cloud deployments even more efficient. Increased Security Features Security continues to be a top priority for Kubernetes deployments. The community is actively enhancing security features to address vulnerabilities and emerging threats. These efforts include improvements to network policies, stronger identity and access management (IAM), and more advanced encryption mechanisms. For instance, the 2024 CNCF open-source project velocity report highlighted that Keycloak, which joined CNCF last year as an incubating project, is playing a vital role in advancing open-source IAM, backed by a large and active community. Edge Computing Kubernetes is playing a crucial role in the evolution of edge computing. By enabling consistent deployment, monitoring, and management of applications at the edge, Kubernetes significantly reduces latency, enhances real-time processing capabilities, and supports emerging use cases like IoT and 5G. Projects like KubeEdge and K3s are at the forefront of this movement. We can expect further optimizations for lightweight and resource-constrained environments, making Kubernetes even more suitable for edge computing scenarios. Conclusion Kubernetes has revolutionized cloud-native computing, transforming how we develop, deploy, and manage applications. As Kelsey Hightower noted in Google's Kubernetes Podcast, "We are only halfway through its journey, with the next decade expected to see Kubernetes mature to the point where it 'gets out of the way' by doing its job so well that it becomes naturally integrated into the background of our infrastructure." Kubernetes' influence will only grow, shaping the future of technology and empowering organizations to innovate and thrive in an increasingly complex landscape. References: "10 Years of Kubernetes" by Bob Killen et al, 2024 CNCF Annual Survey 2023 by CNCF, 2023 "As we reach mid-year 2024, a look at CNCF, Linux Foundation, and top 30 open source project velocity" by Chris Aniszczyk, CNCF, 2024 "Orchestration Celebration: 10 Years of Kubernetes" by Adrian Bridgwater, 2024 "Kubernetes: Beyond Container Orchestration" by Pratik Prakash, 2022 "The Staggering Ecological Impacts of Computation and the Cloud" by Steven Gonzalez Monserrate, 2022 This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report

When developing an enterprise system — whether it is a completely new system or simply the addition of a new feature — it is not uncommon to need to retrieve a significant volume of records (a few hundred or even thousands) from an underlying relational database. A usual scenario is having a list of identifiers (user IDs, for example) and needing to retrieve a set of data from each of them. Disregarding all the ease provided by Object-Relational Mappers, among which Hibernate can be underlined as a very reliable option, a straightforward way to address such a situation is to build a SQL SELECT query and bind all identifiers as a comma-delimited list of expressions in combination with a SQL IN operator. The snippet below shows the query structure. SQL SELECT column1, column2, ..., columnM FROM table WHERE identifier IN (id1, id2, id3, ..., idN) Some database management systems (DBMS), such as PostgreSQL, do not implement or define by default any restrictions on executing queries like that. Oracle, however, does not follow this policy. On the contrary, depending on the number of elements that make up the comma-delimited expression list used with the SQL IN operator or the number of records retrieved by the query, two corresponding errors may be triggered by the DBMS: ORA-01795: Maximum number of expressions in a list is 1000 ORA-00913: Too many values As indicated by the ORA-01795 error message, a comma-delimited list of expressions can contain no more than 1000 expressions. Then, a SQL query like described above can have up to 1,000 expressions with the IN operator. This upper bound may vary depending on the Oracle version. For example, the limit of 1,000 expressions is present up to the Oracle 21c version. As of the 23ai version, this value has been increased to 65,535. All tests and implementations in this article are based on the Oracle 19c version. As mentioned in ORA-00913 documentation, this error typically occurs in two scenarios: When a subquery in a WHERE or HAVING clause returns too many columns When a VALUES or SELECT clause returns more columns than are listed in the INSERT However, the ORA-00913 error also appears when SELECT queries consist of more than 65,535 hard-coded values. In the context of comma-delimited expressions, although the documentation states that "a comma-delimited list of sets of expressions can contain any number of sets, but each set can contain no more than 1000 expressions," such an upper bound is empirically verified. Given that the ORA-01795 and the ORA-00913 errors may constitute hindrances for systems needing to fetch more records than the bounds defined by Oracle, this article presents five strategies to work around these restrictions. As all strategies rely on hard coding SQL queries as Java Strings, two caveats about this approach are worth noting. Caveat 1: Chris Saxon, an Oracle Developer Advocate and active member of The Oracle Mentors (or AskTOM, for short), advised: Do not dyamically create a statement with hard wired IN list values or OR'ed expressions. [Oracle database]'ll spend more time parsing queries than actually executing them. Caveat 2: According to Billy Verreynne, an Oracle APEX expert and Oracle ACE, this approach: impacts on the amount of memory used by the Shared Pool, contributes to Shared Pool fragmentation, increases hard parsing, burns a load of CPU cycles, and degrades performance. QSplitter Before diving into each strategy, it is important to introduce a tool that will be used in almost all of them. It has been called QSplitter: a Java class responsible for splitting a collection into subcollections whose maximum size is also passed as a parameter. The splitCollection method returns a list in which each element is a subcollection extracted from the original collection. Its implementation is presented below. Java public class QSplitter<T> { public List<List<T>> splitCollection(Collection<T> collection, int maxCollectionSize) { var collectionArray = collection.toArray(); var splitCollection = new ArrayList<List<T>>(); var partitions = (collection.size() / maxCollectionSize) + (collection.size() % maxCollectionSize == 0 ? 0 : 1); Spliterator<T>[] spliterators = new Spliterator[partitions]; IntStream.range(0, partitions).forEach(n -> { var fromIndex = n * maxCollectionSize; var toIndex = fromIndex + maxCollectionSize > collection.size() ? fromIndex + collection.size() % maxCollectionSize : fromIndex + maxCollectionSize; spliterators[n] = Spliterators .spliterator(collectionArray, fromIndex, toIndex, Spliterator.SIZED); splitCollection.add(new ArrayList<>()); }); IntStream.range(0, partitions) .forEach(n -> spliterators[n].forEachRemaining(splitCollection.get(n)::add)); return splitCollection; } } Another element to describe is the database used for testing. Only one table was created and the SQL code used is shown below. SQL CREATE TABLE employee ( ID NUMBER GENERATED ALWAYS as IDENTITY(START with 1 INCREMENT by 1), NAME VARCHAR2(500), EMAIL VARCHAR2(500), STREET_NAME VARCHAR2(500), CITY VARCHAR2(500), COUNTRY VARCHAR2(500)); CREATE UNIQUE INDEX PK_EMPLOYEE ON EMPLOYEE ("ID"); 1. N Query Strategy The first strategy is the simplest: split the collection of IDs into subcollections, considering Oracle's upper limit (L = 1,000) for the comma-delimited list of expressions. Each subcollection is then used to build a query that fetches the corresponding subset of all desired records. The next script illustrates the generic queries generated for a collection of 3,000 IDs. Note: This strategy implies executing at least N = collection size / 1,000 queries. SQL SELECT column1, column2, ..., columnM FROM table WHERE id IN (id11, id12, id13, ..., id1L); SELECT column1, column2, ..., columnM FROM table WHERE id IN (id21, id22, id23, ..., id2L); SELECT column1, column2, ..., columnM FROM table WHERE id IN (id31, id32, id33, ..., id3L); To implement all strategies, two classes were implemented: UserService and UserDao. The service uses QSplitter to split the original collection and then pass each generated subcollection to the DAO. It builds and executes each respective query. The implementation is presented below. To keep the focus on the strategy details, some methods were hidden: getConnection for getting an Oracle java.sql.Connection and resultSetToUsers which creates a new User instance for each record in the java.sql.ResultSet. Java public class UserService { private final QSplitter<Long> qsplitter = new QSplitter<>(); private final UserDao dao = new UserDao(); public List<User> findUsersByNQuery(List<Long> ids) throws SQLException { var users = new ArrayList<User>(); this.qsplitter.splitCollection(ids).stream().map(this.dao::findByIds).forEach(users::addAll); return users; } } public class UserDao { private Function<Collection<Long>, String> buildSimpleSelectIn = ids -> new StringBuilder() .append("SELECT id, name, email, street_name, city, country FROM employee WHERE id IN (") .append(ids.stream().map(Object::toString).collect(Collectors.joining(","))) .append(")").toString(); public List<User> findByIds(Collection<Long> ids) throws SQLException { try (var rs = this.getConnection().prepareStatement(buildSimpleSelectIn.apply(ids)).executeQuery()) { var users = new ArrayList<User>(); this.resultSetToUsers(rs, users); return users; } } } 2. Disjunctions of Expression Lists Strategy The second strategy takes advantage of the feature stated in the Oracle Expression Lists documentation: "A comma-delimited list of sets of expressions can contain any number of sets." In this case, to avoid executing the N queries proposed by the previous approach, this strategy proposes to build a query composed of lists of expressions interrelated by the OR operator. The code below describes the new query. Note: If the number of IDs hard coded in the query exceeds the upper limit of 65,535, a second query must be created to avoid the ORA-00913 error. SQL SELECT column1, column2, ..., columnM FROM table WHERE id IN (id11, id12, id13, ..., id1N) OR id IN (id21, id22, id23, ..., id2N) OR id IN (id31, id32, id33, ..., id3N); The QSplitter tool must receive a new method responsible for grouping the subcollections (split from the original collection) according to the maximum number of elements allowed by Oracle. Java public class QSplitter<T> { public static final int MAX_ORACLE_IN_CLAUSE_ELEMENTS = 1000; public static final int MAX_ORACLE_RETRIEVE_ELEMENTS = 65535; public List<List<List<T>>> splitAndGroupCollection(Collection<T> collection) { var groupedCollection = new ArrayList<List<List<T>>>(); var splitCollection = this.splitCollection(collection, MAX_ORACLE_IN_CLAUSE_ELEMENTS); if (collection.size() <= MAX_ORACLE_RETRIEVE_ELEMENTS) { groupedCollection.add(splitCollection); return groupedCollection; } groupedCollection.add(new ArrayList<>()); splitCollection.forEach(partition -> { if (groupedCollection.getLast().size() * MAX_ORACLE_IN_CLAUSE_ELEMENTS + partition.size() > MAX_ORACLE_RETRIEVE_ELEMENTS) { groupedCollection.add(new ArrayList<>()); } groupedCollection.getLast().add(partition); }); return groupedCollection; } } The next code snippet shows the UserService using the new QSplitter method to split and group the ID collection. The UserDao has a java.util.function.Function that builds the query using this second strategy. Java public class UserService { public List<User> findUsersByDisjunctionsOfExpressionLists(List<Long> ids) throws SQLException { return this.dao.findByDisjunctionsOfExpressionLists(this.qsplitter.splitAndGroupCollection(ids)); } } public class UserDao { private Function<List<List<Long>>, String> buildSelectDisjunctions = idsList -> new StringBuilder("SELECT id, name, email, street_name, city, country FROM employee WHERE ") .append(idsList.stream().map(ids -> new StringBuilder() .append("id IN (").append(ids.stream().map(Object::toString) .collect(Collectors.joining(","))).append(")")) .collect(Collectors.joining(" OR "))).toString(); public List<User> findByDisjunctionsOfExpressionLists(List<List<List<Long>>> idsList) throws SQLException { var users = new ArrayList<User>(); try (var conn = this.getConnection()) { for (var ids : idsList) { try (var rs = conn.prepareStatement(buildSelectDisjunctions.apply(ids)).executeQuery()) { this.resultSetToUsers(rs, users); } } } return users; } } 3. Multivalued Expression Lists Strategy An alternative trick to get around the ORA-00913 error is to rewrite the expression list of IDs into a list of multivalued expressions. Any second value can be used as a second column to construct tuples. This simple change increases the capacity of the list of expressions from 1,000 to 65,535. Of course, if the size of the original collection exceeds the limit of 65,535, another query must be created. The structure of the query is described below. SQL SELECT column1, column2, ..., columnM FROM table WHERE (id, 0) IN ((id1, 0), (id2, 0), (id3, 0), ..., (id65535, 0)) Since the Java code is very similar to the previous strategy, it will be omitted here. 4. Union All Strategy Another option is to create a sequence of UNION ALL queries with the structure presented in the first strategy. SQL SELECT column1, column2, ..., columnM FROM table WHERE id IN (id1, id2, id3, ..., idL) UNION ALL SELECT column1, column2, ..., columnM FROM table WHERE id IN (idL+1, idL+2, idL+3, ..., id2L) UNION ALL SELECT column1, column2, ..., columnM FROM table WHERE id IN (id2L+1, id2L+2, id2L+3, ..., id3L) The implementation can reuse the code from the buildSimpleSelectIn function created for the first strategy. In this case, all generated SELECTs are joined by UNION ALL operators. It is worth noting that there is no restriction on the number of expression lists. So only one query is executed. The code is presented below. Java public class UserService { public List<User> findUsersByUnionAll(List<Long> ids) throws SQLException { return this.dao.findByUnionAll(this.qsplitter.splitCollection(ids)); } } public class UserDao { public List<User> findByUnionAll(List<List<Long>> idsList) throws SQLException { var query = idsList.stream().map(buildSimpleSelectIn).collect(Collectors.joining(" UNION ALL ")); try (var rs = this.getConnection().prepareStatement(query).executeQuery()) { var users = new ArrayList<User>(); this.resultSetToUsers(rs, users); return users; } } } 5. Temp Table Strategy The last strategy involves creating a temporary table into which all IDs should be inserted. The values inserted into the table must exist after the end of the transaction. To achieve this, the ON COMMIT PRESERVE ROWS clause must be part of the SQL statement for creating the table. The complete command is presented below. SQL CREATE GLOBAL TEMPORARY TABLE employee_id (emp_id NUMBER) ON COMMIT PRESERVE ROWS; The implementation doesn't use the QSplitter tool since there is no need to split the original collection of IDs. On the side of UserDao, it first inserts all IDs into the temp table (the JDBC batch API is employed). Next, a query with a JOIN clause for relating all IDs from the temp table is executed. The code is detailed below. Java public class UserService { public List<User> findUsersByTempTable(List<Long> ids) throws SQLException { return this.dao.findByTemporaryTable(ids); } } public class UserDao { public List<User> findByTemporaryTable(List<Long> ids) throws SQLException { var queryInsertTempTable = "INSERT INTO employee_id (emp_id) VALUES (?)"; var querySelectUsers = """ SELECT id, name, email, street_name, city, country FROM employee JOIN employee_id ON id = emp_id ORDER BY id """; try (var conn = this.getConnection()) { try (var pstmt = conn.prepareStatement(queryInsertTempTable)) { for (var id : ids) { pstmt.setLong(1, id); pstmt.addBatch(); } pstmt.executeBatch(); } var users = new ArrayList<User>(); try (var rs = conn.prepareStatement(querySelectUsers).executeQuery()) { this.resultSetToUsers(rs, users); } return users; } } } Performance The performance of the five strategies was compared considering collections of IDs ranging in size from 1,000 to 100,000. The code was implemented using Oracle JDK 21.0.4 and java.time.* API was used to count the wall time of each strategy. All tests were performed on the Windows 11 Pro for Workstations operating system installed on a machine with an Intel(R) Xeon(R) W-1290 CPU 3.20 GHz and 64 GB of RAM. As shown in the graph below, wall time for collections up to 10,000 IDs was very similar for all strategies. However, from this threshold onwards the performance of the N Queries strategy degraded considerably compared to the others. Such a behavior can be a result of the volume of I/O operations and external delays caused by executing multiple database queries. By removing the curve of the slowest strategy from the graph, the behavior of the most efficient ones can be better analyzed. Although there is not a big performance difference between them, it is visible that the Temp Table strategy dominates with lower times in all experiments. Even though it performs an additional step of inserting all IDs into the temporary table, this strategy does not need to use the QSplitter tool to generate subcollections and takes advantage of the use of the JOIN clause to avoid indicating hard-coded IDs in the SELECT. This tradeoff may have benefited the strategy's performance. Conclusion The need to retrieve a large volume of records from the Oracle database based on a large list of identifiers is not an uncommon task during the development of enterprise systems. This scenario can lead to Oracle errors ORA-01795 and ORA-00913. How difficult it is to work around these errors depends on various particularities of the source code and the system architecture. So, the five strategies presented constitute a toolkit for developers to avoid these errors and incorporate what best fits the system's needs. The complete source code is available on a GitHub repository.

By Thiago Nascimento CORE

These days, restaurants, food banks, home kitchens, and any other business that deals with products and foods that go bad quickly need to have good food inventory management. Kitchens stay organized and waste is kept to a minimum by keeping track of stock, checking expiration dates, and managing usage well. I will show you how to make a Food Inventory Management App in this guide. With this app, users can: Add food items or ingredients to the inventory. Monitor the quantity of each item. Remove items when they’re used or expired. Optionally, generate recipes or suggest uses for the items. The Food Inventory Management App will not only track food items but also generate recipes based on the available ingredients using a Hugging Face model. I will use Next.js for the front end, Material-UI for the user interface, Firebase Firestore for real-time database functionality, and a Hugging Face model for recipe generation. Setting Up the Environment for Development We need to set up our working environment before we start writing code for our Food Inventory Management App. 1. Install Node.js and npm The first step is to install Node.js and npm. Go to the Node.js website and get the Long Term Support version for your computer's running system. Follow the steps given for installation. 2. Making a Project With Next.js Start up your terminal and go to the location where you want to make your project. After that, run these commands: npx create-next-app@latest food-inventory-management-app (With the @latest flag, npm gets the most recent version of the Next.js starting setup.) cd food-inventory-management-app It will make a new Next.js project and take you to its path. You'll be given a number of configuration choices during the setup process, set them as given below: Would you like to use TypeScript? No Would you like to use ESLint? Yes Would you like to use Tailwind CSS? No Would you like to use the src/ directory? No Would you like to use App Router? Yes Would you like to customize the default import alias? No 3. Installing Firebase and Material-UI In the directory of your project, execute the following command: npm install @mui/material @emotion/react @emotion/styled firebase Setting Up Firebase Launch a new project on the Firebase Console. Click "Add app" after your project has been built, then choose the web platform (</>). Give your app a name when you register it, such as "Food Inventory Management App." Make a copy of the Firebase setup file. Afterwards, this will be useful. 4. Create a Firebase Configuration File Make a new file called firebase.js in the root directory of your project and add the following code, replacing the placeholders with the real Firebase settings for your project: JavaScript import { initializeApp } from 'firebase/app'; import { getFirestore } from 'firebase/firestore'; const firebaseConfig = { apiKey: "YOUR_API_KEY", authDomain: "YOUR_PROJECT_ID.firebaseapp.com", projectId: "YOUR_PROJECT_ID", storageBucket: "YOUR_PROJECT_ID.appspot.com", messagingSenderId: "YOUR_MESSAGING_SENDER_ID", appId: "YOUR_APP_ID" }; const app = initializeApp(firebaseConfig); const db = getFirestore(app); export { db }; Building a Flask API for Recipe Generation Using Hugging Face I'll show you how to make a Flask-based API that uses a Hugging Face model to make recipes. With a POST request, users will be able to send ingredients to the API. It will then use a pre-trained model from Hugging Face to return a recipe based on those ingredients. We will use environment variables to safely handle Hugging Face tokens. 1. Setting Up the Python Environment Install Python if not already present (brew install python). Verify installation (python3 --version). Install dependencies (pip install Flask flask-cors transformers huggingface_hub). 2. Setting Up the Hugging Face Token Go to the Hugging Face website. If you already have an account, click on Sign In. If not, click Sign Up to create a new account. Navigate to the dropdown menu, and select Settings. In the Settings menu, look for the Access Tokens tab on the left side of the page and click on it. Under the Access Tokens section, you will see a button to create a new token. Click on New Token. Give your token a descriptive name (e.g., "Food Inventory App Token"). Choose Read as the token scope for basic access to models and datasets. If you need write access for uploading models or data, choose Write. Click Generate Token. The token will be displayed on the screen. After generating the token, copy it. Make sure to save it in a secure place, as you will need it for authentication when making API calls. 3. Keeping Hugging Face API Token Safe Your Hugging Face API token should be kept safely in an environment variable instead of being written in your script as code. To do this: Create an .env file in the root of your project: (touch .env). Inside this file, add your Hugging Face token (HF_TOKEN=your_hugging_face_token_here). Load this environment variable securely in your Flask app using Python’s os module. Python import os huggingface_token = os.getenv('HF_TOKEN') 4. Building the Flask API Flask app with Hugging Face's recipe generation model (note: this sample model is free). Name the file as backend.py. Python import os from flask import Flask, request, jsonify from flask_cors import CORS from huggingface_hub import login from transformers import pipeline app = Flask(__name__) CORS(app) # Securely get the Hugging Face token from the environment huggingface_token = os.getenv('HF_TOKEN') if huggingface_token: login(token=huggingface_token) # Load Hugging Face food recipe model pipeline try: model_name = "flax-community/t5-recipe-generation" recipe_generator = pipeline("text2text-generation", model=model_name) except Exception as e: print(f"Error loading model: {e}") recipe_generator = None @app.route('/generate_recipe', methods=['POST']) def generate_recipe(): data = request.json print("Hello") ingredients = data.get('ingredients') if not ingredients: return jsonify({"error": "Ingredients not provided."}), 500 if recipe_generator: try: response = recipe_generator(f"Generate a recipe using the following ingredients: {ingredients}") return jsonify({"recipe": response[0]['generated_text']}) except Exception as e: print(f"Error generating recipe: {e}") return jsonify({"error": "Error generating recipe"}), 500 else: return jsonify({"error": "Recipe generator model is not available."}), 500 if __name__ == '__main__': app.run(debug=True, port=5001) Note: The flax-community/t5-recipe-generation model is loaded using the Hugging Face pipeline. This model can be utilized to generate recipes using the given/stored ingredients. Building the Core Components for the Food Inventory Management 1. Import All Necessary Libraries TypeScript 'use client'; import React, { useEffect, useState } from 'react'; import { Box, Stack, Typography, Button, TextField, IconButton, Tabs, Tab } from '@mui/material'; import { DatePicker, LocalizationProvider } from '@mui/x-date-pickers'; import { AdapterDayjs } from '@mui/x-date-pickers/AdapterDayjs'; import { collection, addDoc, deleteDoc, doc, onSnapshot, updateDoc, query, where, getDocs } from 'firebase/firestore'; import { db } from './firebase'; // Firebase configuration import DeleteIcon from '@mui/icons-material/Delete'; import dayjs from 'dayjs'; import axios from 'axios'; In this step, we set up our component with the basic layout and imports it needs. This is a client-side component, as shown by the 'use client' command at the top. 2. Utility Functions TypeScript # We define a utility function that capitalizes the first letter of a string. const capitalizeFirstLetter = (string) => string.charAt(0).toUpperCase() + string.slice(1); 3. State Management and useEffect for Firestore Snapshot We need to set up states to keep track of pantry items, new item input, expiration dates, search queries, active tabs, and recipe suggestions. items: Stores pantry items newItem: Stores the name of the item to be added expirationDate: Stores the expiration date of the new item searchQuery: Stores the search input for filtering items tabIndex: Stores the current tab (Available, Soon to Expire, Expired) recipe: Stores the generated recipe The useEffect hook monitors changes in Firestore data using the onSnapshot method, ensuring that the pantry items are always up to date. TypeScript export default function Pantry() { const [items, setItems] = useState([]); const [newItem, setNewItem] = useState(''); const [expirationDate, setExpirationDate] = useState(null); const [searchQuery, setSearchQuery] = useState(''); const [tabIndex, setTabIndex] = useState(0); const [recipe, setRecipe] = useState(''); // Fetch items from Firestore and update state in real-time useEffect(() => { const unsubscribe = onSnapshot(collection(db, 'pantryItems'), (snapshot) => { const itemsList = snapshot.docs.map((doc) => ({ id: doc.id, name: doc.data().name, quantity: doc.data().quantity, expirationDate: doc.data().expirationDate, })); setItems(itemsList); }); return () => unsubscribe(); }, []); 4. Add a New Item to Firestore This function is used to add a new item to the Firestore database. If the item is already present, its quantity is increased. Alternatively, the new item can be added with a designated expiration date. TypeScript // Add a new item to Firestore const addItemToFirestore = async () => { if (newItem.trim() !== '' && expirationDate) { const q = query(collection(db, 'pantryItems'), where('name', '==', newItem)); const querySnapshot = await getDocs(q); if (querySnapshot.empty) { await addDoc(collection(db, 'pantryItems'), { name: newItem, quantity: 1, expirationDate: expirationDate.toISOString() }); } else { querySnapshot.forEach(async (document) => { const itemRef = doc(db, 'pantryItems', document.id); await updateDoc(itemRef, { quantity: document.data().quantity + 1 }); }); } setNewItem(''); setExpirationDate(null); } }; 5. Remove an Item from Firestore or Decrease Its Quantity This function either decreases the quantity of an existing item or removes the item entirely if the quantity reaches zero. TypeScript // Remove an item or decrease its quantity const removeItemFromFirestore = async (id) => { const itemRef = doc(db, 'pantryItems', id); const itemDoc = await getDoc(itemRef); if (itemDoc.exists()) { const currentQuantity = itemDoc.data().quantity; if (currentQuantity > 1) { await updateDoc(itemRef, { quantity: currentQuantity - 1 }); } else { await deleteDoc(itemRef); } } }; 6. Fetch Recipe Suggestions From Flask Backend This function sends a list of available items and items that are close to their expiration date to the Flask backend for recipe generation. The backend generates a recipe and stores it in the recipe state. TypeScript // Fetch recipe suggestions using ingredients const fetchRecipeSuggestions = async (availableItems, soonToExpireItems) => { const ingredients = [...availableItems, ...soonToExpireItems].map(item => item.name).join(', '); try { const response = await axios.post('http://127.0.0.1:5001/generate_recipe', { ingredients }); setRecipe(response.data.recipe); } catch (error) { console.error('Error fetching recipe suggestions:', error.message); setRecipe('Error fetching recipe suggestions. Please try again later.'); } }; 7. Filter and Categorize Items Based on Expiration The pantry items are sorted according to their expiration dates. Three categories can be established: Available Items, Soon to Expire, and Expired Items. TypeScript // Filter and categorize items based on expiration const filteredItems = items.filter((item) => item.name.toLowerCase().includes(searchQuery.toLowerCase())); const soonToExpireItems = filteredItems.filter((item) => dayjs(item.expirationDate).diff(dayjs(), 'day') <= 7); const expiredItems = filteredItems.filter((item) => dayjs(item.expirationDate).diff(dayjs(), 'day') <= 0); const availableItems = filteredItems.filter((item) => !soonToExpireItems.includes(item) && !expiredItems.includes(item)); Building the UI Components for the Food Inventory Management TypeScript return ( <LocalizationProvider dateAdapter={AdapterDayjs}> <Box> {/* Add new pantry item */} <Stack spacing={2}> <Typography>Add Pantry Item</Typography> <TextField label="Add Pantry Item" value={newItem} onChange={(e) => setNewItem(e.target.value)} /> <DatePicker label="Expiration Date" value={expirationDate} onChange={(newValue) => setExpirationDate(newValue)} /> <Button onClick={addItemToFirestore}>Add Item</Button> </Stack> {/* Search and Tabs */} <TextField label="Search Pantry Items" value={searchQuery} onChange={(e) => setSearchQuery(e.target.value)} /> <Tabs value={tabIndex} onChange={(e, newValue) => setTabIndex(newValue)}> <Tab label="Available Items" /> <Tab label="Soon to Expire" /> <Tab label="Expired Items" /> </Tabs> {/* Display Items */} {tabIndex === 0 && availableItems.map((item) => ( <Box key={item.id}> <Typography>{capitalizeFirstLetter(item.name)} - {item.quantity}</Typography> <IconButton onClick={() => removeItemFromFirestore(item.id)}><DeleteIcon /></IconButton> </Box> ))} {tabIndex === 1 && soonToExpireItems.map((item) => ( <Box key={item.id}> <Typography>{capitalizeFirstLetter(item.name)} - {item.quantity} (Expires: {dayjs(item.expirationDate).format('YYYY-MM-DD')})</Typography> <IconButton onClick={() => removeItemFromFirestore(item.id)}><DeleteIcon /></IconButton> </Box> ))} {tabIndex === 2 && expiredItems.map((item) => ( <Box key={item.id}> <Typography>{capitalizeFirstLetter(item.name)} - {item.quantity} (Expired: {dayjs(item.expirationDate).format('YYYY-MM-DD')})</Typography> <IconButton onClick={() => removeItemFromFirestore(item.id)}><DeleteIcon /></IconButton> </Box> ))} {/* Fetch Recipe Suggestions */} <Button onClick={() => fetchRecipeSuggestions(availableItems, soonToExpireItems)}>Get Recipe Suggestions</Button> {recipe && <Typography>{recipe}</Typography>} </Box> </LocalizationProvider> ); } Explanation of UI Components 1. <Box> (Container) Purpose: Acts as a flexible container for managing layout, padding, and alignment Used for: Wrapping sections like the form, search bar, and item lists 2. <Stack> (Vertical/Horizontal Layout) Purpose: Organizes child components in a vertical or horizontal layout Used for: Structuring form elements and item listings with proper spacing 3. <Typography> (Text Display) Purpose: Renders and styles text content Used for: Displaying headings, item names, expiration dates, and recipe suggestions 4. <TextField> (Input Field) Purpose: Provides a text input field. Used for: Inputting new pantry item names and search queries 5. <DatePicker> (Date Selection) Purpose: Allows users to pick a date from a calendar Used for: Selecting expiration dates for pantry items, integrated with the Day.js adapter 6. <Button> (Clickable Button) Purpose: A clickable button for actions Used for: Adding items to Firestore, fetching recipes, and interacting with the database 7. <Tabs> and <Tab> (Tab Navigation) Purpose: Creates a tabbed interface for navigation Used for: Switching between available, soon-to-expire, and expired items 8. <IconButton> (Icon-Based Button) Purpose: Button with an icon for quick actions. Used for: Deleting or reducing the quantity of items, using a delete icon 9. <LocalizationProvider> (Date Localization) Purpose: Manages date localization and formatting Used for: Ensuring correct display and handling of dates in the date picker 10. <DeleteIcon> (Icon) Purpose: Displays a delete icon for action Used for: Indicating delete action on buttons for item removal 11. Recipe Suggestion Section Purpose: Displays recipe suggestions based on available ingredients Used for: Showing the recipe generated by the Flask API when the "Get Recipe Suggestions" button is clicked 12. <Grid> (Responsive Layout) Purpose: Creates responsive layouts with flexible columns. Used for aligning content: Organizing elements like forms and buttons within a structured grid. Dividing UI into columns: Structuring content into columns and rows for a clean, responsive layout on various screen sizes. Running the Food Inventory Management Application 1. Start the Development Server TypeScript npm run dev Navigate to http://localhost:3000 as prompted by opening your browser. 2. Start the Flask Development Server Start the Flask Development Server along with the below. Python python backend.py This will initiate the Flask API at http://127.0.0.1:5001. Please remember the interaction between React and Flask is done using Axios to send HTTP requests and display the results in real time. The Hugging Face model I used is free to use. If you want to use a different model, like Llama, you can do that too. Sample Image of the Food Inventory Management Application After Development Conclusion Congratulations! You have successfully developed a functional Food Inventory Management Application. Happy coding!

Java 23 is finally out, and we can start migrating our project to it. The very first pitfall comes quickly when switching to the latest JDK 23 with compilation issues when using the Lombok library in your project. Let's begin with the symptom description first. Description The Lombok library heavily relies on annotations. It's used for removing a lot of boilerplate code; e.g., getters, setters, toString, loggers, etc. @Slf4j usage for simplified logging configuration Maven compilation errors coming from Lombok and Java 23 look like this: Plain Text [INFO] --- [compiler:3.13.0:compile [ (default-compile) @ sat-core --- [WARNING] Parameter 'forceJavacCompilerUse' (user property 'maven.compiler.forceJavacCompilerUse') is deprecated: Use forceLegacyJavacApi instead [INFO] Recompiling the module because of changed source code [INFO] Compiling 50 source files with javac [debug parameters release 23] to target\classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] spring-advanced-training\sat-core\src\main\java\com\github\aha\sat\core\aop\BeverageLogger.java:[21,2] error: cannot find symbol symbol: variable log location: class BeverageLogger ... [INFO] 16 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] [BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 3.090 s [INFO] Finished at: 2024-09-26T08:45:59+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal [org.apache.maven.plugins:maven-compiler-plugin:3.13.0:compile (default-compile) on project sat-core: Compilation failure: Compilation failure: [ERROR] spring-advanced-training\sat-core\src\main\java\com\github\aha\sat\core\aop\BeverageLogger.java:[21,2] error: cannot find symbol [ERROR] symbol: variable log [ERROR] location: class BeverageLogger ... Note: The @Slf4j annotation is just an example. It's demonstrated here because these are the first errors in the build logs. However, it's related to any other already mentioned Lombok annotation. Explanation The compilation error is caused by a change in the behavior of annotation processing in Java 23. See JDK 23 Release notes and this statement: As of JDK 23, annotation processing is only run with some explicit configuration of annotation processing or with an explicit request to run annotation processing on the javac command line. This is a change in behavior from the existing default of looking to run annotation processing by searching the class path for processors without any explicit annotation processing related options needing to be present. You can find more details about it here. Solution In order to be able to use Lombok with the new Java 23, we need to turn on the full compilation processing. It can be done in Maven as: To have the latest maven-compiler-version (it's version 3.13.0 at the time of writing this article) Setup maven.compiler.proc property with full value. XML <properties> ... <java.version>23</java.version> <maven-compiler-plugin.version>3.13.0</maven-compiler-plugin.version> <maven.compiler.proc>full</maven.compiler.proc> </properties> <build> <plugins> ... <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>${maven-compiler-plugin.version}</version> <configuration> <source>${java.version}</source> <target>${java.version}</target> </configuration> </plugin> </plugins> </build> It's all we need to make our project compilable again. Plain Text [INFO] --- compiler:3.13.0:compile (default-compile) @ sat-core --- [WARNING] Parameter 'forceJavacCompilerUse' (user property 'maven.compiler.forceJavacCompilerUse') is deprecated: Use forceLegacyJavacApi instead [INFO] Recompiling the module because of changed source code. [INFO] Compiling 50 source files with javac [debug parameters release 23] to target\classes [INFO] [INFO] --- resources:3.3.1:testResources (default-testResources) @ sat-core --- [INFO] Copying 2 resources from src\test\resources to target\test-classes Conclusion This article has covered the issue related to using the Lombok library and upgrading to JDK 23. The complete change (but with more changes) is visible in this GitHub commit.

By Arnošt Havelka CORE