The latest and popular trending topics on DZone.
Artificial intelligence (AI) and machine learning (ML) are two fields that work together to create computer systems capable of perception, recognition, decision-making, and translation. Separately, AI is the ability for a computer system to mimic human intelligence through math and logic, and ML builds off AI by developing methods that "learn" through experience and do not require instruction. In the AI/ML Zone, you'll find resources ranging from tutorials to use cases that will help you navigate this rapidly growing field.
Java is an object-oriented programming language that allows engineers to produce software for multiple platforms. Our resources in this Zone are designed to help engineers with Java program development, Java SDKs, compilers, interpreters, documentation generators, and other tools used to produce a complete application.
JavaScript (JS) is an object-oriented programming language that allows engineers to produce and implement complex features within web browsers. JavaScript is popular because of its versatility and is preferred as the primary choice unless a specific function is needed. In this Zone, we provide resources that cover popular JS frameworks, server applications, supported data types, and other useful topics for a front-end engineer.
Open source refers to non-proprietary software that allows anyone to modify, enhance, or view the source code behind it. Our resources enable programmers to work or collaborate on projects created by different teams, companies, and organizations.
Instant APIs With Copilot and API Logic Server
Build a Time-Tracking App With ClickUp API Integration Using Openkoda
This article dives into the world of distributed systems and explores a fundamental principle called the CAP theorem. Distributed systems play a crucial role in many modern applications, and the CAP theorem helps us understand the trade-offs inherent in these systems. What Are Distributed Systems? Distributed systems distribute computations and data across multiple interconnected nodes within a network. This can involve offloading processing power or geographically dispersing data for faster delivery. Unlike centralized systems that store data in a single location, distributed systems spread data across multiple interconnected points. This distribution brings its own set of challenges, with the first hurdle being network failures. Network links can break, creating network partitions where the entire network gets divided into isolated groups. Each partition loses communication with the others. A system's ability to withstand such partitions is called partition tolerance. It's important to note that achieving 100% partition tolerance is practically impossible; all distributed systems are susceptible to network partitions at some point. These partitions, though temporary, disrupt communication between parts of the system. Let's consider a social media platform like Instagram or Facebook. If a partition occurs, a newly posted photo might not be visible to users in another partition until the network recovers. This leads us to another crucial property that the distributed system has. The “Consistency”. You already noticed that if two partitions occurred, then data seen by two partitions are different (or in other words inconsistent). Consistency is a measurement of whether the system data is correct over the network at a given time. Consistency plays a crucial role in financial applications. Unlike social media posts, how fast your data is consistent across the system is a measurement of how consistent your system is. If not this might cause serious problems such as a double spending problem. If you don’t know the double expending problem, then it is about a financial system that holds each person’s balance. Assume Alice has 100$ in her account and all the servers consistent with Alice’s account balance of 100$. Alice bought 90$ worth of watches online from server A. Server A completed the transaction and sent a notification to other servers to deduct Alice’s account 90$. But before the message propagates, a sudden partition occurs and server B does not get Alice’s transaction. If Alice calls server B and performs another transaction, the server still considers Alice to have 100$ and lets her spend it again. If Alice buys a bag for 50$ from server B, the transaction still passes. As you can see, in this kind of financial system, consistency is a big matter and it should have higher consistency. In contrast to social media platforms, it does not matter how fast you receive your friend’s update. Now we know financial transaction systems need higher consistency. But how are we supposed to achieve it? There can be many consistency levels but let’s analyze the following levels which are used mostly for distributed systems: Linearizability Sequential Consistency Causal Consistency Eventual Consistency Linearizability Linearizability is the highest consistency level. This consistency algorithm works by adding a read lock for all the nodes in the system if any of the node data needs to be updated. By adding a read lock for all the nodes, we can make sure that any of the nodes in the system do not read partially updated data. The lock gets removed once all the data is in a consistent state. If there is network partitioning, the system takes more time to come to a consistent state. What will happen if a client connects to a node and requests to read data while the node is locked? Well, the client has to wait until the node releases the lock. This leads us to the third important property, Availability. The availability is a property when the client requests something from a node, it responds. If the node is not ready to serve the request, the client gets no or failed response. With the Linearizability example, we are locking the node so that the client does not get a response. This means until data become consistent, nodes are not available. We can conclude that if consistency is higher, we cannot achieve higher availability. Sequential Consistency In contrast to Linearizability, the Sequential Consistency is a relaxed consistency model. Instead of locking all the nodes for all the data updates, sequential consistency locks nodes in chronological order. Think about a chat application. When two people chat, both people’s messages should be in proper chronological order. If not, it would become a mess to understand it. Let us understand it with an example. Alice needs to send a message to the chat group and she sends the message to the system. Assuming the system has no partitions, her message propagates to all the nodes. Now Bob also needs to send a message to the group but a network partition occurred and some nodes do not get updated with Bob’s message. Now if Alice sends another message in this partitioned system some nodes are still not updated with the previous Bobs message. If the node does not update Bob’s message and only adds Alice’s message, the chat becomes a mess. In this scenario, a sequentially consistent system puts a write lock on the node where only it can write the node if all previously published data is already added. Until the node gets updated with previous data, it has to wait until previously sent messages reach the node. In this consistency mode, we are only considering the sequential consistency of each node data. Here you can see nodes are available than in the linearizability model where the write lock gets applied until the order of the events is resolved. The client can still get a response from the given node in the partitioned system but up to the last data in the correct sequential order. Causal Consistency The Causal Consistency is a much more relaxed consistency model that doesn’t care about order as well. Causal Consistency only considers about relation between data. For example, think about a social media platform where you can post images and put comments. Alice posts a photo to the platform and at the same time, Bob also posts a photo to the platform. These two posts do not have any relation. The order of the post does not affect the third person Charlie looking at both of the photos. But, for each photo, there is a comment section. When Charlie sees comments on the photos those comments should be in chronological order to him to understand it. Comments in this system are Causally consistent. This means that order does matter for some cases, but not for all the scenarios. If there are unrelated entities(Such as posts) having more relaxed consistency while comments have a dependency on their related posts. Eventual Consistency Now we can understand the Eventual Consistency easily. Think about the same social media platform without a comment feature. Now Alice and Bob post photos and Charlie can see their post on his feed. It does not matter which order he received the data. This leads us to think about another important fact about availability. In the linearizability consistency level, we could not achieve higher availability due to locking. But in Eventual Consistency, we don’t need to have any locks. Therefore server nodes are available at all times. What CAP Theorem Is All About? Now we discovered all the pieces of the CAP theorem and it is time to complete the puzzle. We have discussed three properties that the distributed system has. The Partition Tolerance tells us the system can tolerate partitioning, The Consistency which maintains data consistent over the distributed system, and the Availability which makes sure the system always responds to client requests. CAP theorem states that we can only select two of these properties out of three. Let’s have a look at all three cases. Select Partition Tolerance and Consistency Over the Availability This is the scenario in which we allow the system to have a partition. But we need to keep the system consistent. The financial transaction scenario we have discussed belongs to this category. To maintain consistency, we have to lock the nodes until data becomes consistent over the system. Only then is it allowed to read from the system. Therefore the availability is limited. Select Partition Tolerance and Availability Over the Consistency This is the scenario in which we don’t need strict consistency in the system. Remind about the social media system discussed in the eventual consistency. We don’t need consistent data but rather have data in whatever order. Having a relaxed consistency level, we don’t need to lock nodes and nodes are always available. Availability and Consistency Over the Partition Tolerance This kind of system is more likely a centralized system rather than a distributed system. We cannot build a system without having network partitioning. If we ensure Availability and Consistency at the same time, then there cannot be any partitions. Which means it is a centralized system. Therefore, we don’t discuss both available and Consistent systems in distributed systems. Example Use Cases for CAP Theorem Now you know what the CAP theorem is. Let’s see some example tools used in all three above cases. Partition Tolerance and Consistent System Redis and MongoDB are popular database solutions for CP(Consistent and Partition-tolerant systems). These databases are built to work as distributed databases and let there be partitions. Even if there are partitions, it lets you have consistent data over all the Databases connected. Partition Tolerance and Available System This system does not much care about consistency on the system. Rather it cares about being responsive and faster operation. This includes databases such as Cassandra and CouchDB. These databases are built to operate in distributed environments and faster operation. Consistent and Available System This type of system is not meant to be distributed but rather a centralized system. Databases such as SQLite and H2 are built to run on a single node. These databases are always consistent and available since they don’t need to lock nodes and it is the only node. But you cannot have it on a distributed system as it cannot tolerate partitioning. Why is MYSQL not listed for any of those? Well MySQL is generally considered a CA system designed to provide good consistency and availability while sacrificing partition tolerance. The reason for this is that MYSQL runs in a master-slave mode where the master is the node that the client to access. Therefore it is not a true distributed system that we are talking. But with different configurations, still you can work it as a CP system. As you can see, the CAP theorem imposes limitations on distributed systems. As system designers and developers, we must choose which properties to prioritize: consistency, availability, or partition tolerance. The chosen consistency model will determine whether we prioritize high availability or sacrifice some availability to ensure consistency. Many tools exist to help you design your application based on these trade-offs. Choose wisely! The next time you encounter a distributed system, consider these three crucial properties.
In Java programming, object creation or instantiation of a class is done with "new" operator and with a public constructor declared in the class as below. Java Clazz clazz = new Clazz(); We can read the code snippet as follows: Clazz() is the default public constructor called with "new" operator to create or instantiate an object for Clazz class and assigned to variable clazz, whose type is Clazz. While creating a singleton, we have to ensure only one single object is created or only one instantiation of a class takes place. To ensure this, the following common things become the prerequisite. All constructors need to be declared as "private" constructors. It prevents the creation of objects with "new" operator outside the class. A private constant/variable object holder to hold the singleton object is needed; i.e., a private static or a private static final class variable needs to be declared. It holds the singleton object. It acts as a single source of reference for the singleton object By convention, the variable is named as INSTANCE or instance. A static method to allow access to the singleton object by other objects is required. This static method is also called a static factory method, as it controls the creation of objects for the class. By convention, the method is named as getInstance(). With this understanding, let us delve deeper into understanding singleton. Following are the 6 ways one can create a singleton object for a class. 1. Static Eager Singleton Class When we have all the instance properties in hand, and we like to have only one object and a class to provide a structure and behavior for a group of properties related to each other, we can use the static eager singleton class. This is well-suited for application configuration and application properties. Java public class EagerSingleton { private static final EagerSingleton INSTANCE = new EagerSingleton(); private EagerSingleton() {} public static EagerSingleton getInstance() { return INSTANCE; } public static void main(String[] args) { EagerSingleton eagerSingleton = EagerSingleton.getInstance(); } } The singleton object is created while loading the class itself in JVM and assigned to the INSTANCE constant. getInstance() provides access to this constant. While compile-time dependencies over properties are good, sometimes run-time dependencies are required. In such a case, we can make use of a static block to instantiate singleton. Java public class EagerSingleton { private static EagerSingleton instance; private EagerSingleton(){} // static block executed during Class loading static { try { instance = new EagerSingleton(); } catch (Exception e) { throw new RuntimeException("Exception occurred in creating EagerSingleton instance"); } } public static EagerSingleton getInstance() { return instance; } } The singleton object is created while loading the class itself in JVM as all static blocks are executed while loading. Access to the instance variable is provided by the getInstance() static method. 2. Dynamic Lazy Singleton Class Singleton is more suited for application configuration and application properties. Consider heterogenous container creation, object pool creation, layer creation, facade creation, flyweight object creation, context preparation per requests, and sessions, etc.: they all require dynamic construction of a singleton object for better "separation of concern." In such cases, dynamic lazy singletons are required. Java public class LazySingleton { private static LazySingleton instance; private LazySingleton(){} public static LazySingleton getInstance() { if (instance == null) { instance = new LazySingleton(); } return instance; } } The singleton object is created only when the getInstance() method is called. Unlike the static eager singleton class, this class is not thread-safe. Java public class LazySingleton { private static LazySingleton instance; private LazySingleton(){} public static synchronized LazySingleton getInstance() { if (instance == null) { instance = new LazySingleton(); } return instance; } } The getInstance() method needs to be synchronized to ensure the getInstance() method is thread-safe in singleton object instantiation. 3. Dynamic Lazy Improved Singleton Class Java public class LazySingleton { private static LazySingleton instance; private LazySingleton(){} public static LazySingleton getInstance() { if (instance == null) { synchronized (LazySingleton.class) { if (instance == null) { instance = new LazySingleton(); } } } return instance; } } Instead of locking the entire getInstance() method, we could lock only the block with double-checking or double-checked locking to improve performance and thread contention. Java public class EagerAndLazySingleton { private EagerAndLazySingleton(){} private static class SingletonHelper { private static final EagerAndLazySingleton INSTANCE = new EagerAndLazySingleton(); } public static EagerAndLazySingleton getInstance() { return SingletonHelper.INSTANCE; } } The singleton object is created only when the getInstance() method is called. It is a Java memory-safe singleton class. It is a thread-safe singleton and is lazily loaded. It is the most widely used and recommended. Despite performance and safety improvement, the only objective to create just one object for a class is challenged by memory reference, reflection, and serialization in Java. Memory reference: In a multithreaded environment, reordering of read and writes for threads can occur on a referenced variable, and a dirty object read can happen anytime if the variable is not declared volatile. Reflection: With reflection, the private constructor can be made public and a new instance can be created. Serialization: A serialized instance object can be used to create another instance of the same class. All of these affect both static and dynamic singletons. In order to overcome such challenges, it requires us to declare the instance holder as volatile and override equals(), hashCode() and readResolve() of default parent class of all classes in Java, Object.class. 4. Singleton With Enum The issue with memory safety, reflection, and serialization can be avoided if enums are used for static eager singleton. Java public enum EnumSingleton { INSTANCE; } These are static eager singletons in disguise, thread safe. It is good to prefer an enum where a static eagerly initialized singleton is required. 5. Singleton With Function and Libraries While understanding the challenges and caveats in singleton is a must to appreciate, why should one worry about reflection, serialization, thread safety, and memory safety when one can leverage proven libraries? Guava is such a popular and proven library, handling a lot of best practices for writing effective Java programs. I have had the privilege of using the Guava library to explain supplier-based singleton object instantiation to avoid a lot of heavy-lifting lines of code. Passing a function as an argument is the key feature of functional programming. While the supplier function provides a way to instantiate object producers, in our case, the producer must produce only one object and should keep returning the same object repeatedly after a single instantiation. We can memoize/cache the created object. Functions defined with lambdas are usually lazily invoked to instantiate objects and the memoization technique helps in lazily invoked dynamic singleton object creation. Java import com.google.common.base.Supplier; import com.google.common.base.Suppliers; public class SupplierSingleton { private SupplierSingleton() {} private static final Supplier<SupplierSingleton> singletonSupplier = Suppliers.memoize(()-> new SupplierSingleton()); public static SupplierSingleton getInstance() { return singletonSupplier.get(); } public static void main(String[] args) { SupplierSingleton supplierSingleton = SupplierSingleton.getInstance(); } } Functional programming, supplier function, and memoization help in the preparation of singletons with a cache mechanism. This is most useful when we don't want heavy framework deployment. 6. Singleton With Framework: Spring, Guice Why worry about even preparing an object via supplier and maintaining cache? Frameworks like Spring and Guice work on POJO objects to provide and maintain singleton. This is heavily used in enterprise development where many modules each require their own context with many layers. Each context and each layer are good candidates for singleton patterns. Java import org.springframework.beans.factory.config.ConfigurableBeanFactory; import org.springframework.context.annotation.AnnotationConfigApplicationContext; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Scope; class SingletonBean { } @Configuration public class SingletonBeanConfig { @Bean @Scope(value = ConfigurableBeanFactory.SCOPE_SINGLETON) public SingletonBean singletonBean() { return new SingletonBean(); } public static void main(String[] args) { AnnotationConfigApplicationContext applicationContext = new AnnotationConfigApplicationContext(SingletonBean.class); SingletonBean singletonBean = applicationContext.getBean(SingletonBean.class); } } Spring is a very popular framework. Context and Dependency Injection are the core of Spring. import com.google.inject.AbstractModule; import com.google.inject.Guice; import com.google.inject.Injector; interface ISingletonBean {} class SingletonBean implements ISingletonBean { } public class SingletonBeanConfig extends AbstractModule { @Override protected void configure() { bind(ISingletonBean.class).to(SingletonBean.class); } public static void main(String[] args) { Injector injector = Guice.createInjector(new SingletonBeanConfig()); SingletonBean singletonBean = injector.getInstance(SingletonBean.class); } } Guice from Google is also a framework to prepare singleton objects and an alternative to Spring. Following are the ways singleton objects are leveraged with "Factory of Singletons." Factory Method, Abstract Factory, and Builders are associated with the creation and construction of specific objects in JVM. Wherever we envision the construction of an object with specific needs, we can discover the singleton's need. Further places where one can check out and discover singleton are as follows. Prototype or Flyweight Object pools Facades Layering Context and class loaders Cache Cross-cutting concerns and aspect-oriented programming Conclusion Patterns appear when we solve use cases for our business problems and for our non-functional requirement constraints like performance, security, and CPU and memory constraints. Singleton objects for a given class is such a pattern, and requirements for its use will fall in place to discover. The class by nature is a blueprint to create multiple objects, yet the need for dynamic heterogenous containers to prepare "context," "layer,", "object pools," and "strategic functional objects" did push us to make use of declaring globally accessible or contextually accessible objects. Thanks for your valuable time, and I hope you found something useful to revisit and discover.
Effective conflict management is essential for technology teams to maintain a productive and innovative work environment. Google's Project Aristotle, an internal study aimed at understanding the factors that contribute to successful teams, highlighted the importance of psychological safety and open communication in fostering collaboration and navigating disagreements among team members. This article will explore conflict management strategies inspired by Google's Project Aristotle and additional data points, using a use case to illustrate their application in real-life scenarios. Conflict Management Strategies 1. Embrace Diverse Perspectives Technology teams often consist of individuals with varied backgrounds, expertise, and perspectives. Encourage team members to appreciate and learn from these differences, as diverse viewpoints can lead to more creative and innovative solutions. 2. Foster Psychological Safety Create an environment where team members feel comfortable expressing their thoughts, concerns, and ideas without fear of ridicule or retribution. Psychological safety promotes constructive feedback, open communication, and effective conflict resolution. 3. Encourage Collaboration Organize joint brainstorming sessions and workshops to encourage collaboration between team members with different expertise. This cooperative approach can help build trust and rapport among team members, making it easier to navigate conflicts when they occur. 4. Utilize Effective Communication Tools Leverage various communication tools, such as instant messaging, video conferencing, and project management software, to facilitate seamless communication among team members. This can help prevent misunderstandings and ensure that everyone stays informed and engaged. 5. Offer Training in Soft Skills In addition to technical expertise, soft skills such as communication, empathy, and emotional intelligence are crucial for effective conflict management in technology teams. Provide training and resources to help team members develop these essential interpersonal skills. 6. Conduct Regular Retrospectives Hold regular retrospectives to reflect on project progress, team dynamics, and potential areas of improvement. These meetings provide an opportunity for team members to openly discuss any conflicts or challenges they have encountered and collaborate on solutions. 7. Adapt To Change and Manage Expectations Technology projects often involve rapidly changing requirements and priorities. Help team members adapt to change by setting realistic expectations, being transparent about project status, and providing support during transitions. 8. Promote Work-Life Balance Encourage team members to maintain a healthy work-life balance, as excessive stress and burnout can exacerbate conflicts. Support flexible work arrangements and create a culture that values downtime and self-care. Use Case: A Software Development Team at Google The Google Maps team consists of engineers, product managers, and UX designers with diverse backgrounds and expertise. As the project progresses, a conflict arises between engineers and UX designers over the implementation of a particular feature. The engineers argue that the proposed design is too resource-intensive, while the UX designers insist that it is essential for a seamless user experience. By applying the conflict management strategies mentioned above, the Google Maps team can effectively address this conflict: The team leader promotes open discussion and values each team member's unique perspective, leading to more innovative ideas and solutions. The team leader ensures that all team members feel comfortable expressing their concerns and ideas, creating an environment where constructive feedback and open communication are valued. The team leader organizes a joint brainstorming session between the engineers and UX designers to explore alternative solutions that meet both the user experience goals and the technical constraints. The team uses Google Workspace tools like Google Meet, Google Chat, and Google Docs to facilitate seamless communication, collaboration, and documentation of decisions. Google provides workshops and resources on effective communication, negotiation, and problem-solving, helping team members develop the interpersonal skills needed to navigate conflicts successfully. After resolving the conflict, the team holds a retrospective meeting to reflect on the situation, discuss lessons learned, and identify ways to prevent similar conflicts in the future. The team leader communicates any changes in project requirements or priorities clearly, helping team members adapt and manage their expectations. The company supports flexible work arrangements and encourages team members to maintain a healthy work-life balance, reducing stress and potential conflicts. Conclusion By applying the lessons learned from Google's Project Aristotle and incorporating additional data points, technology teams can effectively manage conflicts and foster a harmonious and productive work environment. Embracing diverse perspectives, fostering psychological safety, encouraging collaboration, utilizing communication tools, offering soft skills training, conducting regular retrospectives, adapting to change, and promoting work-life balance are key strategies that empower team members to navigate and resolve conflicts effectively. With a proactive approach to conflict management, technology teams can become more engaged, satisfied, and high-performing.
The AIDocumentLibraryChat project has been extended to support questions for searching relational databases. The user can input a question and then the embeddings search for relevant database tables and columns to answer the question. Then the AI/LLM gets the database schemas of the relevant tables and generates based on the found tables and columns a SQL query to answer the question with a result table. Dataset and Metadata The open-source dataset that is used has 6 tables with relations to each other. It contains data about museums and works of art. To get useful queries of the questions, the dataset has to be supplied with metadata and that metadata has to be turned in embeddings. To enable the AI/LLM to find the needed tables and columns, it needs to know their names and descriptions. For all datatables like the museum table, metadata is stored in the column_metadata and table_metadata tables. Their data can be found in the files: column_metadata.csv and table_metadata.csv. They contain a unique ID, the name, the description, etc. of the table or column. That description is used to create the embeddings the question embeddings are compared with. The quality of the description makes a big difference in the results because the embedding is more precise with a better description. Providing synonyms is one option to improve the quality. The Table Metadata contains the schema of the table to add only the relevant table schemas to the AI/LLM prompt. Embeddings To store the embeddings in Postgresql, the vector extension is used. The embeddings can be created with the OpenAI endpoint or with the ONNX library that is provided by Spring AI. Three types of embeddings are created: Tabledescription embeddings Columndescription embeddings Rowcolumn embeddings The Tabledescription embeddings have a vector based on the table description and the embedding has the tablename, the datatype = table, and the metadata id in the metadata. The Columndescription embeddings have a vector based on the column description and the embedding has the tablename, the dataname with the column name, the datatype = column, and the metadata id in the metadata. The Rowcolumn embeddings have a vector based on the content row column value. That is used for the style or subject of an artwork to be able to use the values in the question. The metadata has the datatype = row, the column name as dataname, the tablename, and the metadata id. Implement the Search The search has 3 steps: Retrieve the embeddings Create the prompt Execute query and return result Retrieve the Embeddings To read the embeddings from the Postgresql database with the vector extension, Spring AI uses the VectorStore class in the DocumentVSRepositoryBean: Java @Override public List<Document> retrieve(String query, DataType dataType) { return this.vectorStore.similaritySearch( SearchRequest.query(query).withFilterExpression( new Filter.Expression(ExpressionType.EQ, new Key(MetaData.DATATYPE), new Value(dataType.toString())))); } The VectorStore provides a similarity search for the query of the user. The query is turned in an embedding and with the FilterExpression for the datatype in the header values, the results are returned. The TableService class uses the repository in the retrieveEmbeddings method: Java private EmbeddingContainer retrieveEmbeddings(SearchDto searchDto) { var tableDocuments = this.documentVsRepository.retrieve( searchDto.getSearchString(), MetaData.DataType.TABLE, searchDto.getResultAmount()); var columnDocuments = this.documentVsRepository.retrieve( searchDto.getSearchString(), MetaData.DataType.COLUMN, searchDto.getResultAmount()); List<String> rowSearchStrs = new ArrayList<>(); if(searchDto.getSearchString().split("[ -.;,]").length > 5) { var tokens = List.of(searchDto.getSearchString() .split("[ -.;,]")); for(int i = 0;i<tokens.size();i = i+3) { rowSearchStrs.add(tokens.size() <= i + 3 ? "" : tokens.subList(i, tokens.size() >= i +6 ? i+6 : tokens.size()).stream().collect(Collectors.joining(" "))); } } var rowDocuments = rowSearchStrs.stream().filter(myStr -> !myStr.isBlank()) .flatMap(myStr -> this.documentVsRepository.retrieve(myStr, MetaData.DataType.ROW, searchDto.getResultAmount()).stream()) .toList(); return new EmbeddingContainer(tableDocuments, columnDocuments, rowDocuments); } First, documentVsRepository is used to retrieve the document with the embeddings for the tables/columns based on the search string of the user. Then, the search string is split into chunks of 6 words to search for the documents with the row embeddings. The row embeddings are just one word, and to get a low distance, the query string has to be short; otherwise, the distance grows due to all the other words in the query. Then the chunks are used to retrieve the row documents with the embeddings. Create the Prompt The prompt is created in the TableService class with the createPrompt method: Java private Prompt createPrompt(SearchDto searchDto, EmbeddingContainer documentContainer) { final Float minRowDistance = documentContainer.rowDocuments().stream() .map(myDoc -> (Float) myDoc.getMetadata().getOrDefault(MetaData.DISTANCE, 1.0f)).sorted().findFirst().orElse(1.0f); LOGGER.info("MinRowDistance: {}", minRowDistance); var sortedRowDocs = documentContainer.rowDocuments().stream() .sorted(this.compareDistance()).toList(); var tableColumnNames = this.createTableColumnNames(documentContainer); List<TableNameSchema> tableRecords = this.tableMetadataRepository .findByTableNameIn(tableColumnNames.tableNames()).stream() .map(tableMetaData -> new TableNameSchema(tableMetaData.getTableName(), tableMetaData.getTableDdl())).collect(Collectors.toList()); final AtomicReference<String> joinColumn = new AtomicReference<String>(""); final AtomicReference<String> joinTable = new AtomicReference<String>(""); final AtomicReference<String> columnValue = new AtomicReference<String>(""); sortedRowDocs.stream().filter(myDoc -> minRowDistance <= MAX_ROW_DISTANCE) .filter(myRowDoc -> tableRecords.stream().filter(myRecord -> myRecord.name().equals(myRowDoc.getMetadata() .get(MetaData.TABLE_NAME))).findFirst().isEmpty()) .findFirst().ifPresent(myRowDoc -> { joinTable.set(((String) myRowDoc.getMetadata() .get(MetaData.TABLE_NAME))); joinColumn.set(((String) myRowDoc.getMetadata() .get(MetaData.DATANAME))); tableColumnNames.columnNames().add(((String) myRowDoc.getMetadata() .get(MetaData.DATANAME))); columnValue.set(myRowDoc.getContent()); this.tableMetadataRepository.findByTableNameIn( List.of(((String) myRowDoc.getMetadata().get(MetaData.TABLE_NAME)))) .stream().map(myTableMetadata -> new TableNameSchema( myTableMetadata.getTableName(), myTableMetadata.getTableDdl())).findFirst() .ifPresent(myRecord -> tableRecords.add(myRecord)); }); var messages = createMessages(searchDto, minRowDistance, tableColumnNames, tableRecords, joinColumn, joinTable, columnValue); Prompt prompt = new Prompt(messages); return prompt; } First, the min distance of the rowDocuments is filtered out. Then a list row of documents sorted by distance is created. The method createTableColumnNames(...) creates the tableColumnNames record that contains a set of column names and a list of table names. The tableColumnNames record is created by first filtering for the 3 tables with the lowest distances. Then the columns of these tables with the lowest distances are filtered out. Then the tableRecords are created by mapping the table names to the schema DDL strings with the TableMetadataRepository. Then the sorted row documents are filtered for MAX_ROW_DISTANCE and the values joinColumn, joinTable, and columnValue are set. Then the TableMetadataRepository is used to create a TableNameSchema and add it to the tableRecords. Now the placeholders in systemPrompt and the optional columnMatch can be set: Java private final String systemPrompt = """ ... Include these columns in the query: {columns} \n Only use the following tables: {schemas};\n %s \n """; private final String columnMatch = """ Join this column: {joinColumn} of this table: {joinTable} where the column has this value: {columnValue}\n """; The method createMessages(...) gets the set of columns to replace the {columns} placeholder. It gets tableRecords to replace the {schemas} placeholder with the DDLs of the tables. If the row distance was beneath the threshold, the property columnMatch is added at the string placeholder %s. Then the placeholders {joinColumn}, {joinTable}, and {columnValue} are replaced. With the information about the required columns the schemas of the tables with the columns and the information of the optional join for row matches, the AI/LLM is able to create a sensible SQL query. Execute Query and Return Result The query is executed in the createQuery(...) method: Java public SqlRowSet searchTables(SearchDto searchDto) { EmbeddingContainer documentContainer = this.retrieveEmbeddings(searchDto); Prompt prompt = createPrompt(searchDto, documentContainer); String sqlQuery = createQuery(prompt); LOGGER.info("Sql query: {}", sqlQuery); SqlRowSet rowSet = this.jdbcTemplate.queryForRowSet(sqlQuery); return rowSet; } First, the methods to prepare the data and create the SQL query are called and then queryForRowSet(...) is used to execute the query on the database. The SqlRowSet is returned. The TableMapper class uses the map(...) method to turn the result into the TableSearchDto class: Java public TableSearchDto map(SqlRowSet rowSet, String question) { List<Map<String, String>> result = new ArrayList<>(); while (rowSet.next()) { final AtomicInteger atomicIndex = new AtomicInteger(1); Map<String, String> myRow = List.of(rowSet .getMetaData().getColumnNames()).stream() .map(myCol -> Map.entry( this.createPropertyName(myCol, rowSet, atomicIndex), Optional.ofNullable(rowSet.getObject( atomicIndex.get())) .map(myOb -> myOb.toString()).orElse(""))) .peek(x -> atomicIndex.set(atomicIndex.get() + 1)) .collect(Collectors.toMap(myEntry -> myEntry.getKey(), myEntry -> myEntry.getValue())); result.add(myRow); } return new TableSearchDto(question, result, 100); } First, the result list for the result maps is created. Then, rowSet is iterated for each row to create a map of the column names as keys and the column values as values. This enables returning a flexible amount of columns with their results. createPropertyName(...) adds the index integer to the map key to support duplicate key names. Summary Backend Spring AI supports creating prompts with a flexible amount of placeholders very well. Creating the embeddings and querying the vector table is also very well supported. Getting reasonable query results needs the metadata that has to be provided for the columns and tables. Creating good metadata is an effort that scales linearly with the amount of columns and tables. Implementing the embeddings for columns that need them is an additional effort. The result is that an AI/LLM like OpenAI or Ollama with the "sqlcoder:70b-alpha-q6_K" model can answer questions like: "Show the artwork name and the name of the museum that has the style Realism and the subject of Portraits." The AI/LLM can within boundaries answer natural language questions that have some fit with the metadata. The amount of embeddings needed is too big for a free OpenAI account and the "sqlcoder:70b-alpha-q6_K" is the smallest model with reasonable results. AI/LLM offers a new way to interact with relational databases. Before starting a project to provide a natural language interface for a database, the effort and the expected results have to be considered. The AI/LLM can help with questions of small to middle complexity and the user should have some knowledge about the database. Frontend The returned result of the backend is a list of maps with keys as column names and values column values. The amount of returned map entries is unknown, because of that the table to display the result has to support a flexible amount of columns. An example JSON result looks like this: JSON {"question":"...","resultList":[{"1_name":"Portrait of Margaret in Skating Costume","2_name":"Philadelphia Museum of Art"},{"1_name":"Portrait of Mary Adeline Williams","2_name":"Philadelphia Museum of Art"},{"1_name":"Portrait of a Little Girl","2_name":"Philadelphia Museum of Art"}],"resultAmount":100} The resultList property contains a JavaScript array of objects with property keys and values. To be able to display the column names and values in an Angular Material Table component, these properties are used: TypeScript protected columnData: Map<string, string>[] = []; protected columnNames = new Set<string>(); The method getColumnNames(...) of the table-search.component.ts is used to turn the JSON result in the properties: TypeScript private getColumnNames(tableSearch: TableSearch): Set<string> { const result = new Set<string>(); this.columnData = []; const myList = !tableSearch?.resultList ? [] : tableSearch.resultList; myList.forEach((value) => { const myMap = new Map<string, string>(); Object.entries(value).forEach((entry) => { result.add(entry[0]); myMap.set(entry[0], entry[1]); }); this.columnData.push(myMap); }); return result; } First, the result set is created and the columnData property is set to an empty array. Then, myList is created and iterated with forEach(...). For each of the objects in the resultList, a new Map is created. For each property of the object, a new entry is created with the property name as the key and the property value as the value. The entry is set on the columnData map and the property name is added to the result set. The completed map is pushed on the columnData array and the result is returned and set to the columnNames property. Then a set of column names is available in the columnNames set and a map with column name to column value is available in the columnData. The template table-search.component.html contains the material table: HTML @if(searchResult && searchResult.resultList?.length) { <table mat-table [dataSource]="columnData"> <ng-container *ngFor="let disCol of columnNames" matColumnDef="{{ disCol }"> <th mat-header-cell *matHeaderCellDef>{{ disCol }</th> <td mat-cell *matCellDef="let element">{{ element.get(disCol) }</td> </ng-container> <tr mat-header-row *matHeaderRowDef="columnNames"></tr> <tr mat-row *matRowDef="let row; columns: columnNames"></tr> </table> } First, the searchResult is checked for existence and objects in the resultList. Then, the table is created with the datasource of the columnData map. The table header row is set with <tr mat-header-row *matHeaderRowDef="columnNames"></tr> to contain the columnNames. The table rows and columns are defined with <tr mat-row *matRowDef="let row; columns: columnNames"></tr>. The cells are created by iterating the columnNames like this: <ng-container *ngFor="let disCol of columnNames" matColumnDef="{{ disCol }">. The header cells are created like this: <th mat-header-cell *matHeaderCellDef>{{ disCol }</th>. The table cells are created like this: <td mat-cell *matCellDef="let element">{{ element.get(disCol) }</td>. element is the map of the columnData array element and the map value is retrieved with element.get(disCol). Summary Frontend The new Angular syntax makes the templates more readable. The Angular Material table component is more flexible than expected and supports unknown numbers of columns very well. Conclusion To question a database with the help of an AI/LLM takes some effort for the metadata and a rough idea of the users what the database contains. AI/LLMs are not a natural fit for query creation because SQL queries require correctness. A pretty large model was needed to get the required query correctness, and GPU acceleration is required for productive use. A well-designed UI where the user can drag and drop the columns of the tables in the result table might be a good alternative for the requirements. Angular Material Components support drag and drop very well. Before starting such a project the customer should make an informed decision on what alternative fits the requirements best.
Ansible is one of the fastest-growing Infrastructure as Code (IaC) and automation tools in the world. Many of us use Ansible for Day 1 and Day 2 operations. One of the best analogies to understand the phases/stages/operations is defined on RedHat's website: "Imagine you're moving into a house. If Day 1 operations are moving into the house (installation), Day 2 operations are the 'housekeeping' stage of a software’s life cycle." Simply put, in a software lifecycle: Day 0: Design/planning phase - This phase involves preparation, initial planning, brainstorming, and preparing for the project. Typical activities in this phase are defining the scope, gathering requirements, assembling the development team, and setting up the development environments. For example, the team discusses the CI/CD platform to integrate the project with, the strategy for project management, etc. Day 1: Development/deployment phase - This phase marks the actual development activities such as coding, building features, and implementation based on the requirements gathered in the planning phase. Additionally, testing will begin to ensure early detection of issues (in development lingo, "bugs"). Day 2: Maintenance phase - This phase in which your project/software goes live and you keep a tap on the health of the project. You may need to patch or update the software and file feature requests/issues based on user feedback for your development team to work on. This is the phase where monitoring and logging (observability) play a crucial role. Ansible is an open-source tool written in Python and uses YAML to define the desired state of configuration. Ansible is used for configuration management, application deployment, and orchestration. It simplifies the process of managing and deploying software across multiple servers, making it one of the essential tools for system administrators, developers, and IT operations teams. With AI, generating Ansible code has become simpler and more efficient. Check out the following article to learn how Ansible is bringing AI tools to your Integrated Development Environment: "Automation, Ansible, AI." RedHat Ansible Lightspeed with IBM Watsonx code assistant At its core, Ansible employs a simple, agentless architecture, relying on SSH to connect to remote servers and execute tasks. This eliminates the need for installing any additional software or agents on target machines, resulting in a lightweight and efficient automation solution. Key Features of Ansible Here is a list of key features that Ansible offers: Infrastructure as Code (IaC) Ansible allows you to define your infrastructure and configuration requirements in code, enabling you to version control, share, and replicate environments with ease. For example, say you plan to move your on-premises application to a cloud platform. Instead of provisioning the cloud services and installing the dependencies manually, you can define the required cloud services and dependencies for your application like compute, storage, networking, security, etc., in a configuration file. That desired state is taken care of by Ansible as an Infrastructure as Code tool. In this way, setting up your development, test, staging, and production environments will easily avoid repetition. Playbooks Ansible playbooks are written in YAML format and define a series of tasks to be executed on remote hosts. Playbooks offer a clear, human-readable way to describe complex automation workflows. Using playbooks, you define the required dependencies and desired state for your application. Modules Ansible provides a vast collection of modules for managing various aspects of systems, networks, cloud services, and applications. Modules are idempotent, meaning they ensure that the desired state of the system is achieved regardless of its current state. For example, ansible.bultin.command is a module that helps you to execute commands on a remote machine. You can either use modules that are built in, like dnf, yum, etc., as part of Ansible Core, or you can develop your own modules in Ansible. To further understand the Ansible modules, check out this topic on RedHat. Inventory Management Ansible uses an inventory file to define the hosts it manages. This inventory can be static or dynamic, allowing for flexible configuration management across different environments. An inventory file (.ini or .yaml) is a list of hosts or nodes on which you install, configure, or set up a software, add a user, or change the permissions of a folder, etc. Refer to how to build an inventory for best practices. Roles Roles in Ansible provide a way to organize and reuse tasks, variables, and handlers. They promote code reusability and help maintain clean and modular playbooks. You can group tasks that are repetitive as a role to reuse or share with others. One good example is pinging a remote server, you can move the tasks, variables, etc., under a role to reuse. Below is an example of a role directory structure with eight main standard directories. You will learn about a tool to generate this defined structure in the next section of this article. Shell roles/ common/ # this hierarchy represents a "role" tasks/ # main.yml # <-- tasks file can include smaller files if warranted handlers/ # main.yml # <-- handlers file templates/ # <-- files for use with the template resource ntp.conf.j2 # <------- templates end in .j2 files/ # bar.txt # <-- files for use with the copy resource foo.sh # <-- script files for use with the script resource vars/ # main.yml # <-- variables associated with this role defaults/ # main.yml # <-- default lower priority variables for this role meta/ # main.yml # <-- role dependencies library/ # roles can also include custom modules module_utils/ # roles can also include custom module_utils lookup_plugins/ # or other types of plugins, like lookup in this case webtier/ # same kind of structure as "common" was above, done for the webtier role monitoring/ # "" fooapp/ Beyond Automation Ansible finds applications in several areas. Configuration management: Ansible simplifies the management of configuration files, packages, services, and users across diverse IT infrastructures. Application deployment: Ansible streamlines the deployment of applications by automating tasks such as software installation, configuration, and version control. Continuous Integration/Continuous Deployment (CI/CD): Ansible integrates seamlessly with CI/CD pipelines, enabling automated testing, deployment, and rollback of applications. Orchestration: Ansible orchestrates complex workflows involving multiple servers, networks, and cloud services, ensuring seamless coordination and execution of tasks. Security automation: Ansible helps enforce security policies, perform security audits, and automate compliance checks across IT environments. Cloud provisioning: Ansible's cloud modules facilitate the provisioning and management of cloud resources on platforms like IBM Cloud, AWS, Azure, Google Cloud, and OpenStack. The list is not exhaustive, so only a subset of applications is included above. Ansible can act as a security compliance manager by enforcing security policies and compliance standards across infrastructure and applications through patch management, configuration hardening, and vulnerability remediation. Additionally, Ansible can assist in setting up monitoring and logging, automating disaster recovery procedures (backup and restore processes, failovers, etc.,), and integrating with a wide range of tools and services, such as version control systems, issue trackers, ticketing systems, and configuration databases, to create end-to-end automation workflows. Tool and Project Ecosystem Ansible provides a wide range of tools and programs like Ansible-lint, Molecule for testing Ansible plays and roles, yamllint, etc. Here are additional tools that are not mentioned in the Ansible docs: Ansible Generator: Creates the necessary folder/directory structure; comes in handy when you create Ansible roles AWX: Provides a web-based user interface, REST API, and task engine built on top of Ansible; Comes with an awx-operator if you are planning to set up on a container orchestration platform like RedHat OpenShift Ansible VS Code extension by Red Hat: Syntax highlighting, validation, auto-completion, auto-closing Jinja expressions ("{{ my_variable }") etc. The Ansible ecosystem is very wide. This article gives you just a glimpse of the huge set of tools and frameworks. You can find the projects in the Ansible ecosystem on Ansible docs. Challenges With Ansible Every tool or product comes with its own challenges. Learning curve: One of the major challenges with Ansible is the learning curve. Mastering the features and best practices can be time-consuming, especially for users new to infrastructure automation or configuration. Complexity: Initially, understanding the terminology, folder structure, and hierarchy challenges the user. Terms like inventory, modules, plugins, tasks, playbooks, etc., are hard to understand in the beginning. As the number of nodes/hosts increases, the complexity of managing the playbooks, and orchestrating increases. Troubleshooting and error handling: For beginners, troubleshooting errors and debugging playbooks can be challenging. Especially, understanding error messages and identifying the root cause of failures requires familiarity with Ansible's syntax and modules, etc. Conclusion In this article, you learned that Ansible as an open-source tool can be used not only for automation but also for configuration, deployment, and security enablement. You also understood the features, and challenges and learned about the tools Ansible and the community offers. Ansible will become your go-to Infrastructure as Code tool once you pass the initial learning curve. To overcome the initial complexity, here's a GitHub repository with Ansible YAML code snippets to start with. Happy learning. If you like this article, please like and share it with your network.
In the age of digital transformation, businesses across the globe are increasingly relying on complex supply chain operations to streamline their processes, enhance productivity, and drive growth. However, as these supply chains become more interconnected and digitized, they also become more vulnerable to a myriad of cybersecurity threats. These threats can disrupt operations, compromise sensitive data, and ultimately, undermine business integrity and customer trust. The cybersecurity risks associated with supply chain operations are not just a concern for large corporations but also for small and medium-sized businesses. In fact, according to a report by the Ponemon Institute, 61% of U.S. companies experienced a data breach caused by a third-party vendor. This alarming statistic underscores the urgent need for businesses, developers, and cyber professionals to prioritize building resilient cybersecurity into their supply chain operations. This article aims to provide a comprehensive guide to understanding and addressing the unique cybersecurity challenges inherent in supply chain operations. By integrating cybersecurity measures into every facet of the supply chain, businesses can not only safeguard their operations and sensitive data but also gain a competitive edge in today's digital marketplace. We will explore the current state of supply chain cybersecurity, delve into the specific threats and challenges it presents, and present potential solutions and best practices. The goal is to equip businesses, developers, and cyber professionals with the knowledge and tools they need to fortify their supply chains against the ever-evolving landscape of cyber threats. Understanding Supply Chain Cybersecurity Supply chain cybersecurity is a critical aspect of risk management that focuses on protecting the supply chain from cyber threats. It involves securing all digital interactions and data exchanges that occur within the supply chain, from the initial sourcing of materials to the delivery of the final product to the customer. A supply chain is inherently complex, involving numerous entities such as suppliers, manufacturers, distributors, and retailers. Each of these entities represents a potential point of vulnerability that can be exploited by cybercriminals. Common types of cyber threats to supply chains include malware, phishing attacks, data breaches, and more sophisticated threats like Advanced Persistent Threats (APTs). One of the key challenges in supply chain cybersecurity is the interdependent nature of the supply chain. A single weak link in the chain can compromise the entire operation. For example, a cyberattack on a supplier could disrupt production, leading to delays, financial loss, and damage to the company's reputation. Moreover, the growing trend of digital transformation has led to an increase in the use of technologies such as Internet of Things (IoT) devices, cloud computing, and artificial intelligence in supply chain operations. While these technologies offer numerous benefits, they also increase the surface area for potential cyberattacks. Understanding the importance of supply chain cybersecurity and the unique threats it faces is the first step toward building a more secure and resilient supply chain. The next sections will delve deeper into the specific challenges of implementing cybersecurity in supply chain operations and discuss potential strategies and solutions. Challenges in Building Resilient Cybersecurity Into Supply Chain Operations Building resilient cybersecurity into supply chain operations presents a unique set of challenges due to the complex, interconnected nature of supply chains. These challenges can broadly be categorized into technical challenges, organizational challenges, and regulatory challenges. Technical Challenges The digital transformation of supply chains has led to the integration of various technologies such as IoT devices, cloud platforms, and AI-based systems. While these technologies have enhanced efficiency and productivity, they have also increased the complexity of the cybersecurity landscape. Ensuring the security of these diverse technologies, each with its own set of vulnerabilities, is a significant technical challenge. Organizational Challenges Supply chains involve multiple entities, including suppliers, manufacturers, distributors, and retailers. Each of these entities may have different cybersecurity protocols, making it difficult to implement consistent security measures across the entire supply chain. Additionally, there is often a lack of awareness and understanding of cybersecurity risks among these entities, particularly small and medium-sized businesses. Regulatory Challenges The regulatory environment for cybersecurity is rapidly evolving, with different countries and regions implementing their own set of rules and standards. Navigating this complex regulatory landscape and ensuring compliance can be a challenge, especially for global supply chains. Resource Constraints Many organizations, particularly small and medium-sized businesses, lack the resources necessary to implement robust cybersecurity measures. This includes financial resources, as well as human resources such as skilled cybersecurity professionals. Evolving Cyber Threats The nature of cyber threats is continually evolving, with cybercriminals employing increasingly sophisticated techniques. Keeping up with these threats and ensuring that cybersecurity measures are up-to-date is a constant challenge. Strategies for Building Resilient Cybersecurity Into Supply Chain Operations The rise of interconnected Internet of Things (IoT) devices and Industrial Control Systems (ICS) within supply chains has significantly expanded the attack surface for cyber adversaries. Vulnerabilities in software, hardware, or human behavior can be exploited to disrupt operations, steal intellectual property, or compromise critical infrastructure. To mitigate these risks and build resilient cybersecurity within supply chains, developers and security professionals must adopt a multi-layered, technically-focused approach. 1. Threat Intelligence Integration Proactive threat intelligence gathering and analysis are crucial in today's cyber landscape. Integrating threat intelligence feeds specific to the supply chain industry allows developers and security professionals to: Identify emerging threats: Identify and prioritize emerging threats before they are weaponized. This provides valuable time to develop patches, update security configurations, and implement mitigation strategies. Focus vulnerability assessments: Focus vulnerability assessments on the most relevant threats facing the supply chain. This ensures resources are allocated efficiently and critical vulnerabilities are addressed promptly. 2. Secure Coding Practices and SDLC Integration Building security into software from the outset is paramount. Here are key strategies for developers: Secure coding training: Implement mandatory secure coding training programs for developers. These programs should cover secure coding practices, common vulnerabilities, and coding standards specific to the supply chain industry. Static code analysis tools: Utilize static code analysis tools to identify potential vulnerabilities within code early in the development lifecycle. This allows for early remediation and reduces the risk of vulnerabilities being introduced into production systems. Secure Software Development Lifecycles (SDLCs): Integrate security considerations throughout the entire SDLC. This includes security requirements gathering, threat modeling, code reviews, and penetration testing to ensure the final product is secure and resilient. 3. Zero Trust Security Model Implementation Zero Trust security models assume no inherent trust within the network. This principle should be applied to all aspects of the supply chain: Least Privilege Access Control: Implement the principle of least privilege for all users, devices, and applications within the supply chain network. Grant access only to the minimum resources required for users to perform their designated tasks. Multi-Factor Authentication (MFA): Enforce strong authentication protocols, including multi-factor authentication (MFA), for all access attempts across the entire supply chain ecosystem. Continuous monitoring and microsegmentation: Implement continuous monitoring of network activity and system logs to detect suspicious behavior. Consider network segmentation and micro-segmentation strategies to limit the potential impact of a successful cyberattack. 4. Data Encryption in Transit and at Rest Data security is paramount within the supply chain. To ensure the confidentiality and integrity of sensitive data: Data encryption in transit: Encrypt all data in transit between systems and devices within the supply chain. This protects sensitive information from interception during network communication. Data encryption at rest: Encrypt all sensitive data at rest on storage devices and databases throughout the supply chain. This ensures that even if an attacker gains access to storage systems, the data will be unreadable. 5. Continuous Vulnerability Management Security vulnerabilities are constantly being discovered and exploited. A comprehensive vulnerability management program should be implemented: Vulnerability scanning and patch management: Regularly conduct vulnerability scans across all IT and ICS systems within the supply chain. Prioritize patching critical vulnerabilities identified during scans to minimize the window of exploitation. Penetration testing: Conduct regular penetration testing to identify exploitable weaknesses in security controls and configurations. This proactive approach simulates real-world attacks, helping to uncover vulnerabilities that may be missed by automated scans. 6. Secure Configuration Management Maintaining secure configurations of all systems across the supply chain is essential. This includes: Automated configuration management tools: Implement automated configuration management tools to ensure consistent and secure configurations across all devices and systems within the supply chain. Configuration baselines and change management: Establish security baselines for all system configurations and implement a robust change management process to track and review any modifications. 7. Security Awareness Training Human error is often a significant factor in successful cyberattacks. Ongoing security awareness training for all stakeholders within the supply chain is crucial: Educate employees on recognizing phishing scams and social engineering tactics commonly used by cybercriminals. Emphasize the importance of verifying sender legitimacy and avoiding suspicious links or attachments in emails. Secure coding practices: For developers, security awareness training should cover secure coding practices, common vulnerabilities in supply chain software, and the importance of secure coding throughout the SDLC. Supply chain-specific threats: Train all employees on the specific cyber threats relevant to the supply chain industry. This includes understanding the risks associated with IoT devices, ICS vulnerabilities, and data security best practices within the supply chain ecosystem. 8. Vendor Risk Management Building a secure supply chain requires extending security considerations beyond your organization's internal systems. Vendor Risk Management (VRM) is a critical practice for identifying and mitigating cybersecurity risks posed by third-party vendors throughout the supply chain ecosystem. VRM Best Practices Vendor assessment: Conduct thorough assessments of the cybersecurity posture of potential and existing vendors. This assessment should evaluate the vendor's: Security controls and incident response plans Patch management practices to ensure timely vulnerability remediation Data security measures like encryption and access controls Compliance with relevant security regulations (e.g., PCI DSS, HIPAA) Contractual security considerations: Integrate security expectations and accountability clauses within vendor contracts. This ensures clarity on: The vendor's responsibility for maintaining secure systems and data handling practices Reporting requirements for security incidents or vulnerabilities The right to conduct security audits of the vendor's systems Case Studies To illustrate the importance of building resilient cybersecurity into supply chain operations and how it can be achieved, let's consider two case studies: Case Study 1: Building Cybersecurity Resilience in a Global Pharmaceutical Supply Chain Company Acme Pharmaceuticals, a multinational pharmaceutical company with a complex global supply chain network Challenge Acme faced increasing concerns about cybersecurity threats targeting their supply chain. These threats included potential attacks on: Manufacturing facilities of third-party vendors Logistics and transportation systems used to deliver critical materials and finished products Intellectual property theft of proprietary drug formulas Strategies Implemented Vendor Risk Management: Acme implemented a rigorous VRM program. They assessed the cybersecurity posture of all major vendors, including raw material suppliers, contract manufacturers, and logistics providers. Security controls, data security practices, and incident response plans were evaluated. Contracts were updated to include security expectations and reporting requirements for vulnerabilities or breaches. Threat intelligence integration: Acme subscribed to a threat intelligence feed specializing in the pharmaceutical industry. This feed provided insights into emerging cyber threats targeting the healthcare sector. The intelligence was used to prioritize vendor assessments and identify potential weaknesses in their own security posture. Secure coding practices: Acme partnered with key vendors to promote secure coding practices within their software development lifecycles. This included training for vendor developers on secure coding principles and code review processes to identify and eliminate vulnerabilities. Data encryption in transit and at rest: Acme implemented data encryption for all sensitive data throughout the supply chain. This included encrypting data during transportation between facilities and at rest on storage devices and databases. Continuous monitoring and microsegmentation: Acme implemented continuous monitoring of their network and vendor systems. Network segmentation and micro-segmentation strategies were employed to limit the potential impact of a successful cyberattack. Results By implementing these strategies, Acme significantly improved the cybersecurity resilience of their supply chain. Vendor assessments identified and mitigated potential security risks. Threat intelligence provided early warnings of emerging threats. Secure coding practices within the vendor network reduced the likelihood of software vulnerabilities. Data encryption protected sensitive information, and continuous monitoring allowed for the rapid detection and response to suspicious activity. Case Study 2: Security of a Just-In-Time (JIT) Supply Chain for a Tech Startup Company NovaTech, a fast-growing tech startup that relies on a Just-in-Time (JIT) inventory management system for their electronics manufacturing Challenge NovaTech's JIT system minimized inventory storage costs but also increased reliance on a network of interconnected suppliers and manufacturers. This complex ecosystem presented a larger attack surface for potential cyberattacks. Security concerns included: Disruptions to production caused by cyberattacks on supplier IT systems Theft of intellectual property related to NovaTech's hardware designs Ransomware attacks on critical manufacturing equipment within the supply chain Strategies Implemented Zero Trust security model: NovaTech implemented a Zero Trust security model across their entire supply chain. This model assumed no inherent trust within the network and required continuous verification for all users, devices, and applications attempting to access resources. Secure configuration management: Automated configuration management tools were implemented to ensure consistent and secure configurations across all devices and systems within the supply chain. This included routers, switches, and manufacturing equipment used by NovaTech and their vendors. Security awareness training: NovaTech conducted comprehensive security awareness training programs for their employees and partnered with vendors to offer similar training for their workforce. This training emphasized best practices for secure password management, phishing email identification, and reporting suspicious activity. Penetration testing: NovaTech conducted regular penetration testing of their own systems and, when possible, collaborated with key vendors to conduct penetration testing of their critical infrastructure. This proactive approach helped identify and address potential vulnerabilities before they could be exploited by cybercriminals. Cybersecurity incident response plan: A comprehensive incident response plan was developed and tested to ensure a coordinated and rapid response in the event of a cyberattack. The plan outlined roles and responsibilities for NovaTech and their vendors during a security incident. Results NovaTech's commitment to cybersecurity throughout their JIT supply chain significantly reduced their risk of cyberattacks. The Zero Trust model ensured that only authorized users and devices could access critical resources. Secure configuration management minimized the risk of misconfigured systems creating vulnerabilities. Security awareness training empowered employees and vendors to identify and report suspicious activity. Penetration testing identified and addressed potential weaknesses in security posture. A well-defined incident response plan ensured a swift and coordinated response to security incidents. Conclusion In conclusion, building a resilient cybersecurity system within the supply chain is a continuous, collaborative effort that involves all stakeholders. The importance of proactive threat intelligence gathering and analysis cannot be overstated, as it provides crucial insights for prioritizing security measures. Additionally, extending security considerations to include Vendor Risk Management (VRM) and adopting a Zero Trust security model are key strategies for defending against evolving cyber threats, particularly in complex and interconnected systems like Just-In-Time (JIT) supply chains. Secure configuration management also plays a vital role in maintaining a consistent security posture. Ultimately, the commitment to continuous monitoring, layered security, and active participation from all stakeholders is what will safeguard an organization's operations, data, and reputation in the digital marketplace.
In the dynamic landscape of modern technology, the realm of Incident Management stands as a crucible where professionals are tested and refined. Incidents, ranging from minor hiccups to critical system failures, are not mere disruptions but opportunities for growth and learning. Within this crucible, we have traversed the challenging terrain of Incident Management. The collective experiences and insights offer a treasure trove of wisdom, illuminating the path for personal and professional development. In this article, we delve deep into the core principles and lessons distilled from the crucible of Incident Management. Beyond the technical intricacies lies a tapestry of skills and virtues—adaptability, resilience, effective communication, collaborative teamwork, astute problem-solving, and a relentless pursuit of improvement. These are the pillars upon which successful incident response is built, shaping not just careers but entire mindsets and approaches to life's challenges. Through real-world anecdotes and practical wisdom, we unravel the transformative power of Incident Management. Join us on this journey of discovery, where each incident is not just a problem to solve but a stepping stone towards personal and professional excellence. Incident Management Essentials: Navigating Through Challenges Incident Management is a multifaceted discipline that requires a strategic approach and a robust set of skills to navigate through various challenges effectively. At its core, Incident Management revolves around the swift and efficient resolution of unexpected issues that can disrupt services, applications, or systems. One of the fundamental aspects of Incident Management is the ability to prioritize incidents based on their impact and severity. This involves categorizing incidents into different levels of urgency and criticality, akin to triaging patients in a hospital emergency room. By prioritizing incidents appropriately, teams can allocate resources efficiently, focus efforts where they are most needed, and minimize the overall impact on operations and user experience. Clear communication channels are another critical component of Incident Management. Effective communication ensures that all stakeholders, including technical teams, management, customers, and other relevant parties, are kept informed throughout the incident lifecycle. Transparent and timely communication not only fosters collaboration but also instills confidence in stakeholders that the situation is being addressed proactively. Collaboration and coordination are key pillars of successful incident response. Incident Management often involves cross-functional teams working together to diagnose, troubleshoot, and resolve issues. Collaboration fosters collective problem-solving, encourages knowledge sharing, and enables faster resolution times. Additionally, establishing well-defined roles, responsibilities, and escalation paths ensures a streamlined and efficient response process. Proactive monitoring and alerting systems play a crucial role in Incident Management. Early detection of anomalies, performance issues, or potential failures allows teams to intervene swiftly before they escalate into full-blown incidents. Implementing robust monitoring tools, setting up proactive alerts, and conducting regular health checks are essential proactive measures to prevent incidents or mitigate their impact. Furthermore, incident documentation and post-mortem analysis are integral parts of Incident Management. Documenting incident details, actions taken, resolutions, and lessons learned not only provides a historical record but also facilitates continuous improvement. Post-incident analysis involves conducting a thorough root cause analysis, identifying contributing factors, and implementing corrective measures to prevent similar incidents in the future. In essence, navigating through challenges in Incident Management requires a blend of technical expertise, strategic thinking, effective communication, collaboration, proactive monitoring, and a culture of continuous improvement. By mastering these essentials, organizations can enhance their incident response capabilities, minimize downtime, and deliver superior customer experiences. Learning from Challenges: The Post-Incident Analysis The post-incident analysis phase is a critical component of Incident Management that goes beyond resolving the immediate issue. It serves as a valuable opportunity for organizations to extract meaningful insights, drive continuous improvement, and enhance resilience against future incidents. Here are several key points to consider during the post-incident analysis: Root Cause Analysis (RCA) Conducting a thorough RCA is essential to identify the underlying factors contributing to the incident. This involves tracing back the chain of events, analyzing system logs, reviewing configurations, and examining code changes to pinpoint the root cause accurately. RCA helps in addressing the core issues rather than just addressing symptoms, thereby preventing recurrence. Lessons Learned Documentation Documenting lessons learned from each incident is crucial for knowledge management and organizational learning. Capture insights, observations, and best practices discovered during the incident response process. This documentation serves as a valuable resource for training new team members, refining incident response procedures, and avoiding similar pitfalls in the future. Process Improvement Recommendations Use the findings from post-incident analysis to recommend process improvements and optimizations. This could include streamlining communication channels, revising incident response playbooks, enhancing monitoring and alerting thresholds, automating repetitive tasks, or implementing additional failover mechanisms. Continuous process refinement ensures a more effective and efficient incident response framework. Cross-Functional Collaboration Involve stakeholders from various departments, including technical teams, management, quality assurance, and customer support, in the post-incident analysis discussions. Encourage open dialogue, share insights, and solicit feedback from diverse perspectives. Collaborative analysis fosters a holistic understanding of incidents and promotes collective ownership of incident resolution and prevention efforts. Implementing Corrective and Preventive Actions (CAPA) Based on the findings of the post-incident analysis, prioritize and implement corrective actions to address immediate vulnerabilities or gaps identified. Additionally, develop preventive measures to mitigate similar risks in the future. CAPA initiatives may include infrastructure upgrades, software patches, security enhancements, or policy revisions aimed at strengthening resilience and reducing incident frequency. Continuous Monitoring and Feedback Loop Establish a continuous monitoring mechanism to track the effectiveness of implemented CAPA initiatives. Monitor key metrics such as incident recurrence rates, mean time to resolution (MTTR), customer satisfaction scores, and overall system stability. Solicit feedback from stakeholders and iterate on improvements iteratively to refine incident response capabilities over time. By embracing a comprehensive approach to post-incident analysis, organizations can transform setbacks into opportunities for growth, innovation, and enhanced operational excellence. The insights gleaned from each incident serve as stepping stones towards building a more resilient and proactive incident management framework. Enhancing Post-Incident Analysis With AI The integration of Artificial Intelligence is revolutionizing Post-Incident Analysis, offering advanced capabilities that significantly augment traditional approaches. Here's how AI can elevate the PIA process: Pattern Recognition and Incident Detection AI algorithms excel in analyzing extensive historical data to identify patterns indicative of potential incidents. By detecting anomalies in system behavior or recognizing error patterns in logs, AI efficiently flags potential incidents for further investigation. This automated incident detection streamlines identification efforts, reducing manual workload and response times. Advanced Root Cause Analysis (RCA) AI algorithms are adept at processing complex data sets and correlating multiple variables. In RCA, AI plays a pivotal role in pinpointing the root cause of incidents by analyzing historical incident data, system logs, configuration changes, and performance metrics. This in-depth analysis facilitated by AI accelerates the identification of underlying issues, leading to more effective resolutions and preventive measures. Predictive Analysis and Proactive Measures Leveraging historical incident data and trends, AI-driven predictive analysis forecasts potential issues or vulnerabilities. By identifying emerging patterns or risk factors, AI enables proactive measures to mitigate risks before they escalate into incidents. This proactive stance not only reduces incident frequency and severity but also enhances overall system reliability and stability. Continuous Improvement via AI Insights AI algorithms derive actionable insights from post-incident analysis data. By evaluating the effectiveness of implemented corrective and preventive actions (CAPA), AI offers valuable feedback on intervention impact. These insights drive ongoing process enhancements, empowering organizations to refine incident response strategies, optimize resource allocation, and continuously enhance incident management capabilities. Integrating AI into Post-Incident Analysis empowers organizations with data-driven insights, automation of repetitive tasks, and proactive risk mitigation, fostering a culture of continuous improvement and resilience in Incident Management. Applying Lessons Beyond Work: Personal Growth and Resilience The skills and lessons gained from Incident Management are highly transferable to various aspects of life. For instance, adaptability is crucial not only in responding to technical issues but also in adapting to changes in personal circumstances or professional environments. Teamwork teaches collaboration, conflict resolution, and empathy, which are essential in building strong relationships both at work and in personal life. Problem-solving skills honed during incident response can be applied to tackle challenges in any domain, from planning a project to resolving conflicts. Resilience, the ability to bounce back from setbacks, is a valuable trait that helps individuals navigate through adversity with determination and a positive mindset. Continuous improvement is a mindset that encourages individuals to seek feedback, reflect on experiences, identify areas for growth, and strive for excellence. This attitude of continuous learning and development not only benefits individuals in their careers but also contributes to personal fulfillment and satisfaction. Dispelling Misconceptions: What Lessons Learned Isn't We highlight common misconceptions about lessons learned, clarifying that it's not about: Emergency mindset: Lessons learned don't advocate for a perpetual emergency mindset but emphasize preparedness and maintaining a healthy, sustainable pace in incident response and everyday operations. Assuming all situations are crises: It's essential to discern between true emergencies and everyday challenges, avoiding unnecessary stress and overreaction to non-critical issues. Overemphasis on structure and protocol: While structure and protocols are important, rigid adherence can stifle flexibility and outside-the-box thinking. Lessons learned encourage a balance between following established procedures and embracing innovation. Decisiveness at the expense of deliberation: Rapid decision-making is crucial during incidents, but rushing decisions can lead to regrettable outcomes. It's about finding the right balance between acting swiftly and ensuring thorough deliberation to avoid hasty or ill-informed decisions. Short-term focus: Lessons learned extend beyond immediate goals and short-term fixes. It promotes a long-term perspective, strategic planning, and continuous improvement to address underlying issues and prevent recurring incidents. Minimizing risk to the point of stagnation: While risk mitigation is important, excessive risk aversion can lead to missed opportunities for growth and innovation. Lessons learned encourage a proactive approach to risk management that balances security with strategic decision-making. One-size-fits-all approach: Responses to incidents and lessons learned should be tailored to the specific circumstances and individuals involved. Avoiding a one-size-fits-all approach ensures that solutions are effective, relevant, and scalable across diverse scenarios. Embracing Growth: Conclusion In conclusion, Incident Management is more than just a set of technical processes or procedures. It's a mindset, a culture, and a journey of continuous growth and improvement. By embracing the core principles of adaptability, communication, teamwork, problem-solving, resilience, and continuous improvement, individuals can not only excel in their professional roles but also lead more fulfilling and meaningful lives.
The Data Story At the core of every software application, from the simplest to the most complex, operating at scale to serve millions of users with low-latency requests, lies a foundational element: data. For over three decades, relational database management systems (RDBMS) have been at the forefront of this domain. These systems, from simply storing data in a table format consisting of rows for records and columns for attributes, have undergone significant advancements and innovations that have revolutionized structured data and semi-unstructured storage. Relational database models have established themselves as the foundation of structured data handling, are renowned for their reliability, and battle-tested their efficacy in supporting massive big data scales for enterprise applications. However, as we evolve deeper into the era of big data and artificial intelligence (AI), the limitations of traditional RDBMS in handling unstructured data, such as images, videos, audio, and natural language have become increasingly apparent. Enter the vector database, a cutting-edge innovation tailored for the age of AI and significantly change the recommendation systems. Unlike RDBMS, which excels in managing structured data, vector databases are designed to handle and query high-dimensional vector embeddings, a form of unstructured data representation that is central to modern machine learning algorithms. Introduction: Vector DB Vector embeddings allow complex data like text, images, and sounds to be transformed into numerical vectors, capturing the essence of the data in a way that machines can process. This transformation is crucial for tasks such as similarity search, recommendation systems, and natural language processing, where understanding the nuanced relationships between data points is key. Vector databases leverage specialized indexing and search algorithms to efficiently query these embeddings, enabling applications that were previously challenging or impossible with traditional RDBMS. Fundamental Difference of RDBMS and Vectors The application interacts with the database by executing various transactions and actions, which are stored in the form of rows and columns. When it comes to the vector database, the action might look a bit different. Below, you can see the different types of files, which will be read and processed by many types of AI models and create vector embeddings. Example in Action Consider the process of transforming a comprehensive movie database, such as IMDB, into a format where each movie is represented by vector embeddings and stored in a vector database. This transformation allows the database to leverage the power of vector embeddings to significantly enhance the user search experience. Because these vectors are organized within a three-dimensional space, search engineers can more efficiently perform queries across the movie database. This spatial organization not only streamlines the retrieval process but also enables the implementation of sophisticated search functionalities, such as finding movies with similar themes or genres, thereby creating a more intuitive and responsive search experience for users. Now, we will demonstrate in Python how to convert textual movie data, similar to the tables mentioned above, into vector representations using BERT (Bidirectional Encoder Representations from Transformers), a pre-trained deep learning model developed by Google. This process entails several crucial steps for transforming the text into a format that the model can process, followed by the extraction of meaningful embeddings. Let's break down each step. Step 1 Python #Import Libraries import sqlite3 from transformers import BertTokenizer, BertModel import torch sqlite3: This imports the SQLite3 library, which allows Python to interact with SQLite databases. It's used here to access a database containing IMDB movie information. from transformers import BertTokenizer, BertModel: These imports from the Hugging Face transformers library bring in the necessary tools to tokenize text data (BertTokenizer) and to load the pre-trained BERT model (BertModel) for generating vector embeddings. import torch: This imports PyTorch, a deep learning framework that BERT and many other models in the transformers library are built on. It's used for managing tensors, which are multi-dimensional arrays that serve as the basic building blocks of data for neural networks. Step 2 Python #Initialize Tokenizer and Model tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertModel.from_pretrained('bert-base-uncased') tokenizer: This initializes the BERT tokenizer, configuring it to split input text into tokens that the BERT model can understand. The from_pretrained('bert-base-uncased') method loads a tokenizer trained in lowercase English text. model: This initializes the BERT model itself, also using the from_pretrained method to load a version trained in lowercase English. This model is what will generate the embeddings from the tokenized text. Step 3 Python # Connect to Database and Fetch Movie Data conn = sqlite3.connect('path/to/your/movie_database.db') cursor = conn.cursor() cursor.execute("SELECT name, genre, release_date, length FROM movies") movies = cursor.fetchall() conn = sqlite3.connect('path/to/your/movie_database.db'): Opens a connection to an SQLite database file that contains your movie data cursor = conn.cursor(): Creates a cursor object which is used to execute SQL commands through the connection cursor.execute(...): Executes an SQL command to select specific columns (name, genre, release date, length) from the movies table movies = cursor.fetchall(): Retrieves all the rows returned by the SQL query and stores them in the variable movies Step 4 Python #Convert Movie Data to Vector Embeddings movie_vectors = [] for movie in movies: movie_data = ', '.join(str(field) for field in movie) inputs = tokenizer(movie_data, return_tensors="pt", padding=True, truncation=True, max_length=512) with torch.no_grad(): outputs = model(**inputs) movie_vector = outputs.last_hidden_state[:, 0, :].numpy() movie_vectors.append(movie_vector) movie_vectors = []: Initializes an empty list to store the vector embeddings for each movie For loop: Iterates over each movie retrieved from the database movie_data = ', '.join(...): Concatenates the movie's details into a single string inputs = tokenizer(...): Uses the BERT tokenizer to prepare the concatenated string for the model, converting it into a tensor with torch.no_grad():: Temporarily disables gradient computation, which is unnecessary during inference (model.predict) outputs = model(**inputs): Feeds the tokenized input to the BERT model to get the embeddings movie_vector = ...: Extracts the embedding of the [CLS] token, which represents the entire input sequence movie_vectors.append(movie_vector): Adds the movie's vector embedding to the list Output movie_vectors: At the end of this script, you have a list of vector embeddings, one for each movie in your database. These vectors encapsulate the semantic information of the movies' names, genres, release dates, and durations in a form that machine learning models can work with. Conclusion In our example of vector database, movies such as "Inception" and "The Matrix" known for their action-packed, thought-provoking narratives, or "La La Land" and "Eternal Sunshine of the Spotless Mind," which explore complex romantic themes are transformed into high-dimensional vectors using BERT, a deep learning model. These vectors capture not just the overt categories like genre or release year, but also subtler thematic and emotional nuances encoded in their descriptions. Once stored in a vector database, these embeddings can be queried efficiently to perform similarity searches. When a user searches for a film with a particular vibe or thematic element, the streaming service can quickly identify and suggest films that are "near" the user's interests in the vector space, even if the user's search terms don't directly match the movie's title, genre, or other metadata. For instance, a search for "dream manipulation movies" might not only return "Inception" but also suggest "The Matrix," given their thematic similarities represented in the vector space. This method of storage and retrieval significantly enriches the user experience on streaming platforms, facilitating a discovery process that aligns content with both the user's interests and current mood. It’s designed to lead to "aha moments," where users uncover hidden gems, especially valuable when navigating the vast catalogs and offerings of streaming services. By detailing the creation and application of vector embeddings from textual movie data, we demonstrate the significant use of machine learning and vector databases in revolutionizing search capabilities and elevating the user experience in digital content ecosystems, particularly within streaming video services.
Unit testing is an essential practice in software development that involves testing individual codebase components to ensure they function correctly. In Spring-based applications, developers often use Aspect-Oriented Programming (AOP) to separate cross-cutting concerns, such as logging, from the core business logic, thus enabling modularization and cleaner code. However, testing aspects in Spring AOP pose unique challenges due to their interception-based nature. Developers need to employ appropriate strategies and best practices to facilitate effective unit testing of Spring AOP aspects. This comprehensive guide aims to provide developers with detailed and practical insights on effectively unit testing Spring AOP aspects. The guide covers various topics, including the basics of AOP, testing the pointcut expressions, testing around advice, testing before and after advice, testing after returning advice, testing after throwing advice, and testing introduction advice. Moreover, the guide provides sample Java code for each topic to help developers understand how to effectively apply the strategies and best practices. By following the guide's recommendations, developers can improve the quality of their Spring-based applications and ensure that their code is robust, reliable, and maintainable. Understanding Spring AOP Before implementing effective unit testing strategies, it is important to have a comprehensive understanding of Spring AOP. AOP, or Aspect-Oriented Programming, is a programming paradigm that enables the separation of cross-cutting concerns shared across different modules in an application. Spring AOP is a widely used aspect-oriented framework that is primarily implemented using runtime proxy-based mechanisms. The primary objective of Spring AOP is to provide modularity and flexibility in designing and implementing cross-cutting concerns in a Java-based application. The key concepts that one must understand in Spring AOP include: Aspect: An aspect is a module that encapsulates cross-cutting concerns that are applied across multiple objects in an application. Aspects are defined using aspects-oriented programming techniques and are typically independent of the application's core business logic. Join point: A join point is a point in the application's execution where the aspect can be applied. In Spring AOP, a join point can be a method execution, an exception handler, or a field access. Advice: Advice is an action that is taken when a join point is reached during the application's execution. In Spring AOP, advice can be applied before, after, or around a join point. Pointcut: A pointcut is a set of joint points where an aspect's advice should be applied. In Spring AOP, pointcuts are defined using expressions that specify the join points based on method signatures, annotations, or other criteria. By understanding these key concepts, developers can effectively design and implement cross-cutting concerns in a Java-based application using Spring AOP. Challenges in Testing Spring AOP Aspects Unit testing Spring AOP aspects can be challenging compared to testing regular Java classes, due to the unique nature of AOP aspects. Some of the key challenges include: Interception-based behavior: AOP aspects intercept method invocations or join points, which makes it difficult to test their behavior in isolation. To overcome this challenge, it is recommended to use mock objects to simulate the behavior of the intercepted objects. Dependency Injection: AOP aspects may rely on dependencies injected by the Spring container, which requires special handling during testing. It is important to ensure that these dependencies are properly mocked or stubbed to ensure that the aspect is being tested in isolation and not affected by other components. Dynamic proxying: Spring AOP relies on dynamic proxies, which makes it difficult to directly instantiate and test aspects. To overcome this challenge, it is recommended to use Spring's built-in support for creating and configuring dynamic proxies. Complex pointcut expressions: Pointcut expressions can be complex, making it challenging to ensure that advice is applied to the correct join points. To overcome this challenge, it is recommended to use a combination of unit tests and integration tests to ensure that the aspect is being applied correctly. Transaction management: AOP aspects may interact with transaction management, introducing additional complexity in testing. To overcome this challenge, it is recommended to use a combination of mock objects and integration tests to ensure that the aspect is working correctly within the context of the application. Despite these challenges, effective unit testing of Spring AOP aspects is crucial for ensuring the reliability, maintainability, and correctness of the application. By understanding these challenges and using the recommended testing approaches, developers can ensure that their AOP aspects are thoroughly tested and working as intended. Strategies for Unit Testing Spring AOP Aspects Unit testing Spring AOP Aspects can be challenging, given the system's complexity and the multiple pieces of advice involved. However, developers can use various strategies and best practices to overcome these challenges and ensure effective unit testing. One of the most crucial strategies is to isolate aspects from dependencies when writing unit tests. This isolation ensures that the tests focus solely on the aspect's behavior without interference from other modules. Developers can accomplish this by using mocking frameworks such as Mockito, EasyMock, or PowerMockito, which allow them to simulate dependencies' behavior and control the test environment. Another best practice is to test each piece of advice separately. AOP Aspects typically consist of multiple pieces of advice, such as "before," "after," or "around" advice. Testing each piece of advice separately ensures that the behavior of each piece of advice is correct and that it functions correctly in isolation. It's also essential to verify that the pointcut expressions are correctly configured and target the intended join points. Writing tests that exercise different scenarios helps ensure the correctness of point-cut expressions. Aspects in Spring-based applications often rely on beans managed by the ApplicationContext. Mocking the ApplicationContext allows developers to provide controlled dependencies to the aspect during testing, avoiding the need for a fully initialized Spring context. Developers should also define clear expectations for the behavior of the aspect and use assertions to verify that the aspect behaves as expected under different conditions. Assertions help ensure that the aspect's behavior aligns with the intended functionality. Finally, if aspects involve transaction management, developers should consider testing transactional behavior separately. This can be accomplished by mocking transaction managers or using in-memory databases to isolate the transactional aspect of the code for testing. By employing these strategies and best practices, developers can ensure effective unit testing of Spring AOP Aspects, resulting in robust and reliable systems. Sample Code: Testing a Logging Aspect To gain a better understanding of testing Spring AOP aspects, let's take a closer look at the sample code. We will analyze the testing process step-by-step, emphasizing important factors to take into account, and providing further information to ensure clarity. Let's assume that we will be writing unit tests for the following main class: Java import org.aspectj.lang.JoinPoint; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Before; import org.springframework.stereotype.Component; @Aspect @Component public class LoggingAspect { @Before("execution(* com.example.service.*.*(..))") public void logBefore(JoinPoint joinPoint) { System.out.println("Logging before " + joinPoint.getSignature().getName()); } } The LoggingAspect class logs method executions with a single advice method, logBefore, which executes before methods in the com.example.service package. The LoggingAspectTest class contains unit tests for the LoggingAspect. Let's examine each part of the test method testLogBefore() in detail: Java import org.aspectj.lang.JoinPoint; import org.aspectj.lang.Signature; import org.junit.jupiter.api.Test; import static org.mockito.Mockito.*; public class LoggingAspectTest { @Test void testLogBefore() { // Given LoggingAspect loggingAspect = new LoggingAspect(); // Creating mock objects JoinPoint joinPoint = mock(JoinPoint.class); Signature signature = mock(Signature.class); // Configuring mock behavior when(joinPoint.getSignature()).thenReturn(signature); when(signature.getName()).thenReturn("methodName"); // When loggingAspect.logBefore(joinPoint); // Then // Verifying interactions with mock objects verify(joinPoint, times(1)).getSignature(); verify(signature, times(1)).getName(); // Additional assertions can be added to ensure correct logging behavior } } In the above code, there are several sections that play a vital role in testing. Firstly, the Given section sets up the test scenario. We do this by creating an instance of the LoggingAspect and mocking the JoinPoint and Signature objects. By doing so, we can control the behavior of these objects during testing. Next, we create mock objects for the JoinPoint and Signature using the Mockito mocking framework. This allows us to simulate behavior without invoking real instances, providing a controlled environment for testing. We then use Mockito's when() method to specify the behavior of the mock objects. For example, we define that when thegetSignature() method of the JoinPoint is called, it should return the mock Signature object we created earlier. In the When section, we invoke the logBefore() method of the LoggingAspect with the mocked JoinPoint. This simulates the execution of the advice before a method call, which triggers the logging behavior. Finally, we use Mockito's verify() method to assert that specific methods of the mocked objects were called during the execution of the advice. For example, we verify that the getSignature() and getName() methods were called once each. Although not demonstrated in this simplified example, additional assertions can be added to ensure the correctness of the aspect's behavior. For instance, we could assert that the logging message produced by the aspect matches the expected format and content. Additional Considerations Testing pointcut expressions: Pointcut expressions define where advice should be applied within the application. Writing tests to verify the correctness of pointcut expressions ensures that the advice is applied to the intended join points. Testing aspect behavior: Aspects may perform more complex actions beyond simple logging. Unit tests should cover all aspects of the aspect's behavior to ensure its correctness, including handling method parameters, logging additional information, or interacting with other components. Integration testing: While unit tests focus on isolating aspects, integration tests may be necessary to verify the interactions between aspects and other components of the application, such as service classes or controllers. By following these principles and best practices, developers can create thorough and reliable unit tests for Spring AOP aspects, ensuring the stability and maintainability of their applications. Conclusion Unit testing Spring AOP aspects is crucial for reliable and correct aspect-oriented code. To create robust tests, isolate aspects, use mocking frameworks, test each advice separately, verify pointcut expressions, and assert expected behavior. Sample code provided as a starting point for Java applications. With proper testing strategies in place, developers can confidently maintain and evolve AOP-based functionalities in their Spring app.
I remember back when mobile devices started to gain momentum and popularity. While I was excited about a way to stay in touch with friends and family, I was far less excited about limits being placed on call length minutes and the number of text messages I could utilize … before being forced to pay more. Believe it or not, the #646 (#MIN) and #674 (#MSG) contact entries were still lingering in my address book until a recent clean-up effort. At one time, those numbers provided a handy mechanism to determine how close I was to hitting the monthly limits enforced by my service provider. Along some very similar lines, I recently found myself in an interesting position as a software engineer – figuring out how to log less to avoid exceeding log ingestion limits set by our observability platform provider. I began to wonder how much longer this paradigm was going to last. The Toil of Evaluating Logs for Ingestion I remember the first time my project team was contacted because log ingestion thresholds were exceeding the expected limit with our observability partner. A collection of new RESTful services had recently been deployed in order to replace an aging monolith. From a supportability perspective, our team had made a conscious effort to provide the production support team with a great deal of logging – in the event the services did not perform as expected. There were more edge cases than there were regression test coverage, so we were expecting alternative flows to trigger results that would require additional debugging if they did not process as expected. Like most cases, the project had aggressive deadlines that could not be missed. When we were instructed to “log less” an unplanned effort became our priority. The problem was, we weren’t 100% certain how best to proceed. We didn’t know what components were in a better state of validation (to have their logs reduced), and we weren’t exactly sure how much logging we would need to remove to no longer exceed the threshold. To our team, this effort was a great example of what has become known as toil: “Toil is the kind of work that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” – Eric Harvieux (Google Site Reliability Engineering) Every minute our team spent on reducing the amount of logs ingested into the observability platform came at the expense of delivering fewer features and functionality for our services. After all, this was our first of many planned releases. Seeking a “Log Whatever You Feel Necessary” Approach What our team really needed was a scenario where our observability partner was fully invested in the success of our project. In this case, it would translate to a “log whatever you feel necessary” approach. Those who have walked this path before will likely be thinking “this is where JV has finally lost his mind.” Stay with me here as I think I am on to something big. Unfortunately, the current expectation is that the observability platform can place limits on the amount of logs that can be ingested. The sad part of this approach is that, in doing so, observability platforms put their needs ahead of their customers – who are relying on and paying for their services. This is really no different from a time when I relied on the #MIN and #MSG contacts in my phone to make sure I lived within the limits placed on me by my mobile service provider. Eventually, my mobile carrier removed those limits, allowing me to use their services in a manner that made me successful. The bottom line here is that consumers leveraging observability platforms should be able to ingest whatever they feel is important to support their customers, products, and services. It’s up to the observability platforms to accommodate the associated challenges as customers desire to ingest more. This is just like how we engineer our services in a demand-driven world. I cannot imagine telling my customer, “Sorry, but you’ve given us too much to process this month.” Pay for Your Demand – Not Ingestion The better approach here is the concept of paying for insights and not limiting the actual log ingestion. After all, this is 2024 – a time when we all should be used to handling massive quantities of data. The “pay for your demand – not ingestion” concept has been considered a “miss” in the observability industry… until recently when I read that Sumo Logic has disrupted the DevSecOps world by removing limits on log ingestion. This market-disruptor approach embraces the concept of “log whatever you feel necessary” with a north star focused on eliminating silos of log data that were either disabled or skipped due to ingestion thresholds. Once ingested, AI/ML algorithms help identify and diagnose issues – even before they surface as incidents and service interruptions. Sumo Logic is taking on the burden of supporting additional data because they realize that customers are willing to pay a fair price for the insights gained from their approach. So what does this new strategy to observability cost expectations look like? It can be difficult to pinpoint exactly, but as an example, if your small-to-medium organization is producing an average of 25 MB of log data for ingestion per hour, this could translate into an immediate 10-20% savings (using Sumo Logic’s price estimator) on your observability bill. In taking this approach, every single log is available in a custom-built platform that scales along with an entity’s observability growth. As a result, AI/ML features can draw upon this information instantly to help diagnose problems – even before they surface with consumers. When I think about the project I mentioned above, I truly believe both my team and the production support team would have been able to detect anomalies faster than what we were forced to implement. Instead, we had to react to unexpected incidents that impacted the customer’s experience. Conclusion I was able to delete the #MIN and #MSG entries from my address book because my mobile provider eliminated those limits, providing a better experience for me, their customer. My readers may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” – J. Vester In 2023, I also started thinking hard about toil and making a conscious effort to look for ways to avoid or eliminate this annoying productivity killer. The concept of “zero dollar ingest” has disrupted the observability market by taking a lead from the mobile service provider's playbook. Eliminating log ingestion thresholds puts customers in a better position to be successful with their own customers, products, and services (learn more about Sumo Logic’s project here). From my perspective, not only does this adhere to my mission statement, it provides a toil-free solution to the problem of log ingestion, data volume, and scale. Have a really great day!