DZone Spotlight

The inverted MoSCoW framework reverses traditional prioritization, focusing on what a product team won’t build rather than what it will. Deliberately excluding features helps teams streamline development, avoid scope creep, and maximize focus on what truly matters. While it aligns with Agile principles of simplicity and efficiency, it also requires careful implementation to avoid rigidity, misalignment, or stifling innovation. Used thoughtfully, it’s a powerful tool for managing product scope and driving strategic clarity. Read on and learn how to make the inverted MoSCoW framework work for your team. Starting With MoSCoW The MoSCoW prioritization framework is a tool that helps product teams focus their efforts by categorizing work into four levels: Must-Have: These are non-negotiable requirements without which the software won’t function or meet its goals. Think of them as your product’s foundation. For example, a functional payment gateway is a “must-have” in an e-commerce app.Should-Have: These features add significant value but are not mission-critical. If time or resources are tight, these can be deferred without breaking the product. For example, a filtering option on a search page might enhance usability, but it isn’t essential to release the app.Could-Have: These are “nice-to-haves” — features that are not essential but would improve user experience if implemented. For example, animations or light/dark modes might fit here.Won’t-Have (this time): These are features explicitly deprioritized for this release cycle. They’re still documented but deferred for later consideration. For instance, integrating a recommendation engine for your MVP might be a “won’t-have.” To learn more about the original MoSCoW framework, see the Chapter 10 of the DSDM Agile Project Framework manual. MoSCoW Benefits The MoSCoW framework offers significant benefits, including clarity and focus by distinguishing critical features from optional ones, which helps product teams prioritize effectively. It supports aligning development efforts with time, budget, and technical capacity constraints. Its adaptability allows teams to respond to changes quickly, dropping lower-priority items when necessary. Additionally, the framework fosters team alignment by creating a shared understanding across cross-functional groups, ensuring everyone works toward common goals. MoSCoW Shortcomings However, the MoSCoW framework also has notable shortcomings, including its tendency to oversimplify prioritization by overlooking nuances like feature dependencies, where a “must-have” might rely on a “could-have.” It often lacks a quantitative assessment, relying instead on subjective judgments from stakeholders or leadership, which can misalign priorities by neglecting measurable impact or effort. Without solid discipline, it risks inflating the “must-have” category, overwhelming teams, and diluting focus. Additionally, the framework may emphasize output over outcomes, leading teams to prioritize delivering features without ensuring they achieve desired customer or business results. Known MoSCoW anti-patterns are: Applied Without Strategic Context: When priorities are assigned without tying them to a clear product vision, business outcomes, or user needs, the framework becomes arbitrary. Ensure “must-haves” directly support critical business or customer goals.The “Must-Have Creep:” Stakeholders often push too many items into the “must-have” category, which undermines prioritization and can lead to overcommitment. Push back by asking, “What happens if we don’t deliver this?”Static Priorities: MoSCoW works best in an iterative context but can fail if teams treat the categories as rigid. Priorities should be reviewed frequently, especially during discovery phases or as new constraints emerge.Ignoring Dependencies: A feature might seem low-priority but could block a higher-priority item. Consequently, technical dependencies should always be considered during prioritization.Siloed Decision-Making: If product managers assign priorities without consulting developers, it may lead to technical infeasibilities or underestimating the complexity of “must-haves.” Turning MoSCoW Upside Down: Meet the Inverted MoSCoW Let’s run a little thought experiment: Do you remember the principle of the Agile Manifesto that “Simplicity — the art of maximizing the amount of work not done — is essential?” So, why not turn MoSCoW upside down? The inverted MoSCoW framework flips the original framework, focusing primarily on what a product team will not build. Instead of prioritizing features or tasks to be included in a release, this approach emphasizes deliberate exclusions, helping teams identify and articulate boundaries. Here’s how it’s structured: Won’t-Have (Absolutely Not): Features or tasks explicitly ruled out for the foreseeable future. These could be ideas that don’t align with the product vision, are too costly or complex, or don’t deliver enough value. The debate ends here.Could-Have (But Unlikely): Features that might be considered someday but aren’t practical or impactful enough to be prioritized soon. They are low-value additions or enhancements with minimal urgency. Maybe we’ll get to them, perhaps we won’t — no promises.Should-Have (Under Very Specific Conditions): Features that might be built under exceptional circumstances. These could address edge cases, serve niche audiences, or depend on favorable future conditions, like extra resources or demand. Unless something significant changes, forget about them.Deferred Consideration (Maybe in the Future): These features or improvements are explicitly out of scope. The team acknowledges they may be somewhat important but intentionally excludes them, for now, to stay focused on core objectives. What Are the Benefits of an Inverted MoSCoW? Turning the original on its head has several advantages as we change the perspective and gain new insights: Aligns With Agile’s Simplicity Principle: The inverted MoSCoW approach reinforces Agile’s focus on maximizing the amount of work not done. By prioritizing exclusions, it ensures the team spends its energy on the most valuable and impactful work.Improves Focus and Efficiency: Defining what will not be built reduces distraction, scope creep, and debate during development. Teams avoid wasting time on “shiny object” features that may feel exciting but offer limited value.Encourages Strategic Restraint: By explicitly stating exclusions, the inverted framework helps ensure resources are allocated to problems that matter most. It also guards against overpromising or committing to ideas that lack clear value.Facilitates Transparent Communication: Stakeholders often feel disappointed when their ideas aren’t included. The inverted structure clarifies why specific ideas are excluded, fostering alignment and reducing conflict.Enables Long-Term Thinking: Teams can park features or ideas in “Could-Have (But Unlikely)” or “Must-Have (Future Consideration)” categories, ensuring they remain documented without distracting from immediate priorities.Prevents Cognitive Overload: Developers and product teams can stay laser-focused on what matters most without getting bogged down by debates over “extras.” It simplifies decision-making by narrowing the scope upfront. If we compare both approaches in a simplified version, it would look like this: AspectOriginal MoSCoWInverted MoSCoWFocusWhat will be builtWhat won’t be builtApproachInclusion-orientedExclusion-orientedGoalMaximize delivery of prioritized featuresMinimize waste and distractionsStakeholder RoleDefine priorities collaborativelyUnderstand and accept exclusionsScope Creep RiskMedium – “Should/Could” items may creep into workLow – Explicitly avoids unnecessary featuresAlignment with AgileSupports incremental deliveryEmbraces simplicity and focus “Flipping MoSCoW” aligns with Agile principles by minimizing unnecessary work, improving focus, and reducing cognitive overload. It fosters transparent communication, encourages strategic restraint, and documents ideas for future consideration, ensuring product teams target what truly matters. By anchoring exclusions in product vision, engaging stakeholders early, and revisiting decisions regularly, teams can avoid scope creep and manage expectations effectively while maintaining flexibility to adapt. Practical Steps to Use the Inverted MoSCoW Framework If you were to use the inverted MoSCoW framework to identify valuable work, consider the following practical steps to familiarize team members and stakeholders with the approach and get the communication right: Start With Product Vision and Strategy: Anchor exclusions in the product vision and strategy. For example, if you want to create a lightweight, user-friendly app, explicitly rule out features that add unnecessary complexity or bloat.Engage Stakeholders Early: Discuss exclusions with stakeholders upfront to set expectations and reduce future conflict. Use the inverted framework to clarify decisions to avoid being considered a black hole for decisions or simple “nay-sayers” by stakeholders.Build a Backlog of Exclusions — an Anti-Product Backlog: Maintain a list of features that won’t be built. This anti-product backlog serves as a transparent guide for future discussions.Revisit Regularly: Just as priorities can shift, so can exclusions. Reassess your “Could-Have” and “Should-Have” lists periodically to determine if conditions have changed — inspection and adaption will be crucial to maximizing value creation within the given constraints by the organization.Document the Rationale: For every exclusion, document why it was made, as they are dynamic and context-dependent. This context helps prevent revisiting the same debates and ensures alignment across teams and stakeholders. What isn’t feasible or aligned today might become critical in the future. Keep an archive of exclusions and periodically reassess them. Moreover, it would be helpful to consider applying complementary practices with the inverted MoSCoW framework, for example: Combine Frameworks for Robust Prioritization: Pair the inverted MoSCoW framework with other tools like Impact-Effort Matrices to identify low-effort, high-impact features that might deserve reconsideration or Opportunity Solution Trees to visualize how exclusions align with overarching goals.Use Prototyping and Experimentation: Validate an idea’s potential impact through lightweight prototypes or experiments before ruling it out. This ensures that promising concepts aren’t prematurely excluded. Also, expect practical challenges you will have to address when utilizing the inverted MoSCoW framework, for example: Resistance to Exclusions: Stakeholders often struggle to accept that their ideas are being excluded. To counter this, frame exclusions positively—focus on the value of prioritization and the benefits of delivering a lean, focused product.Exclusions as “Final Decisions:” Exclusions aren’t permanent. They’re tools for managing focus and scope at a specific moment. Encourage teams to view them as flexible and open to reassessment.Balance Between Focus and Innovation: While the framework promotes clarity and efficiency, excessive focus on exclusions can hinder creative exploration. Reserve space for continuous product discovery to keep the product competitive. Drawbacks of the Inverted MoSCoW Framework The inverted MoSCoW framework is valuable for defining what a product team will not build, helping teams focus on simplicity and efficiency. However, like its traditional counterpart, it is not without flaws. One significant challenge is the subjectivity in deciding exclusions. Stakeholders may struggle to align on what belongs in the “Won’t-Have” category, leading to potential conflicts or misaligned expectations. Without clear, objective criteria for exclusions, decisions risk being arbitrary or biased, undermining strategic goals and damaging team cohesion. Another critique is the framework’s tendency to encourage over-simplicity, which can stifle innovation or long-term thinking. While focusing on “not building” aligns with Agile principles of simplicity, over-prioritizing exclusions can narrow the product’s scope too much, leaving teams unprepared for future opportunities or changing market conditions. Balancing exclusions with flexibility is crucial, ensuring ideas with strategic potential aren’t entirely dismissed but appropriately categorized for future consideration. The framework also struggles to account for dependencies between excluded and included features. Excluding a “Won’t-Have” feature without understanding its role in supporting other work can inadvertently disrupt development, causing delays or requiring rework. Similarly, failing to consider effort or complexity in exclusions may result in missed opportunities to deliver low-effort, high-impact features. Teams must evaluate dependencies and efforts to ensure exclusions don’t inadvertently hinder progress or innovation. Finally, the inverted MoSCoW framework can become rigid, especially in agile, iterative environments where priorities shift rapidly. Exclusions defined early may no longer align with emerging user needs or business goals, creating tension between strategic intent and practical reality. To mitigate this, teams must treat exclusions as dynamic, revisiting and reassessing them regularly to ensure they remain relevant and effective. By addressing these critiques, the inverted MoSCoW framework can remain a powerful tool for managing focus and simplicity without sacrificing flexibility or strategic foresight. Conclusion The inverted MoSCoW framework is a powerful tool but is most effective as part of a broader prioritization strategy. By emphasizing collaboration, grounding decisions in data, and maintaining flexibility, you can ensure that exclusions support — not hinder — your product’s long-term success. Keep iterating, communicating, and aligning decisions with strategic goals, and the framework will serve as a valuable ally in your product development efforts.

By Stefan Wolpers CORE

In my previous post, I talked about why Console.log() isn’t the most effective debugging tool. In this installment, we will do a bit of an about-face and discuss the ways in which Console.log() is fantastic. Let’s break down some essential concepts and practices that can make your debugging life much easier and more productive. Front-End Logging vs. Back-End Logging Front-end logging differs significantly from back-end logging, and understanding this distinction is crucial. Unlike back-end systems, where persistent logs are vital for monitoring and debugging, the fluid nature of front-end development introduces different challenges. When debugging backends, I’d often go for tracepoints, which are far superior in that setting. However, the frontend, with its constant need to refresh, reload, contexts switch, etc., is a very different beast. In the frontend, relying heavily on elaborate logging mechanisms can become cumbersome. While tracepoints remain superior to basic print statements, the continuous testing and browser reloading in front-end workflows lessen their advantage. Moreover, features like logging to a file or structured ingestion are rarely useful in the browser, diminishing the need for a comprehensive logging framework. However, using a logger is still considered best practice over the typical Console.log for long-term logging. For short-term logging Console.log has some tricks up its sleeve. Leveraging Console Log Levels One of the hidden gems of the browser console is its support for log levels, which is a significant step up from rudimentary print statements. The console provides five levels: log: Standard loggingdebug: Same as log but used for debugging purposesinfo: Informative messages, often rendered like log/debugwarn: Warnings that might need attentionerror: Errors that have occurred While log and debug can be indistinguishable, these levels allow for a more organized and filtered debugging experience. Browsers enable filtering the output based on these levels, mirroring the capabilities of server-side logging systems and allowing you to focus on relevant messages. Customizing Console Output With CSS Front-end development allows for creative solutions, and logging is no exception. Using CSS styles in the console can make logs more visually distinct. By utilizing %c in a console message, you can apply custom CSS: CSS console.customLog = function(msg) { console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold") } console.customLog("Dazzle") This approach is helpful when you need to make specific logs stand out or organize output visually. You can use multiple %c substitutions to apply various styles to different parts of a log message. Stack Tracing With console.trace() The console.trace() method can print a stack trace at a particular location, which can sometimes be helpful for understanding the flow of your code. However, due to JavaScript’s asynchronous behavior, stack traces aren’t always as straightforward as back-end debugging. Still, it can be quite valuable in specific scenarios, such as synchronous code segments or event handling. Assertions for Design-by-Contract Assertions in front-end code allow developers to enforce expectations and promote a “fail-fast” mentality. Using Console.assert(), you can test conditions: JavaScript console.assert(x > 0, 'x must be greater than zero'); In the browser, a failed assertion appears as an error, similar to console.error. An added benefit is that assertions can be stripped from production builds, removing any performance impact. This makes assertions a great tool for enforcing design contracts during development without compromising production efficiency. Printing Tables for Clearer Data Visualization When working with arrays or objects, displaying data as tables can significantly enhance readability. The console.table() method allows you to output structured data easily: JavaScript console.table(["Simple Array", "With a few elements", "in line"]) This method is especially handy when debugging arrays of objects, presenting a clear, tabular view of the data and making complex data structures much easier to understand. Copying Objects to the Clipboard Debugging often involves inspecting objects, and the copy(object) method allows you to copy an object’s content to the clipboard for external use. This feature is useful when you need to transfer data or analyze it outside the browser. Inspecting With console.dir() and dirxml() The console.dir() method provides a more detailed view of objects, showing their properties as you’d see in a debugger. This is particularly helpful for inspecting DOM elements or exploring API responses. Meanwhile, console.dirxml() allows you to view objects as XML, which can be useful when debugging HTML structures. Counting Function Calls Keeping track of how often a function is called or a code block is executed can be crucial. The console.count() method tracks the number of times it’s invoked, helping you verify that functions are called as expected: JavaScript function myFunction() { console.count('myFunction called'); } You can reset the counter using console.countReset(). This simple tool can help you catch performance issues or confirm the correct execution flow. Organizing Logs With Groups To prevent log clutter, use console groups to organize related messages. console.group() starts a collapsible log section and console.groupEnd() closes it: JavaScript console.group('My Group'); console.log('Message 1'); console.log('Message 2'); console.groupEnd(); Grouping makes it easier to navigate complex logs and keeps your console clean. Chrome-Specific Debugging Features Monitoring Functions: Chrome’s monitor() method logs every call to a function, showing the arguments and enabling a method-tracing experience. Monitoring Events: Using monitorEvents(), you can log events on an element. This is useful for debugging UI interactions. For example, monitorEvents(window, 'mouseout') logs only mouseout events. Querying Object Instances: queryObjects(Constructor) lists all objects created with a specific constructor, giving you insights into memory usage and object instantiation. Final Word Front-end debugging tools have come a long way. These tools provide a rich set of features that go far beyond simple console.log() statements. From log levels and CSS styling to assertions and event monitoring, mastering these techniques can transform your debugging workflow. If you read this post as part of my series, you will notice a big change in my attitude toward debugging when we reach the front end. Front-end debugging is very different from back-end debugging. When debugging the backend, I’m vehemently against code changes for debugging (e.g., print debugging), but on the frontend, this can be a reasonable hack. The change in environment justifies it. The short lifecycle, the single-user use case, and the risk are smaller. Video

By Shai Almog CORE

This is an illustrated walkthrough for how to find the cause of high memory usage in Go programs with standard tools, based on our recent experience. There's plenty of information online, but it still took us considerable time and effort to find what works and what doesn't. Hopefully, this post will save time for everyone else. Context At Adiom, we're building an open-source tool for online database migration and real-time replication, called dsync. Our primary focus is on ease of use, robustness, and speed. One of the areas that we're focusing on heavily is data integrity, as Enterprise customers want to ensure data consistency between the source and the destination and know the differences, if there are any. To that end, we have recently come across Merkle Search Trees (MST) that have some interesting properties. It may be worth a whole different post, but for those interested now, here's the link to the original paper. In theory, MSTs should allow us to efficiently and cheaply diff the source and destination datasets. Naturally, we wanted to experiment with that, so we added a dsync verify <source> <destination> command as a POC leveraging an open-source reference implementation from GitHub: jrhy/mast. It worked like a charm on "toy" datasets from 100 to 1000 records. But in one of our larger tests, to our surprise, the binary consumed 3.5GB RAM for a million records on both sides. While not a whole lot in absolute numbers, this was orders of magnitude higher than what we were expecting — maybe 10s or 100s of megabytes — because we only stored the record ID (8 bytes) and a hash (8 bytes) for each record. Our first thought was a memory "leak." Similar to Java, Go manages memory for you and has a garbage collector based on open object references. There's no concept of leaks as such, but rather unexpected or undesirable accumulation of objects that the code kept references to. Unlike Java though, Go doesn't run as byte code on top of a JVM, so it doesn't know which specific objects are accumulating in memory. Part 1: Run Garbage Collector In Go, you can forcefully trigger garbage collection hoping that the memory is consumed by objects that are no longer needed: Go func main() { // App code log.Println("Triggering garbage collection...") runtime.GC() printMemUsage() // More app code } func printMemUsage() { var m runtime.MemStats runtime.ReadMemStats(&m) log.Printf("Alloc = %v MiB", bToMb(m.Alloc)) log.Printf("TotalAlloc = %v MiB", bToMb(m.TotalAlloc)) log.Printf("Sys = %v MiB", bToMb(m.Sys)) log.Printf("NumGC = %v\n", m.NumGC) } Unfortunately, that only freed up a few hundred MBs for us, so it wasn't very effective. Part 2: Examine the Memory In theory, we should be able to do the coredump of the process (dump the raw memory) and then examine it directly. Here are some relevant resources for that: Obtaining the core dump from a running process: Go Wiki: CoreDumpDebuggingSpecific instructions for Mac OS where gcore doesn't work right away: "How to Dump a Core File on MacOS"The viewcore tool to parse the coredump: GitHub: golang/debug Unfortunately, the viewcore didn't work with our coredump, presumably because we're doing development on a Mac laptop with Darwin architecture: Shell user@MBP viewcore % ./viewcore /cores/dsync-52417-20241114T234221Z html failed to parse core: bad magic number '[207 250 237 254]' in record at byte 0x0 If you're using Linux, you should give it a shot. Part 3: Profiling and Debugging We had to resort to investigative debugging instead. The most common sources of memory leaks in Go are: Accumulating and not closing resources such as network connections, files, etc.Global variablesSlices with large backing arraysLingering Go routines that don't finish After a brief examination of the codebase, we didn't see any smoking guns. Our next step was to use pprof. Pprof lets us sample a running process and obtain a heap dump that can later be examined. Pprof Setup Pprof runs as a webserver in your application. Adding it is easy: Go import ( "net/http" _ "net/http/pprof" // Import pprof package for profiling ) func main() { go func() { log.Println(http.ListenAndServe("localhost:8081", nil)) // Start pprof server }() // Your application logic here } In our case, it's accessible on port 8081. When the application is running and shows excessive memory consumption, we can collect heap dumps: Shell curl -s http://localhost:8081/debug/pprof/heap > /tmp/heap1.out I recommend collecting a few just to have a few different samples. Note that pprof actually samples information about allocations, so it's not going to be a 100% representation. By default, it's 1 sample per 512kb allocated. From here on out, it's really smooth sailing. Memory Profiling We examined inuse_space and inuse_objects with pprof that track the active space and the number of objects respectively: Shell go tool pprof --inuse_objects /tmp/heap1.out go tool pprof --inuse_space /tmp/heap1.out The in-use space was particularly fruitful. First, we tried to visualize the memory allocation using the web command, which opens a new web browser window. Shell user@MBP viewcore % go tool pprof --inuse_space /tmp/heap1.out File: dsync Type: inuse_space Time: Nov 14, 2024 at 5:34pm (PST) Entering interactive mode (type "help" for commands, "o" for options) (pprof) web (pprof) The thicker the link on the graph, the more memory went to that node (and that node is bigger, too). An alternative way to do this is to use the top5 command: Shell user@MBP viewcore % go tool pprof --inuse_space /tmp/heap1.out File: dsync Type: inuse_space Time: Nov 14, 2024 at 5:34pm (PST) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top5 Showing nodes accounting for 1670.40MB, 99.37% of 1680.96MB total Dropped 43 nodes (cum <= 8.40MB) Showing top 5 nodes out of 7 flat flat% sum% cum cum% 1471.89MB 87.56% 87.56% 1471.89MB 87.56% github.com/jrhy/mast.split 78MB 4.64% 92.20% 851.79MB 50.67% github.com/adiom-data/dsync/internal/app.(*mastVerify).Run.func1.processMast.1 77MB 4.58% 96.78% 826.13MB 49.15% github.com/adiom-data/dsync/internal/app.(*mastVerify).Run.func2.processMast.1 43.50MB 2.59% 99.37% 43.50MB 2.59% bytes.Clone 0 0% 99.37% 1677.92MB 99.82% github.com/adiom-data/dsync/internal/app.(*source).ProcessSource.func2 (pprof) These told us that the excessive memory was allocated in the mast.split function, and that it is a lot of 4.75kB objects. While we can't see what those objects are, we can see where in the code they were allocated using the list command: Shell user@MBP viewcore % go tool pprof --inuse_space /tmp/heap1.out File: dsync Type: inuse_space Time: Nov 14, 2024 at 5:34pm (PST) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top5 Showing nodes accounting for 1670.40MB, 99.37% of 1680.96MB total Dropped 43 nodes (cum <= 8.40MB) Showing top 5 nodes out of 7 flat flat% sum% cum cum% 1471.89MB 87.56% 87.56% 1471.89MB 87.56% github.com/jrhy/mast.split 78MB 4.64% 92.20% 851.79MB 50.67% github.com/adiom-data/dsync/internal/app.(*mastVerify).Run.func1.processMast.1 77MB 4.58% 96.78% 826.13MB 49.15% github.com/adiom-data/dsync/internal/app.(*mastVerify).Run.func2.processMast.1 43.50MB 2.59% 99.37% 43.50MB 2.59% bytes.Clone 0 0% 99.37% 1677.92MB 99.82% github.com/adiom-data/dsync/internal/app.(*source).ProcessSource.func2 (pprof) list github.com/jrhy/mast.split Total: 1.64GB ROUTINE ======================== github.com/jrhy/mast.split in /Users/alexander/go/pkg/mod/github.com/jrhy/mast@v1.2.32/lib.go 1.44GB 1.54GB (flat, cum) 93.57% of Total . . 82:func split(ctx context.Context, node *mastNode, key interface{}, mast *Mast) (leftLink, rightLink interface{}, err error) { . . 83: var splitIndex int . . 84: for splitIndex = 0; splitIndex < len(node.Key); splitIndex++ { . . 85: var cmp int . . 86: cmp, err = mast.keyOrder(node.Key[splitIndex], key) . . 87: if err != nil { . . 88: return nil, nil, fmt.Errorf("keyCompare: %w", err) . . 89: } . . 90: if cmp == 0 { . . 91: panic("split shouldn't need to handle preservation of already-present key") . . 92: } . . 93: if cmp > 0 { . . 94: break . . 95: } . . 96: } . . 97: var tooBigLink interface{} = nil 8MB 8MB 98: left := mastNode{ . . 99: Node{ 416.30MB 416.30MB 100: make([]interface{}, 0, cap(node.Key)), 458.51MB 458.51MB 101: make([]interface{}, 0, cap(node.Value)), 427.87MB 427.87MB 102: make([]interface{}, 0, cap(node.Link)), . . 103: }, . . 104: true, false, nil, nil, . . 105: } . . 106: left.Key = append(left.Key, node.Key[:splitIndex]...) . . 107: left.Value = append(left.Value, node.Value[:splitIndex]...) . . 108: left.Link = append(left.Link, node.Link[:splitIndex+1]...) . . 109: . . 110: // repartition the left and right subtrees based on the new key . . 111: leftMaxLink := left.Link[len(left.Link)-1] . . 112: if leftMaxLink != nil { . . 113: var leftMax *mastNode . . 114: leftMax, err = mast.load(ctx, leftMaxLink) . . 115: if mast.debug { . . 116: fmt.Printf(" splitting leftMax, node with keys: %v\n", leftMax.Key) . . 117: } . . 118: if err != nil { . . 119: return nil, nil, fmt.Errorf("loading leftMax: %w", err) . . 120: } . 91.39MB 121: leftMaxLink, tooBigLink, err = split(ctx, leftMax, key, mast) . . 122: if err != nil { . . 123: return nil, nil, fmt.Errorf("splitting leftMax: %w", err) . . 124: } . . 125: if mast.debug { . . 126: fmt.Printf(" splitting leftMax, node with keys: %v is done: leftMaxLink=%v, tooBigLink=%v\n", leftMax.Key, leftMaxLink, tooBigLink) . . 127: } . . 128: left.Link[len(left.Link)-1] = leftMaxLink . . 129: } . . 130: if !left.isEmpty() { . . 131: leftLink, err = mast.store(&left) . . 132: if err != nil { . . 133: return nil, nil, fmt.Errorf("store left: %w", err) . . 134: } . . 135: } 1MB 1MB 136: right := mastNode{ . . 137: Node{ 54.24MB 54.24MB 138: make([]interface{}, 0, cap(node.Key)), 56.75MB 56.75MB 139: make([]interface{}, 0, cap(node.Value)), 49.22MB 49.22MB 140: make([]interface{}, 0, cap(node.Link)), . . 141: }, . . 142: true, false, nil, nil, . . 143: } . . 144: right.Key = append(right.Key, node.Key[splitIndex:]...) . . 145: right.Value = append(right.Value, node.Value[splitIndex:]...) . . 146: right.Link = append(right.Link, node.Link[splitIndex:]...) . . 147: right.Link[0] = tooBigLink . . 148: . . 149: rightMinLink := right.Link[0] . . 150: if rightMinLink != nil { . . 151: var rightMin *mastNode . . 152: rightMin, err = mast.load(ctx, rightMinLink) . . 153: if err != nil { . . 154: return nil, nil, fmt.Errorf("load rightMin: %w", err) . . 155: } . . 156: var tooSmallLink interface{} . 9.54MB 157: tooSmallLink, rightMinLink, err = split(ctx, rightMin, key, mast) . . 158: if err != nil { . . 159: return nil, nil, fmt.Errorf("split rightMin: %w", err) . . 160: } . . 161: if mast.debug { . . 162: fmt.Printf(" splitting rightMin, node with keys %v, is done: tooSmallLink=%v, rightMinLink=%v", (pprof) Now we could clearly see what those objects were: arrays with preallocated capacity. These store the []interface{} type, which is 2 words in memory or 16 bytes on our machine (64-bit system). It must be a large number, at least on average (4.75kB / 16 bytes ~= 300). The mystery is half-solved. We weren't sure if that capacity was being used or not. So we used delve, which is a debugger for Go. Debugging A simple code inspection showed that those objects are part of tree nodes. To find out what they actually looked like, we used VS Code debug mode to attach to the running process and inspect these objects: Use the configuration to attach to the running process with the following launch.json: JSON { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Attach to Process", "type": "go", "request": "attach", "mode": "local", "processId": <PID of the running process> } ] } 2. Add a breakpoint in our verification loop where the tree objects are accessed. This immediately let us see that the array length was often much less than the preallocated size (cap) that was set: As a probe, I created a forked implementation that had no caps set in node allocation and the memory usage immediately dropped to 600MB, which is a 6x win. Mystery solved!

If there is one area where AI clearly demonstrates its value, it's knowledge management. Every organization, regardless of size, is inundated with vast amounts of documentation and meeting notes. These documents are often poorly organized, making it nearly impossible for any individual to read, digest, and stay on top of everything. However, with the power of large language models (LLMs), this problem is finally finding a solution. LLMs can read a variety of data and retrieve answers, revolutionizing how we manage knowledge. This potential has sparked discussions about whether search engines like Google could be disrupted by LLMs, given that these models can provide hyper-personalized answers. We are already witnessing this shift, with many users turning to platforms like ChatGPT or Perplexity for their day-to-day questions. Moreover, specialized platforms focusing on corporate knowledge management are emerging. However, despite the growing enthusiasm, there remains a significant gap between what the world perceives AI is capable of today and its actual capabilities. Over the past few months, I’ve explored building various AI-based tools for business use cases, discovering what works and what doesn’t. Today, I’ll share some of these insights on how to create a robust application that is both reliable and accurate. How to Provide LLMs With Knowledge For those unfamiliar, there are two common methods for giving large language models your private knowledge: fine-tuning or training your own model and retrieval-augmented generation (RAG). 1. Fine-Tuning This method involves embedding knowledge directly into the model's weights. While it allows for precise knowledge with fast inference, fine-tuning is complex and requires careful preparation of training data. This method is less common due to the specialized knowledge required. 2. Retrieval-Augmented Generation (RAG) The more widely used approach is to keep the model unchanged and insert knowledge into the prompt, a process some refer to as "in-context learning." In RAG, instead of directly answering user questions, the model retrieves relevant knowledge and documents from a private database, incorporating this information into the prompt to provide context. The Challenges of Simple RAG Implementations While RAG might seem simple and easy to implement, creating a production-ready RAG application for business use cases is highly complex. Several challenges can arise: Messy Real-World Data Real-world data is often not just simple text; it can include images, diagrams, charts, and tables. Normal data parsers might extract incomplete or messy data, making it difficult for LLMs to process. Accurate Information Retrieval Even if you create a database from company knowledge, retrieving relevant information based on user questions can be complicated. Different types of data require different retrieval methods, and sometimes, the information retrieved might be insufficient or irrelevant. Complex Queries Simple questions might require answers from multiple data sources, and complex queries might involve unstructured and structured data. Therefore, simple RAG implementations often fall short in handling real-world knowledge management use cases. Advanced RAG Techniques Thankfully, there are several tactics to mitigate these risks: Better Data Parsers Real-world data is often messy, especially in formats like PDFs or PowerPoint files. Traditional parsers, like PyPDF, might extract data incorrectly. However, newer parsers like LlamaParser, developed by LlamaIndex, offer higher accuracy in extracting data and converting it into an LLM-friendly format. This is crucial for ensuring the AI can process and understand the data correctly. Optimizing Chunk Size When building a vector database, it's essential to break down documents into small chunks. However, finding the optimal chunk size is key. If it is too large, the model might lose context; if it is too small, it might miss critical information. Experimenting with different chunk sizes and evaluating the results can help determine the best size for different types of documents. Reranking and Hybrid Search Reranking involves using a secondary model to ensure the most relevant chunks of data are presented to the model first, improving both accuracy and efficiency. Hybrid search, combining vector and keyword searches, can also provide more accurate results, especially in cases like e-commerce, where exact matches are critical. Agentic RAG This approach leverages agents' dynamic and reasoning abilities to optimize the RAG pipeline. For example, query translation can be used to modify user questions into more retrieval-friendly formats. Agents can also perform metadata filtering and routing to ensure only relevant data is searched, enhancing the accuracy of the results. Building an Agentic RAG Pipeline Creating a robust agentic RAG pipeline involves several steps: 1. Retrieve and Grade Documents First, retrieve the most relevant documents. Then, use the LLM to evaluate whether the documents are relevant to the question asked. 2. Generate Answers If the documents are relevant, generate an answer using the LLM. 3. Web Search If the documents are not relevant, perform a web search to find additional information. 4. Check for Hallucinations After generating an answer, check if the answer is grounded in the retrieved documents. If not, the system can either regenerate the answer or perform additional searches. 5. Use LangGraph and Llama3 Using tools like LangGraph and Llama3, you can define the workflow, setting up nodes and edges that determine the flow of information and the checks performed at each stage. Conclusion As you can see, building a reliable and accurate RAG pipeline involves balancing various factors, from data parsing and chunk sizing to reranking and hybrid search techniques. While these processes can slow down the response time, they significantly improve the accuracy and relevance of the answers provided by the AI. I encourage you to explore these methods in your projects and share your experiences. As AI continues to evolve, the ability to effectively manage and retrieve knowledge will become increasingly critical.

Have you ever wondered how to connect to an SAP Adaptive Server Enterprise (SAP ASE) database from a macOS machine? There are tools, such as RazorSQL, DBeaver, and DataGrip, that you can use to connect to SAP ASE databases from a macOS machine. What about secured connectivity like an SSL-enabled connection to the SAP ASE server? This article will show you how to securely connect to an SAP ASE over an end-to-end encrypted connection. What Is SAP ASE and What Are Its Security Features? SAP Adaptive Server Enterprise, formerly known as Sybase SQL Server, is a high-performance relational database management system (RDMS) designed for enterprise applications and data management. SAP ASE includes several major security features, including identification and authentication control, discretionary access control, role division, accountability, and data confidentiality. One of the main security features of SAP ASE is its support for Secure Sockets Layer (SSL) connectivity. SSL/TLS (Transport Layer Security) protects the transport of information between a client and a server (in this case, your database server) from tampering and eavesdropping by anyone on the network in between. By using SSL/TLS, SAP ASE ensures that the data exchanged between the database and client applications remains confidential and is protected from potential eavesdropping or tampering. SAP ASE offers several security advantages with SSL-enabled connectivity: It facilitates mutual authentication between the client and server, confirming the identities of both parties involved in the communication. This helps to prevent unauthorized access and man-in-the-middle attacks. SSL encrypts all data transmitted between the client and server, protecting sensitive information from being intercepted.SSL guarantees data integrity by identifying any alterations made to the transmitted data during transit. SSL connectivity with SAP ASE requires configuration on both the server and client sides. On the server side, administrators must install SSL certificates and enable SSL support in the database settings. On the client side, applications need to be configured to utilize SSL when connecting to the database. Although this process may involve some extra setup and configuration, the enhanced security that SSL provides is crucial for organizations handling sensitive data or operating in regulated sectors. Connecting to SAP ASE on macOS using tools like DataGrip and DBeaver gives developers and database administrators powerful interfaces for managing and querying their databases. These tools are especially beneficial on macOS, as they provide strong alternatives to native SAP ASE management tools that may not be easily accessible or optimized for the Mac environment. Why Use DataGrip and DBeaver on macOS MacOS users often face challenges when working with SAP ASE, as the official management tools are primarily designed for Windows and Linux environments. DataGrip and DBeaver bridge this gap by offering cross-platform solutions that provide rich features for database management, query execution, and data visualization. These tools support a wide range of databases, including SAP ASE, making them versatile options for professionals working with multiple database systems. DataGrip offers robust support for connecting to and working with Sybase databases. Further down, you will learn how to install and set up DataGrip and enable SSL to connect to a remote SAP ASE database. Downloading and Installing DataGrip Go to the Dowload pageSelect the macOS version.Once downloaded, open the .dmg file and drag the DataGrip icon to your Applications folder. DataGrip Downloads page Connecting to SAP ASE Open DataGrip and click "New Data Source" in the Database Explorer.Select "Sybase" from the list of database types. Setup Sybase as a Data Source on DataGrip 3. Configure the connection details: Host: Your SAP ASE server address (Use the IP address or the Fully Qualified Domain Name - FQDN)Port: Usually 5000 (default SAP ASE port)Database: Your database name (master)User: Your username (SAP ASE username - sa)Password: Your password Configure Sybase on DataGrip 4. For SSL configuration: Click on the "SSH/SSL" tab.Click the Browse icon next to "CA file" and point to the file with the chain of trust CA file. Select a "truststore."Alternatively, you can use "Client certificate," "Client key file," or "Client key password" if you have them.Click on the "Advanced" tab.Set ENABLE_SSL to true.Set HOSTNAME to PRIMARY_ASE.Set IGNORE_WARNINGS to true.Click "Apply" to save the changes. Enable SSL 5. Test the connection to ensure everything is set up correctly. You may receive a notification to Enable TLSv1, EnableTLSv1.1. Click on "Enable TLSv1.1." 6. Your VM Options should contain the following entries: Plain Text -Djdk.tls.disabledAlgorithms=SSLv3, TLSv1.1, DTLSv1.0, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, ECDH 7. A successful connection should show the following message: Plain Text DBMS: Adaptive Server Enterprise (ver. 16.0.04.06) Case sensitivity: plain=exact, delimited=exact Driver: jConnect (TM) for JDBC (TM) (ver. jConnect (TM) for JDBC(TM)/16.0 SP02 PL02 (Build 27276)/P/EBF25374/JDK 1.6.0/jdbcmain/OPT/Fri Oct 9 05:31:52 PDT 2015, JDBC4.0) Ping: 151 ms 8. Click "OK" to Save the configuration and connect to your SAP ASE database. Run Query 1. On the Database Explorer, right-click on the connection. 2. Click New → Query Console. 3. Paste the following SQL queries on the console. Select the query and click on the "Execute" icon to see the result: SQL use master go sp_helpdb go Run Query on the console with DataGrip Connection Best Practices Use the latest JDBC driver (jconn4.jar).Verify driver class: com.sybase.jdbc4.jdbc.SybDriver.Test the connection before saving.Ensure the network/firewall allows database access. Advanced Connection Options Configure SSH tunnels if required.Set up connection pools.Manage authentication methods.Use DataGrip's introspection features to explore database schemas. Additional Considerations JDBC Drivers: Both DataGrip and DBeaver may require you to download and configure the SAP ASE JDBC driver. You can usually download this from SAP's website or your organization's software repository.Network Configuration: Ensure your firewall and network settings allow connections to the SAP ASE server port.Performance: When working with large datasets, consider adjusting the fetch size and other performance-related settings in the connection properties of both tools.Version Compatibility: Always check that the versions of DataGrip, DBeaver, and the JDBC driver are compatible with your SAP ASE server version. Conclusion By utilizing DataGrip and DBeaver on macOS, developers and database administrators can efficiently manage their SAP ASE databases with feature-rich interfaces, advanced querying capabilities, and robust data visualization tools. These applications significantly enhance productivity and provide a seamless database management experience on the Mac platform.Dowload page

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm, blending the strengths of information retrieval and natural language generation. By leveraging large datasets to retrieve relevant information and generate coherent and contextually appropriate responses, RAG systems have the potential to revolutionize applications ranging from customer support to content creation. How Does RAG Work? Let us look at how RAG works. In a traditional setup, you will have a user prompt which is sent to the Large Language Model (LLM), and the LLM provides a completion: But the problem with this setup is that the LLM’s knowledge has a cutoff date, and it does not have insights into business-specific data. Importance of RAG for Accurate Information Retrieval RAG helps alleviate all the drawbacks that are listed above by allowing the LLM to access the knowledge base. Since the LLM now has context, the completions are more accurate and can now include business-specific data. The below diagram illustrates the value add RAG provides to content retrieval: As you can see, by vectorizing business-specific data, which the LLM would not have access to, instead of just sending the prompt to the LLM for retrieval, you send the prompt and context and enable the LLM to provide more effective completions. Challenges With RAG However, as powerful as RAG systems are, they face challenges, particularly in maintaining contextual accuracy and efficiently managing vast amounts of data. Other Challenges include: RAG systems will often find it very difficult to articulate complex relationships between information if it is distributed across a lot of documents.RAG solutions are very limited in their reasoning capabilities on the retrieved data.RAG solutions often tend to hallucinate when they are not able to retrieve desired information. Knowledge Graphs to the Rescue Knowledge graphs are sophisticated data structures that represent information in a graph format, where entities are nodes and relationships are edges. This structure plays a crucial role in overcoming the challenges faced by RAG systems, as it allows for a highly interconnected and semantically rich representation of data, enabling more effective organization and retrieval of information. Benefits of Using Knowledge Graphs for RAG Below are some key advantages for leveraging knowledge graphs: Knowledge graphs help RAG grasp complex information by providing rich context with the interconnected representation of information.With the help of knowledge graphs, RAG solutions can improve their reasoning capabilities when they traverse relationships in a better way. By linking information retrieved to specific aspects of the graph, knowledge graphs help increase factual accuracy. Impact of Knowledge Graphs on RAG Knowledge graphs fundamentally enhance RAG systems by providing a robust framework for understanding and navigating complex data relationships. They enable the AI not just to retrieve information based on keywords, but to also understand the context and interconnections between different pieces of information. This leads to more accurate, relevant, and contextually aware responses, significantly improving the performance of RAG applications. Now let us look at the importance of knowledge graphs in enhancing RAG application through a coding example. To showcase the importance, we will take the example of retrieving a player recommendation for an NFL Fantasy Football draft. We will ask the same question to the RAG application with and without knowledge graphs implemented, and we will see the improvement in the output. RAG Without Knowledge Graphs Let us look at the following code where we implement a RAG solution in its basic level for retrieving a football player of our choosing, which will be provided via a prompt. You can clearly see the output does not retrieve the accurate player based on our prompt. Python from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity import numpy as np # Sample player descriptions players = [ "Patrick Mahomes is a quarterback for the Kansas City Chiefs, known for his strong arm and playmaking ability.", "Derrick Henry is a running back for the Tennessee Titans, famous for his power running and consistency.", "Davante Adams is a wide receiver for the Las Vegas Raiders, recognized for his excellent route running and catching ability.", "Tom Brady is a veteran quarterback known for his leadership and game management.", "Alvin Kamara is a running back for the New Orleans Saints, known for his agility and pass-catching ability." ] # Vectorize player descriptions vectorizer = TfidfVectorizer() player_vectors = vectorizer.fit_transform(players) # Function to retrieve the most relevant player def retrieve_player(query, player_vectors, players): query_vector = vectorizer.transform([query]) similarities = cosine_similarity(query_vector, player_vectors).flatten() most_similar_player_index = np.argmax(similarities) return players[most_similar_player_index] # Function to generate a recommendation def generate_recommendation(query, retrieved_player): response = f"Query: {query}\n\nRecommended Player: {retrieved_player}\n\nRecommendation: Based on the query, the recommended player is a good fit for your team." return response # Example query query = "I need a versatile player." retrieved_player = retrieve_player(query, player_vectors, players) response = generate_recommendation(query, retrieved_player) print(response) We have oversimplified the RAG case for ease of understanding. Below is what the above code does: Imports necessary libraries: TfidfVectorizer from sklearn, cosine_similarity from sklearn, and numpyDefines sample player descriptions with details about their positions and notable skillsPlayer descriptions are vectorized using TF-IDF to convert the text into numerical vectors for precise similarity comparison.Defines a function retrieve_player to find the most relevant player based on a query by calculating cosine similarity between the query vector and player vectorsDefines a function generate_recommendation to create a recommendation message incorporating the query and the retrieved player's description Provides an example query, "I need a versatile player.", which retrieves the most relevant player, generates a recommendation, and prints the recommendation message. Now let's look at the output: PowerShell python ragwithoutknowledgegraph.py Query: I need a versatile player. Recommended Player: Patrick Mahomes is a quarterback for the Kansas City Chiefs, known for his strong arm and playmaking ability. Recommendation: Based on the query, the recommended player is a good fit for your team. As you can see, when we were asked for a versatile player, the recommendation was Patrick Mahomes. RAG With Knowledge Graphs Now let us look at how knowledge graphs can help enhance RAG and give a better recommendation. As you see from the output below, the correct player is recommended based on the prompt. Python import rdflib from rdflib import Graph, Literal, RDF, URIRef, Namespace # Initialize the graph g = Graph() ex = Namespace("http://example.org/") # Define players as subjects patrick_mahomes = URIRef(ex.PatrickMahomes) derrick_henry = URIRef(ex.DerrickHenry) davante_adams = URIRef(ex.DavanteAdams) tom_brady = URIRef(ex.TomBrady) alvin_kamara = URIRef(ex.AlvinKamara) # Add player attributes to the graph g.add((patrick_mahomes, RDF.type, ex.Player)) g.add((patrick_mahomes, ex.team, Literal("Kansas City Chiefs"))) g.add((patrick_mahomes, ex.position, Literal("Quarterback"))) g.add((patrick_mahomes, ex.skills, Literal("strong arm, playmaking"))) g.add((derrick_henry, RDF.type, ex.Player)) g.add((derrick_henry, ex.team, Literal("Tennessee Titans"))) g.add((derrick_henry, ex.position, Literal("Running Back"))) g.add((derrick_henry, ex.skills, Literal("power running, consistency"))) g.add((davante_adams, RDF.type, ex.Player)) g.add((davante_adams, ex.team, Literal("Las Vegas Raiders"))) g.add((davante_adams, ex.position, Literal("Wide Receiver"))) g.add((davante_adams, ex.skills, Literal("route running, catching ability"))) g.add((tom_brady, RDF.type, ex.Player)) g.add((tom_brady, ex.team, Literal("Retired"))) g.add((tom_brady, ex.position, Literal("Quarterback"))) g.add((tom_brady, ex.skills, Literal("leadership, game management"))) g.add((alvin_kamara, RDF.type, ex.Player)) g.add((alvin_kamara, ex.team, Literal("New Orleans Saints"))) g.add((alvin_kamara, ex.position, Literal("Running Back"))) g.add((alvin_kamara, ex.skills, Literal("versatility, agility, pass-catching"))) # Function to retrieve the most relevant player using the knowledge graph def retrieve_player_kg(query, graph): # Define synonyms for key skills synonyms = { "versatile": ["versatile", "versatility"], "agility": ["agility"], "pass-catching": ["pass-catching"], "strong arm": ["strong arm"], "playmaking": ["playmaking"], "leadership": ["leadership"], "game management": ["game management"] } # Extract key terms from the query and match with synonyms key_terms = [] for term, syns in synonyms.items(): if any(syn in query.lower() for syn in syns): key_terms.extend(syns) filters = " || ".join([f"contains(lcase(str(?skills)), '{term}')" for term in key_terms]) query_string = f""" PREFIX ex: <http://example.org/> SELECT ?player ?team ?skills WHERE {{ ?player ex:skills ?skills . ?player ex:team ?team . FILTER ({filters}) } """ qres = graph.query(query_string) best_match = None best_score = -1 for row in qres: skill_set = row.skills.lower().split(', ') score = sum(term in skill_set for term in key_terms) if score > best_score: best_score = score best_match = row if best_match: return f"Player: {best_match.player.split('/')[-1]}, Team: {best_match.team}, Skills: {best_match.skills}" return "No relevant player found." # Function to generate a recommendation def generate_recommendation_kg(query, retrieved_player): response = f"Query: {query}\n\nRecommended Player: {retrieved_player}\n\nRecommendation: Based on the query, the recommended player is a good fit for your team." return response # Example query query = "I need a versatile player." retrieved_player = retrieve_player_kg(query, g) response = generate_recommendation_kg(query, retrieved_player) print(response) Let us look at what the above code does. The code: Imports necessary libraries: rdflib, Graph, Literal, RDF, URIRef, and NamespaceInitializes an RDF graph and a custom namespace ex for defining URIsDefines players as subjects using URIs within the custom namespaceAdds player attributes (team, position, skills) to the graph using triplesDefines a function retrieve_player_kg to find the most relevant player based on a query by matching key terms with skills in the knowledge graphUses SPARQL to query the graph, applying filters based on synonyms of key skills extracted from the queryEvaluates query results to find the best match based on the number of matching skillsDefines a function generate_recommendation_kg to create a recommendation message incorporating the query and the retrieved player's informationProvides an example query "I need a versatile player.", retrieves the most relevant player, generates a recommendation, and prints the recommendation message Now let us look at the output: PowerShell python ragwithknowledgegraph.py Query: I need a versatile player. Recommended Player: Player: AlvinKamara, Team: New Orleans Saints, Skills: versatility, agility, pass-catching Recommendation: Based on the query, the recommended player is a good fit for your team. Conclusion: Leveraging RAG for Enhanced Knowledge Graphs Incorporating knowledge graphs into RAG applications results in more accurate, relevant, and context-aware recommendations, showcasing their importance in improving AI capabilities. Here are a few key takeaways: ragwithoutknowledgegraph.py uses TF-IDF and cosine similarity for text-based retrieval, relying on keyword matching for player recommendations.ragwithknowledgegraph.py leverages a knowledge graph, using RDF data structure and SPARQL queries to match player attributes more contextually and semantically.Knowledge graphs significantly enhance retrieval accuracy by adeptly understanding the intricate relationships and context between data entities.They support more complex and flexible queries, improving the quality of recommendations.Knowledge graphs provide a structured and interconnected data representation, leading to better insights.The illustration demonstrates the limitations of traditional text-based retrieval methods.It highlights the superior performance and relevance of using knowledge graphs in RAG applications.The integration of knowledge graphs significantly enhances AI-driven recommendation systems. Additional Resources Below are some of the resources that help with learning knowledge graphs and their impact on RAG solutions. Courses to Learn More About RAG and Knowledge Graphs https://learn.deeplearning.ai/courses/knowledge-graphs-rag/lesson/1/introductionhttps://ieeexplore.ieee.org/document/10698122 Open-Source Tools and Applications https://neo4j.com/generativeai/

Software erosion — gradual decay in software quality due to unmanaged technical debt and architectural drift — is a persistent challenge in software development. To combat this, jMolecules emerges as a robust, annotation-driven framework that simplifies the expression and enforcement of architectural principles within your codebase. By explicitly defining architectural concepts, jMolecules facilitates governance, quality assurance, and architectural integrity, ensuring your software remains robust. In this article, we'll explore jMolecules's fundamentals, its benefits, and a practical example of integrating it with ArchUnit to enforce Domain-Driven Design (DDD) principles. For further details, you can visit the jMolecules GitHub repository. Why jMolecules? jMolecules provides developers with annotations that make architectural evidence explicit in the code, ensuring clarity and structure. It achieves this through: Expressive Code: Architectural concepts are explicitly represented, aiding in code readability and maintainability.Domain Isolation: Domain-specific code remains free from technical dependencies.Boilerplate Reduction: Simplifies repetitive tasks, enabling developers to focus on logic.Tool Integration: Augmenting code with tools like ByteBuddy for Spring and JPA integration.Enforcing architectural rules with tools like ArchUnit and jQAssistant.Documentation Generation: Automatically generates architectural documentation and validates implementation. Modular Design for Flexibility jMolecules provides modules tailored to different architectural styles, allowing developers to adapt the framework to their needs. Some notable modules include: jmolecules-cqrs-architecture: For Command-Query Responsibility Segregation (CQRS).jmolecules-layered-architecture: For traditional layered architectures.jmolecules-onion-architecture: For Onion architecture.jmolecules-hexagonal-architecture: For Hexagonal architecture.jmolecules-ddd: This defines DDD building blocks like entities, value objects, and aggregates.jmolecules-events: For representing domain events. A Practical Example: Ensuring DDD Compliance With ArchUnit Defining a Domain Entity Let's start by defining a simple domain entity — a credit card. Using the @Entity annotation from jMolecules, we define our entity as follows: Java @Entity public class CreditCard { private BigInteger id; private String number; private String name; private YearMonth expiry; } Validating DDD Compliance To ensure the entity adheres to DDD principles, we can use jMolecules with ArchUnit. There are two approaches: manual validation and annotation-driven validation. Manual Validation Java public class JMoleculesDddUnitTest { @Test void checkTheLayerIntegration() { String packageName = "expert.os.examples"; JavaClasses classes = new ClassFileImporter().importPackages(packageName); JMoleculesArchitectureRules.ensureLayering().check(classes); } @Test void checkDDDIntegration() { String packageName = "expert.os.examples"; JavaClasses classes = new ClassFileImporter().importPackages(packageName); JMoleculesDddRules.all().check(classes); } } Annotation-Driven Validation The integration becomes more concise with annotations: Java @AnalyzeClasses(packages = "expert.os.examples") public class IntegrationSampleTest { @ArchTest private ArchRule dddRules = JMoleculesDddRules.all(); @ArchTest private ArchRule layering = JMoleculesArchitectureRules.ensureLayering(); @ArchTest void detectsViolations(JavaClasses classes) { EvaluationResult result = JMoleculesDddRules.all().evaluate(classes); assertThat(result.hasViolation()).isFalse(); } } Running the tests may reveal violations, such as: Type e.o.e.CreditCard must declare a field or a method annotated with org.jmolecules.ddd.annotation.Identity! This error aligns with Eric Evans' DDD definition of an entity, emphasizing the importance of identity. To resolve this, we annotate the id field with @Identity: Java @Entity public class CreditCard { @Identity private BigInteger id; private String number; private String name; private YearMonth expiry; } With this adjustment, the tests pass, ensuring our entity adheres to DDD principles. Conclusion jMolecules offers a comprehensive suite of validations and tools that are invaluable in real-world projects. While this article briefly introduces the framework's capabilities extend far beyond the scope covered here. Explore the project repository to dive deeper into jMolecules and its integration. By leveraging jMolecules, developers can effectively combat software erosion, maintain architectural integrity, and build high-quality, sustainable software systems. Video

By Otavio Santana CORE

Daily standups are a staple in agile software development, designed to keep teams aligned, unblock issues, and ensure smooth progress. Traditionally, these standups have been synchronous meetings, where the team gathers at a set time each day to share updates. However, with the rise of remote work and distributed teams, asynchronous standups have emerged as a viable alternative. In this blog, we’ll explore the pros and cons of synchronous and asynchronous standups, helping you weigh the options and decide which model best suits your team’s unique needs. Synchronous Standups: Pros and Cons A synchronous standup is a real-time meeting where all team members join at a set time, either in person or via video conferencing, to discuss what they worked on, what they’re working on, and any blockers. Pros Instant communication: Real-time conversations foster quick discussions, clarifications, and resolutions. If someone has a blocker, team members can offer immediate help.Promotes team bonding: Synchronous standups bring the team together, creating a sense of camaraderie. This is especially beneficial for teams that don't often work face-to-face.Encourages accountability: Regular live check-ins ensure that everyone is aligned and accountable for their commitments.Fosters spontaneous discussions: Issues not originally on the agenda can surface and be discussed on the spot, potentially solving problems quicker than in an asynchronous setup. Cons Scheduling difficulties: Time zone differences, personal schedules, and remote work can make it hard to find a time that works for everyone, particularly in global teams.Potential for wasted time: Standups can sometimes veer off-track, leading to longer discussions that take up more time than necessary.Interruptions in deep work: For engineers deeply immersed in coding, pausing for a daily meeting can disrupt focus and productivity. Asynchronous Standups: Pros and Cons In an asynchronous standup, team members provide their updates in a written or recorded format (such as via Slack, a project management tool, or email), allowing each person to respond at their convenience. Pros Flexibility: Team members can provide their updates whenever it works best for them, accommodating different time zones, personal schedules, and varied working hours.Less disruptive: Asynchronous updates allow developers to stay focused on their work without the need to stop mid-task for a meeting.Permanent record: Written updates create a log of daily progress that can be referred back to, aiding transparency and making it easier to track progress over time.Inclusive for all time zones: For distributed teams, asynchronous standups are more equitable, ensuring everyone can participate without scheduling conflicts. Cons Lack of immediacy: Without real-time interaction, urgent issues might not be addressed as quickly, and discussions around blockers might take longer to resolve.Reduced team cohesion: Team members might feel more isolated when not regularly interacting face-to-face, which can hinder team bonding. This is especially a big deal if there is a new member on the team. They may have a really hard time bonding with the team.Potential for miscommunication: Written updates can sometimes lead to misunderstandings or incomplete information, as there’s no immediate opportunity to clarify or ask follow-up questions.Can lead to disengagement: Without the structure of a live meeting, some team members might delay or neglect to provide their updates, reducing the overall effectiveness of the standup. Which Model Is Right for Your Team? Ultimately, whether you choose synchronous or asynchronous standups depends on the specific needs and dynamics of your team. Both approaches have strengths and limitations, and neither is inherently better than the other. Synchronous standups may work best for co-located teams or teams in similar time zones who benefit from spontaneous problem-solving and team bonding.Asynchronous standups could be a better fit for distributed teams that need flexibility and autonomy in their daily routines, especially when time zones make scheduling live meetings challenging. Conclusion There is no one-size-fits-all approach when it comes to running efficient daily standups. The right choice — whether synchronous, asynchronous, or even a hybrid of both — depends on your team’s needs, culture, and workflow. By carefully considering the pros and cons of each model, you can find the balance that best supports your team’s productivity, communication, and overall success.

DevSecOps is an increasingly popular framework for integrating security into every phase of the software development lifecycle. A key phase that organizations should be targeting as soon as possible is their Continuous Integration and Continuous Deployment (CI/CD) pipeline. Doing so can accomplish more than increasing security and decreasing risk. It can also increase productivity and morale. In 2020, the devastating attack on managed software provider SolarWinds brought software supply chain vulnerabilities into public consciousness. Attackers inserted malicious code into the company platform, which was then distributed via standard updates to around 18,000 customers, including several U.S. government agencies and Fortune 500 companies, leading to data theft and possible espionage. A similar attack against Kaseya VSA in 2021 impacted around 1,500 businesses, causing disruption and financial loss, including forcing a Swedish supermarket chain to close temporarily. These attacks demonstrate how a single vulnerability can impact thousands of companies and potentially millions of people. However, despite the increased awareness of the threat, which even led to the 2021 Executive Order on Improving the Nation’s Cybersecurity in the U.S., software supply chain vulnerabilities remain a major problem. Increased reliance on open-source dependencies and third-party integrations continues to expand attack surfaces, making it easier for attackers to infiltrate the systems, while the competitive demand to deliver software within ever shorter timeframes often distracts teams from their security mission. As a result, every company that isn’t actively seeking to improve its software supply chain defenses is vulnerable. DevSecOps and the CI/CD Pipeline A critical component of the software delivery supply chain is the CI/CD pipeline, which is a set of practices development teams use to deliver code changes more frequently and reliably. CI/CD pipelines have become the foundation of fast and efficient software delivery processes that allow for the frequent addition of new features and bug fixes. However, an insecure CI/CD pipeline opens multiple entry points for attacks, including malicious code, compromised source controls, vulnerable dependencies, a compromised build system, the injection of bad artifacts, compromised package management and artifact signing, abuse of privileges, and more. These vulnerabilities introduce a significant security gap in the software supply chain. To secure CI/CD pipelines, many organizations are now integrating DevSecOps best practices into it. A DevSecOps approach ensures that security gets embedded into every phase of the pipeline. There are two main aspects of this: Incorporating application security testing strategies to establish a strong application security posture. These include static code analysis, dynamic application testing, open source and third-party package management, secrets scanning, and vulnerability management. Instituting deployment security measures, including image and container security, infrastructure as code (IaC) and environment security, secrets management, cloud security, and network security. DevSecOps processes and tools enable automated security checks and continuous monitoring that make securing the development process possible. By doing so, they also foster better collaboration among the development, operations, and security teams. This promotes a more secure development environment and more secure and resilient software. The DevSecOps CI/CD Pipeline The DevSecOps CI/CD pipeline uses automation to add key measures and strategies to each of its five stages: code, build, test, deploy, and monitor. 1. Code DevSecOps secures the coding stage of the CI/CD pipeline by automatically evaluating source code to detect vulnerable or faulty code that may have been checked in by developers. These automated evaluations include: Software Composition Analysis (SCA) to detect known vulnerabilities and license issues in open-source dependenciesStatic Application Security Testing (SAST) to automate scanning and analysis of code for early detection of vulnerabilitiesSecrets Scanning and Management to make sure sensitive information, including passwords, API keys, tokens, encryption keys, and more, are not added to the codebase 2. Build This DevSecOps best practice automatically scans binaries and packages after the source code is committed, compiled into executable artifacts, and stored in an artifact repository. Automated scanning of binaries and packages should be continuous until the code is sent to the testing stage, where the code, still in the form of artifacts, is scanned in its entirety. 3. Test At this stage, real-time testing of the application is conducted through attack simulations using PenTesting, SQL injections, cross-site scripting, and more. Artifact scanning and Dynamic Application Security Testing (DAST) are also integrated into the CI/CD pipeline during this stage. 4. Deploy The deployment stage can be leveraged as a key control point to automate security checks and enforce application security measures. Key practices include: Policy checksCompliance verificationDigital scanningInfrastructure as Code (IaC) securityCloud security 5. Monitor Continuous and automated monitoring must be implemented to ensure applications continue to operate as expected. A DevSecOps best practice is to add continuous scanning for Common Vulnerabilities and Exposures (CVEs) and integrate these scans with audit tools that can track and report on compliance with industry and organization rules. The Benefits of Taking a Best Practices DevSecOps CI/CD Approach The most important DevSecOps best practices for CI/CD pipelines are: Shift Left: Integrate security tests and impose security checks as early as possible in the development process to make identifying and addressing vulnerabilities faster and easier.Automate Security Testing: Automatically conduct security tests and scans throughout the pipeline.Monitor Continuously: Identify and mitigate security incidents in real time through continuous monitoring.Collaborate Across Teams: Nurture collaboration among development, operations, and security teams to ensure security remains a top priority for everyone and increase productivity.Utilize Regular Training and Awareness Programs: These programs are essential for ensuring that security best practices remain a priority for everyone with a stake in the CI/CD pipeline. Companies that implement and maintain these DevSecOps best practices for CI/CD pipelines will enjoy several key benefits: Enhanced Security: Early identification and mitigation of vulnerabilities, leading to safer software, increased developer productivity, and cost savings.Accelerated Time-to-Market: Faster delivery of secure software that does not require time-consuming and expensive fixes.Happier and More Productive Teams: The ability to satisfy the needs of the development, operations, and security teams without compromising productivity or security standards.Reduced Security Risks: A dramatically lower risk of security breaches and data loss that can result in fines or damage to the brand.Proactive Compliance: Identification of regulatory and organizational compliance issues before violations occur. Conclusion As part of securing a company’s software supply chain, every software organization should embrace DevSecOps for CI/CD pipelines. However, it’s about more than just security. With DevSecOps for CI/CD pipelines, companies can accelerate software production and, equally important, improve the morale of everyone involved in the software development process, as well as those who track user satisfaction with that software.

PostgreSQL is known for its robustness and flexibility, but to get the most out of it in high-traffic or data-intensive environments, tuning is essential. This guide outlines key tuning tips that database administrators and developers can use to optimize PostgreSQL performance. Key Tuning Tips 1. Memory Configuration Shared Buffers PostgreSQL’s shared_buffers setting controls the amount of memory used for caching data. Set this to about 25-40% of total system memory, but avoid over-allocating, as the OS also needs memory for file caching. Reference: PostgreSQL Shared Buffers Documentation Plain Text shared_buffers = 1GB # Set to 25-40% of system memory Work Mem For complex queries or sorting, work_mem defines how much memory each connection can use for query operations. Increase this value for better performance with larger datasets, but be cautious: this is allocated per query, so increasing it too much could exhaust memory. Reference: PostgreSQL Work Mem Documentation Plain Text work_mem = 16MB # Adjust based on workload 2. Effective Cache Size This is an important setting for query planning, as PostgreSQL uses effective_cache_size to estimate how much memory is available for disk caching. Set it to about 75% of total system memory. Reference: PostgreSQL Effective Cache Size Documentation Plain Text effective_cache_size = 3GB 3. Checkpoint Settings Tuning checkpoint settings can help reduce disk I/O load and improve performance during periods of high write activity. Consider adjusting checkpoint_timeout and checkpoint_completion_target. Reference: PostgreSQL Checkpoint Settings Documentation Plain Text checkpoint_timeout = 15min # Adjust based on workload checkpoint_completion_target = 0.7 # Set to balance write load 4. Autovacuum Tuning Autovacuum is critical for preventing table bloat. Tuning autovacuum settings helps maintain database performance over time. Reference: PostgreSQL Autovacuum Documentation Plain Text autovacuum_vacuum_threshold = 50 autovacuum_analyze_threshold = 50 Adjust these based on the size and activity level of your tables. 5. Query Planning with EXPLAIN and ANALYZE PostgreSQL’s EXPLAIN and ANALYZE tools allow you to understand how queries are executed. Use these commands to identify bottlenecks and optimize slow-running queries. Reference: PostgreSQL EXPLAIN Documentation Plain Text EXPLAIN ANALYZE SELECT * FROM my_table WHERE condition; 6. Connection Pooling For systems handling a large number of concurrent connections, using a connection pooling tool like PgBouncer can greatly reduce overhead. This helps PostgreSQL efficiently manage resources. Reference: PgBouncer Documentation Plain Text pgbouncer.ini # Example configuration for PgBouncer 7. Partitioning Large Tables Partitioning is a powerful tool for optimizing queries on large tables. By breaking a large table into smaller partitions, PostgreSQL can process queries faster. Reference: PostgreSQL Partitioning Documentation SQL CREATE TABLE measurement ( city_id int, logdate date, peaktemp int, unitsales int ) PARTITION BY RANGE (logdate); 8. Indexing Best Practices Use indexes wisely. Over-indexing can lead to performance degradation during writes, but proper indexing improves query performance significantly. Reference: PostgreSQL Indexes Documentation SQL CREATE INDEX idx_measurement_logdate ON measurement (logdate); 9. Parallel Query Execution Leverage PostgreSQL’s parallel query execution to speed up query performance on multi-core systems. Adjust max_parallel_workers and max_parallel_workers_per_gather to enable this. Reference: PostgreSQL Parallel Query Documentation Plain Text max_parallel_workers = 8 max_parallel_workers_per_gather = 4 10. Logging and Monitoring Monitor PostgreSQL’s logs to identify performance bottlenecks. Enable logging for long-running queries. Reference: PostgreSQL Logging Documentation Plain Text log_min_duration_statement = 500ms # Log queries that take more than 500ms Use tools like pg_stat_statements to monitor query performance and identify which queries need optimization. Conclusion These tuning tips provide a solid foundation for optimizing PostgreSQL performance. By adjusting memory settings, utilizing autovacuum, and leveraging parallel execution, you can ensure your PostgreSQL database performs optimally, even under heavy load. Don’t forget to monitor your performance metrics regularly to keep your system running smoothly.