Culture and Methodologies

GitHub Action integration with Playwright enables seamless automated testing and deployment workflows for web applications. GitHub Actions, the platform’s automation tool, allows these tests to be triggered automatically upon code changes, ensuring rapid feedback and efficient bug detection. This integration empowers teams to build, test, and deploy with confidence, automating repetitive tasks and enhancing overall development productivity. By combining the versatility of Playwright with the automation capabilities of GitHub Actions, developers can streamline their workflows, delivering high-quality web applications with speed and precision. What Is Playwright? Microsoft Playwright is an open-source automation framework for end-to-end testing, browser automation, and web scraping. Developed by Microsoft, Playwright provides a unified API to automate interactions with web browsers like Microsoft Edge, Google Chrome, and Mozilla Firefox. It allows developers to write scripts in various programming languages, including Java, Python, JavaScript, and C#. Here are some key features of Playwright: Multi-Browser Support: Playwright supports multiple web browsers, including Firefox, Chrome, Safari, and Microsoft Edge. This allows developers and testers to run their tests on different browsers with a consistent API.Headless and Headful Modes: Playwright can run browsers in both headless mode (without a graphical interface) and headful mode (with a graphical interface), providing flexibility for different use cases.Cross-Browser Testing: Playwright allows you to write tests that run on multiple browsers and platforms, ensuring your web application works correctly across different platforms.Emulation of Mobile Devices and Touch Events: Playwright can emulate various mobile devices and simulate touch events, enabling you to test how your web application behaves on different mobile devices.Parallel Test Execution: Playwright supports parallel test execution, allowing you to run tests concurrently, reducing the overall test suite execution time.Capture Screenshots and Videos: Playwright can capture screenshots and record videos during test execution, helping you visualize the behavior of your application during tests.Intercept Network Requests: You can intercept and modify network requests and responses, which is useful for testing scenarios involving AJAX requests and APIs.Auto-Waiting for Elements: Playwright automatically waits for elements to be ready before performing actions, reducing the need for manual waits and making tests more reliable.Page and Browser Contexts: Playwright allows you to create multiple browser contexts and pages, enabling efficient management of browser instances and isolated environments for testing. What Is GitHub Actions? GitHub Actions is an automation platform offered by GitHub that streamlines software development workflows. It empowers users to automate a wide array of tasks within their development processes. By leveraging GitHub Actions, developers/qa engineers can craft customized workflows that are initiated by specific events such as code pushes, pull requests, or issue creation. These workflows can automate essential tasks like building applications, running tests, and deploying code. Essentially, GitHub Actions provides a seamless way to automate various aspects of the software development lifecycle directly from your GitHub repository. How GitHub Actions Effective in Automation Testing GitHub Actions is a powerful tool for automating various workflows, including QA automation testing. It allows you to automate your software development processes directly within your GitHub repository. Here are some ways GitHub Actions can be effective in QA automation testing: 1. Continuous Integration (CI) GitHub Actions can be used for continuous integration, where automated tests are triggered every time there is a new code commit or a pull request. This ensures that new code changes do not break existing functionality. Automated tests can include unit tests, integration tests, and end-to-end tests. 2. Diverse Test Environments GitHub Actions supports running workflows on different operating systems and environments. This is especially useful for QA testing, as it allows you to test your application on various platforms and configurations to ensure compatibility and identify platform-specific issues. 3. Parallel Test Execution GitHub Actions allows you to run tests in parallel, significantly reducing the time required for test execution. Parallel testing is essential for large test suites, as it helps in obtaining faster feedback on the code changes. 4. Custom Workflows You can create custom workflows tailored to your QA automation needs. For example, you can create workflows that run specific tests based on the files modified in a pull request. This targeted testing approach helps in validating specific changes and reduces the overall testing time. 5. Integration With Testing Frameworks GitHub Actions can seamlessly integrate with popular testing frameworks and tools. Whether you are using Selenium, Cypress, Playwright for web automation, Appium for mobile automation, or any other testing framework, you can configure GitHub Actions to run your tests using these tools In the next section, you will see how we can integrate GitHub Actions with Playwright to execute the test cases. Set Up CI/CD GitHub Actions to Run Playwright Tests Pre-Condition The user should have a GitHub account and already be logged in. Use Cases For automation purposes, we are taking two examples, one of UI and the other of API. Example 1 Below is an example of a UI test case where we log in to the site https://talent500.co/auth/signin. After a successful login, we log out from the application. JavaScript // @ts-check const { test, expect } = require("@playwright/test"); test.describe("UI Test Case with Playwright", () => { test("UI Test Case", async ({ page }) => { await page.goto("https://talent500.co/auth/signin"); await page.locator('[name="email"]').click(); await page.locator('[name="email"]').fill("applitoolsautomation@yopmail.com"); await page.locator('[name="password"]').fill("Test@123"); await page.locator('[type="submit"]').nth(1).click(); await page.locator('[alt="DropDown Button"]').click(); await page.locator('[data-id="nav-dropdown-logout"]').click(); }); }); Example 2 Below is an example of API testing, where we automate using the endpoint https://reqres.in/api for a GET request. Verify the following: GET request with Valid 200 ResponseGET request with InValid 404 ResponseVerification of user details JavaScript // @ts-check const { test, expect } = require("@playwright/test"); test.describe("API Testing with Playwright", () => { const baseurl = "https://reqres.in/api"; test("GET API Request with - Valid 200 Response", async ({ request }) => { const response = await request.get(`${baseurl}/users/2`); expect(response.status()).toBe(200); }); test("GET API Request with - Invalid 404 Response", async ({ request }) => { const response = await request.get(`${baseurl}/usres/invalid-data`); expect(response.status()).toBe(404); }); test("GET Request - Verify User Details", async ({ request }) => { const response = await request.get(`${baseurl}/users/2`); const responseBody = JSON.parse(await response.text()); expect(response.status()).toBe(200); expect(responseBody.data.id).toBe(2); expect(responseBody.data.first_name).toBe("Janet"); expect(responseBody.data.last_name).toBe("Weaver"); expect(responseBody.data.email).toBeTruthy(); }); }); Steps For Configuring GitHub Actions Step 1: Create a New Repository Create a repository. In this case, let’s name it “Playwright_GitHubAction.” Step 2: Install Playwright Install Playwright using the following command: Plain Text npm init playwright@latest Or Plain Text yarn create playwright Step 3: Create Workflow Define your workflow in the YAML file. Here’s an example of a GitHub Actions workflow that is used to run Playwright test cases. In this example, the workflow is triggered on every push and pull request. It sets up Node.js, installs project dependencies, and then runs npx playwright test to execute Playwright tests. Add the following .yml file under the path .github/workflows/e2e-playwright.yml in your project. Plain Text name: GitHub Action Playwright Tests on: push: branches: [main] pull_request: branches: [main] jobs: test: timeout-minutes: 60 runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: actions/setup-node@v3 with: node-version: 18 - name: Install dependencies run: npm ci - name: Install Playwright Browsers run: npx playwright install --with-deps - name: Run Playwright tests run: npx playwright test - uses: actions/upload-artifact@v3 if: always() with: name: playwright-report path: playwright-report/ retention-days: 10 Here’s a breakdown of what this workflow does: Trigger Conditions The workflow is triggered on push events to the main branch. Job Configuration Job name: e2e-testTimeout: 60 minutes (the job will terminate if it runs for more than 60 minutes)Operating system: ubuntu-latest Steps Check out the repository code using actions/checkout@v3.Set up Node.js version 18 using actions/setup-node@v3.Install project dependencies using npm ci.Install Playwright browsers and their dependencies using npx playwright install --with-deps.Run Playwright tests using npx playwright test.Upload the test report directory (playwright-report/) as an artifact using actions/upload-artifact@v3. This step always executes (if: always()), and the artifact is retained for 10 days. Test results will be stored in the playwright-report/ directory. Below is the folder structure where you can see the .yml file and test cases under the tests folder to execute. Execute the Test Cases Commit your workflow file (e2e-playwright.yml) and your Playwright test files. Push the changes to your GitHub repository. GitHub Actions will automatically pick up the changes and run the defined workflow. As we push the code, the workflow starts to run automatically. Click on the above link to open the e2e-test. Click on e2e-test. In the screen below, you can see the code being checked out from GitHub, and the browsers start installing. Once all dependencies and browsers are installed, the test cases start executing. In the screenshot below, you can see all 12 test cases passed in three browsers (Firefox, Chrome, and WebKit). HTML Report Click on the playwright-report link from the Artifacts section. An HTML report is generated locally Click the link above to view the HTML report. Both API and UI test cases are passed. Wrapping Up GitHub Actions automate the testing process, ensuring every code change is thoroughly examined without manual intervention. Playwright’s ability to test across various browsers and platforms guarantees a comprehensive evaluation of your application’s functionality. By combining GitHub Actions and Playwright, developers can streamline workflows, ensure code quality, and ultimately deliver better user experiences.

By Kailash Pathak CORE

Can you rely on pure Scrum to transform your organization and deliver value? Not always. While Scrum excels in simplicity and flexibility, applying it “out of the box” often falls short in corporate contexts due to limitations in product discovery, scaling, and portfolio management. This article explores the conditions under which pure Scrum thrives, the organizational DNA required to support it, and practical scenarios where it works best — along with a candid look at where it struggles. Discover whether pure Scrum is a realistic approach for your team and how thoughtful adaptation can unlock its true potential. Pure Scrum Constraints “Pure Scrum,” described in the Scrum Guide, is an idiosyncratic framework that helps create customer value in a complex environment. However, five main issues are challenging its general corporate application: Pure Scrum focuses on delivery: How can we avoid running in the wrong direction by building things that do not solve our customers’ problems?Pure Scrum ignores product discovery in particular and product management in general. If you think of the Double Diamond, to use a popular picture, Scrum is focused on the right side; see above.Pure Scrum is designed around one team focused on supporting one product or service. Pure Scrum does not address portfolio management. It is not designed to align and manage multiple product initiatives or projects to achieve strategic business objectives. Pure Scrum is based on far-reaching team autonomy: The Product Owner decides what to build, the Developers decide how to build it, and the Scrum team self-manages. While constant feedback loops, from Product Backlog refinement to the Daily Scrum to Sprint Review to the Retrospective, help with the delivery and risk mitigation focus, the lack of “product genes” is more challenging. The idea that the Product Owner knows what is worth building is unconventional. Consequently, many practitioners, particularly from the management level, are skeptical about this level of faith. As a result, most Product Owners are told what to do: requirements, deadlines, feature factories — you name it. Also, having just one Scrum team is rare unless you’re working for a startup in its infancy. Most of the time, multiple teams develop products. Scrum scaling frameworks like LeSS or Nexus try to remedy the issue, often with limited success. Closely related is pure Scrum’s lack of any alignment layer or process. Its lack might also be where SAFe has its most outstanding “achievement” in corporate settings if you like to use that term in conjunction with it. Finally, pure Scrum’s management or leadership approach, which is focused on autonomy and agency at the team level, does not reflect typical organizational structures, which in many cases still resemble the practices of the industrial paradigm. The question is obvious: Under what circumstance can pure Scrum or Scrum out of the box thrive? The Organizational Ecosystem of Pure Scrum Considering the previously identified constraints, we can say that pure Scrum isn’t just a framework — it’s an organizational philosophy that thrives only in specific cultural environments. Think of it like a delicate plant that requires the right conditions to flourish. In these rare environments, teams operate with trust and openness that transforms Scrum from a set of practices into a living, breathing approach to creating value. The Ideal Organizational DNA The most fertile ground — to stick with the plant metaphor — for pure Scrum exists in organizations characterized by a radical commitment to collaboration and continuous learning. These are typically younger, technology-driven companies where innovation isn’t just encouraged — it’s expected: Software product companies, digital service creators, and cutting-edge research and development teams represent the sweet spot. What sets these organizations apart is their ability to embrace uncertainty. Unlike traditional businesses obsessed with predictability, these companies understand that true innovation requires comfort with controlled experimentation. Their leadership doesn’t just tolerate failure; they see it as a crucial learning mechanism. Size and Structure Matter Pure Scrum finds its most natural home in smaller organizations — typically those under 250 employees. These companies possess an agility that allows rapid communication, minimal bureaucratic friction, and the ability to pivot quickly. The organizational structure is typically flat, with decision-making distributed across teams rather than concentrated in hierarchical management layers. Cultural Non-Negotiables For pure Scrum to truly work, an organization must cultivate: Psychological safety, where team members can speak up without fear,A genuine commitment to empirical process control,Leadership that understands and actively supports Agile principles,Funding models that support iterative, incremental delivery,A cultural tolerance for controlled failure and rapid learning — a failure culture. The Counterpoint: Where Scrum Struggles By contrast, pure Scrum suffocates in environments like heavily regulated industries, traditional manufacturing, stuck in the industrial paradigm of the 20th century, and bureaucratic government agencies. These organizations are typically characterized by: Strict processes focused on resilience,Top-down decision-making, often influenced by politics,Resistance to transparency,Punishment-oriented cultures that discourage experimentation. Examples Where Pure Scrum May Work So, pure Scrum is most applicable in organizations where the complexity of the problem space aligns with Scrum’s emphasis on iterative development and rapid feedback loops; however, organizational context does not introduce constraints that require heavy customization. Here are practical scenarios and industries where pure Scrum can work well: Single-Product Focus in Early Scaling Organizations: Pure Scrum thrives in organizations that have grown beyond the startup phase but are not yet burdened by large-scale portfolio management. For example, a SaaS company with one main product and a dedicated team can effectively use Scrum to focus on continuous delivery while fostering alignment through the framework's inherent transparency.Internal Development Teams in Larger Organizations: Departments or units within larger organizations that operate with clear boundaries and minimal dependency on other teams are also well-suited for pure Scrum. For instance, an innovation hub within a legacy organization experimenting with AI-powered tools can avoid the misalignment issues that often plague scaled Scrum setups.New Product Lines in Established Enterprises: When a larger enterprise launches a new product line with a dedicated, self-contained team, pure Scrum can provide the structure needed to iterate quickly and get airborne to start learning in the marketplace. For example, an e-commerce giant rolling out a subscription-based feature can use pure Scrum to ship incremental changes while keeping the focus on customer feedback and delivery speed.Teams With Minimal External Dependencies: Pure Scrum works best where teams control their destiny — such as product teams that own the entire development pipeline from ideation to deployment, covering the problem and the solution space. For instance, a team building a customer-facing app with its own backend can succeed with pure Scrum, as external delays and cross-team coordination are minimized.Organizations Transitioning From Waterfall to Agile: Pure Scrum is an excellent entry point for organizations transitioning from traditional waterfall methodologies to Agile. By clearly focusing on Scrum’s foundational principles, such as delivering shippable Increments and prioritizing transparency, these vanguard teams can build a strong, agile culture before introducing complexities like scaling or hybrid approaches. The common thread in these examples is autonomy and clarity of purpose. Pure Scrum struggles when faced with dependencies, misaligned incentives, or competing priorities, but it excels when teams are empowered to self-manage, focus on a single goal, and deliver customer-centric value in iterative cycles. Conclusion At its core, pure Scrum is less a project management framework and more a reflection of an organization’s fundamental approach to creating value. It requires a profound shift from seeing work as a series of prescribed steps to viewing it as a continuous journey of discovery and adaptation. The most successful implementations, therefore, aren’t about perfectly following a set of rules but embracing a mindset of continuous improvement, radical transparency, and genuine collaboration. Ultimately, applying pure Scrum may lead to identifying an organization’s idiosyncratic way of creating value, thus abandoning the original framework in the long run, which is fine: We are not paid to practice Scrum but to solve our customers’ problems within the given constraints while contributing to the organization’s sustainability. In all other cases, you will struggle to apply Scrum out of the box in a corporate context; the five constraints sketched about will take their toll and require much “engineering” to utilize Scrum advantages while adapting to organizational requirements. The good news is that an undogmatic, skilled approach to adapting Scrum will avoid creating another botched version of the “corporate Scrum” we all have come to despise.

By Stefan Wolpers CORE

When discussing software design and software architecture, the first question that often arises is: What do these terms mean? Despite their importance in software development, they usually need to be understood, with definitions that vary widely depending on the source. This lack of consensus can make drawing clear lines between them challenging. To clarify these concepts, let's explore their distinctions and interconnections, drawing from some of the most influential books in the field: Just Enough Software Architecture by George H. FairbanksBuilding Evolutionary Architectures by Neal Ford, Rebecca Parsons, and Patrick KuaSoftware Architecture for Developers by Simon BrownPhilosophy of Software Design by John OusterhoutSoftware Architecture: The Hard Parts by Neal Ford and Mark RichardsBuilding Microservices by Sam NewmanFundamentals of Software Architecture by Mark Richards and Neal Ford Architecture vs. Design These books provide a wealth of insights but also present diverse perspectives. These references and your experiences as a developer or architect will shape your understanding of design and architecture. I encourage readers to explore multiple sources, form their definitions, and continuously refine their knowledge. Across these references, several recurring themes and definitions emerge. Let's break them down. Software Architecture High-level Perspective: Architecture focuses on the big picture, integrating components, services, and systems into a cohesive strategy.Strategic: It deals with decisions that are hard to change because of their widespread impact. These decisions often involve trade-offs shaped by the context—team size, business goals, and technology constraints."Why" over "How": As emphasized in Fundamentals of Software Architecture, architecture is as much about the reason behind decisions as it is about the decisions themselves. For example, adopting a monolithic or microservices architecture isn't inherently right or wrong—it depends on the organization's context. The "right" choice is determined by the trade-offs and the strategy that aligns with business and technical goals. Software Design Tactical Focus: Design operates at a lower level, closer to the code, addressing the day-to-day decisions developers face."How" over "Why": Design is more concerned with the specific ways to solve coding problems.Guided by Patterns: Unlike architecture, design often has clearer guidelines for "right" and "wrong." For example, excessive if statements in code are universally seen as a code smell. A solution might involve applying the Strategy Design Pattern to simplify decision logic. A practical example of design would be implementing a connection pool in Java to manage database connections efficiently, ensuring that resource use is optimized at runtime. Interplay Between Design and Architecture One of my favorite perspectives comes from First Principles of Software Architecture, which presents architecture and design as a spectrum. Decisions in one domain inevitably influence the other. Architecture Influences Design: Choosing a microservices architecture will directly impact design decisions. For instance, the inability to perform a rollback across multiple services necessitates patterns like Saga for managing distributed transactions.Design Influences Architecture: Applying Domain-Driven Design (DDD), with its tactical patterns like entities, aggregates, and bounded contexts, shapes the architecture by defining layers and enforcing domain boundaries. This interplay highlights the inseparability of design and architecture. Strategic decisions ripple into tactical implementations, and tactical patterns can reshape the strategic approach. Practical Recommendations As with Domain-Driven Design (DDD), start with the strategic (architecture) before moving to the tactical (design). This approach ensures that your software solutions align with broader organizational goals while maintaining the flexibility needed to adapt to change. Books like The Philosophy of Software Design encourage engineers to view their craft extending beyond the code. This mindset can lead to better integration between architectural and design considerations, ultimately producing software that balances simplicity, scalability, and maintainability. Closing Thoughts The goal of this article isn't to settle on a universal definition of software design or architecture — such a definition doesn't exist. Instead, it provides a lens through which you can understand their differences and connections. Combining insights from leading authors with your practical experiences allows you to develop your nuanced understanding of these foundational concepts. The next time you tackle a complex project, remember: Architecture asks whyDesign answers how And both work together to turn ideas into reality

By Otavio Santana CORE

In the modern world of constant change, change is the only constant for every fast-paced business that must provide a compelling and engaging solution to its target market. There are limitations in the waterfall actualization model, such as the inability to effectively meet the timelines and rush that dynamic customer operations often present. But moving to Agile is more than just a method; it's a new style of thinking about the interactions between systems, processes and technologies in the organization. In this paper, I present the views of how I participated in migrating a global beauty e-commerce brand from traditional e-commerce architecture to a more integrated approach governed by agile principles. But it wasn't just about adopting Agile — it was about redesigning processes, integrating systems, and building up processes in order to create an environment that looks and feels seamless. The Technical Need for Evolution Beyond Waterfall Traditional e-commerce architectures, CRM systems, and dedicated loyalty applications provide the main stack for e-commerce to deliver digital assets. Even if those systems worked, they were not adaptable to the constant loops of development cycles that are typically present now. It was clear, however, that deploying agile approaches in day-to-day practices did not address the more complex architectural issues arising from this embedded infrastructure. Some of the critical pain points resulting from this were: Disconnected Systems: The management of omnichannel experiences and syncing of customer data was inefficient because of limited interoperability between platforms.Bottlenecked Release Pipelines: The rigidity of development processes and vertical dependencies were a major bottleneck in product releases and gave the teams little room to adjust to the changing market.Resource Constraints: The technology stack was expensive in terms of resources, with constant manual updates and a lot of resources consumed to maintain uniformity across the environments. The solution required more than an Agile transformation; it required an engineering reconfiguration to offer the necessary tools for modular systems, CI/CD pipelines, and intra-team real-time collaboration. Addressing Agile Transformation in a Legacy-Driven Ecosystem The migration exercise for the client commenced by reviewing their current systems and ongoing projects. This included assessing the attributes of each digital asset and platform and their capabilities to support Agile integration while not interrupting existing project workflows. The solution was to introduce changes in technical improvements simultaneously with the deployment of Agile techniques so as not to disrupt operational processes. We made the following modifications: 1. Modularity in Infrastructure Legacy systems were transformed and modularized to microservices supported by containerization platforms — Docker and orchestration platforms. This made it possible to perform parallel updating and deploying which also accelerates delivery and minimizes disruptions at the system level while enhancing its scalability and fault tolerance. 2. Data Flow Optimization APIs and middleware solutions such as MuleSoft and Apache Kafka were described as enabling connections between isolated, disjointed systems. These integrations allowed the sharing of data on the heartbeat and improved omni-station activity by plugging customers’ e-commerce websites, clients’ relationship management systems, or customer data platforms. 3. Integrated Toolchains For CI/CD processes, tools such as Jenkins and GitLab were configured and utilized to deploy build and release cycles in accordance with the Agile approach. This enabled consistent and reliable deployment pipelines, allowing maximum manual errors and rapid development time frames because more time was saved in the processes. Solving these basic technical problems allowed organizations to take full advantage of Agile while overcoming the problems associated with the traditional models. How to Prioritize and Re-Engineer Digital Assets for Agile Integration In order to assist the client with such a significant change when many projects were already underway, we adopted the Scaled Agile Framework (SAFe) as it helps in integrating Agile practices within very large enterprises. We modified it to enable gradual implementation of Agile processes so that Agile teams targeted the most critical digital assets while continuing with other essential activities. This custom approach enabled us to roll out development more efficiently without interfering with other processes. The SAFe is a comprehensive methodology designed to help organizations successfully implement Agile practices at an enterprise scale. 1. Developing a Digital Asset and Platform Taxonomy We began with mapping each digital asset, such as product display pages, recommendation engines, and customer profiles, to the customer journey. This mapping was crucial in identifying which assets had the greatest impact on user experience and business value. By understanding these touchpoints, we enabled the client to strategically prioritize assets for re-engineering, ensuring Agile integration efforts aligned with customer needs and organizational goals. To create this robust digital product and platform taxonomy, we leveraged insights from established frameworks and research, following four streamlined steps: Define Objectives and Scope: Clearly articulated the purpose and boundaries of the taxonomy to align with organizational goals.Identify Platform Attributes: Focused on key attributes such as centralization, market sides, and offering orientation to differentiate platforms.Develop a Taxonomy Schema: Constructed a structured framework to categorize platforms based on shared features and distinctions.Validate and Refine: Applied the taxonomy to existing platforms, refining it for accuracy and practical application. This taxonomy enabled the creation of pods dedicated to specific functionalities and aligned them with overarching business goals: Early Wins: Pods focused on front-end components like ratings, reviews, and loyalty rewards interfaces, delivering quick, high-impact results with minimal complexity. These projects improved user experience and built stakeholder confidence, setting a positive tone for the larger transformation. By demonstrating tangible progress early, the team gained the trust needed to tackle more complex initiatives.Complex Integration: Back-end systems, such as the CRM platform and data lake, were tackled later, requiring deep integration to ensure scalability and consistent data across channels. These efforts established the foundation for advanced capabilities like personalized recommendations and real-time analytics. Addressing these core systems ensured the stability and flexibility required to support evolving business needs. Creation of Digital Assets and Platform Taxonomy revealed that ~30 pods comprising of ~270 full-time employees will be required to enable the front-end products for the e-commerce giant. 2. Enhancing Technical Readiness To ensure the organization was technically prepared for transformation, we focused on three key areas: Cloud Migration: Non-critical workloads were transitioned to cloud-based services such as AWS Lambda for serverless compute tasks. This migration reduced reliance on local infrastructure, optimized resource allocation, and lowered operational costs while providing the scalability needed for dynamic workloads.API-Led Connectivity: Legacy databases were modernized by exposing them via APIs, using tools like Postman for testing and AWS API Gateway for deployment. This approach enabled seamless integration into Agile workflows without requiring a full system overhaul, offering a cost-effective and incremental path to modernization.Scalability: Through Infrastructure-as-Code (IaC), infrastructure was codified using tools like Terraform and AWS CloudFormation. This approach ensured consistent environments across development, testing, and production, enabling automated provisioning, version control, and repeatable deployments to enhance efficiency and scalability. 3. Building Multi-Disciplinary Agile Pods To drive the Agile transformation, we adopted and customized the Spotify Model to create specialized Agile pods, designed not just to be cross-functional but also aligned with the technical and business scope of the transformation, with a strong focus on mobile and web-focused digital products. Each pod archetype addressed domain-specific complexities while embedding Agile principles across the entire product lifecycle — from design and development to testing and deployment of various digital assets and platforms for e-commerce. By customizing the Spotify Model, we established autonomous pods operating as squads while integrating a tribe structure to promote cross-pod collaboration. This approach fostered alignment across customer-facing platforms, backend systems, and testing pipelines. It also allowed us to accelerate delivery timelines and drive consistent value creation, ensuring the Agile transformation aligned with the client’s strategic goals. In addition to their technical specialization, each pod archetype was tailored to the client’s operational needs and designed to include a balanced mix of roles spanning product, digital, and technical domains. This composition ensured that every pod could address the unique challenges of its domain while contributing to the broader transformation objectives. Product Roles: Product Owner, Business Analyst, Domain SMEDigital Roles: Scrum Master, Customer Experience Specialist, UI Designer, UX DesignerTechnical Roles: Tech Lead, Software Developer, Platform Engineer, Solution Architect, Testing/QA Engineer Agile Pod Type Focus area and tools used Example Agile roles Testing Pods Ensuring quality and seamless delivery by focusing on comprehensive testing and validation, Testing Pods play a critical role in Agile transformations. Their responsibilities include developing automated test cases, maintaining test pipelines, and ensuring CI/CD workflows are efficient and reliable. These pods use tools like Selenium, JUnit, and Postman to automate and manage testing processes, while frameworks such as Test-Driven Development (TDD) and Behavior-Driven Development (BDD) guide their approach to creating robust and effective testing strategies. Product OwnerScrum MasterTech LeadTesting/QA EngineersTest Automation Specialists Digital Assets Pods Driving innovation in customer-facing products such as websites and promotional systems. Focused on enhancing user experience, implementing marketing campaigns, and delivering dynamic content platforms. These pods are built after the concept of Digital Product Pods, which emphasizes cross-functional collaboration to deliver impactful digital solutions and enable rapid prototyping and iterative improvements. Their work often includes leveraging user data to personalize experiences, optimizing platforms for scalability, and ensuring seamless integration with other business systems. This approach not only accelerates time-to-market for digital initiatives but also drives meaningful engagement and measurable business outcomes. Product OwnerBusiness AnalystScrum MasterTech LeadSolution ArchitectFrontend/Backend Engineers Platform Pods Supporting scalable infrastructure by managing backend systems, cloud platforms, and data pipelines. Ensures system reliability, high availability, and performance optimization. We implemented the Infrastructure as Code with Agile principles to create these pods, automating infrastructure management and enabling dynamic scalability across enterprise platforms. These pods develop and maintain key platforms such as Kubernetes-based container orchestration systems, cloud storage solutions like Amazon S3, data streaming tools like Apache Kafka, and CI/CD pipelines powered by Jenkins or GitLab. By providing a robust foundation, these pods enable seamless operations and empower other teams to innovate without infrastructure bottlenecks, driving enterprise-wide digital transformation. Product OwnerDomain Subject Matter Expert (SME)Scrum MasterTech LeadPlatform EngineerCloud SpecialistDevOps EngineerSite Reliability Engineer (SRE)Database AdministratorSecurity Engineer The Digital Assets Pod included experts in product design and development who worked collaboratively to deliver intuitive and scalable customer-facing platforms. Meanwhile, the Platform Pod supported the infrastructure needs of these systems, enabling seamless performance under dynamic business conditions. 4. Bridging Resource and Capability Gaps for Transformation Role Distribution and Expertise To address resource gaps effectively, we conducted a detailed assessment of role distribution and expertise for the client, ensuring alignment with their strategic objectives. Drawing insights from industry best practices in resource planning and team structuring, we recommended retaining critical roles such as Solution Architects and Domain Subject Matter Experts (SMEs) in-house to safeguard long-term innovation and ensure continuity of institutional knowledge. For mid-tier technical roles like Software Developers, a selective insourcing approach was adopted, prioritizing project-specific needs to ensure expertise in key areas of development. To optimize resource allocation further, we advised outsourcing standardized tasks such as Testing and QA engineering, leveraging external vendors to handle repetitive but essential work efficiently. This balanced approach drew upon frameworks outlined in resources like the PMI Talent Triangle and Agile team structuring principles, enabling the client to maintain agility while addressing resource constraints strategically. Upskilling for Agility To build agility within the organization, we focused on upskilling internal teams with targeted training on Agile principles, CI/CD tools, and API integrations. Certification programs were introduced to enhance technical capabilities, including container orchestration with the Certified Kubernetes Administrator (CKA) and cloud architecture skills to align with the demands of the transformation. Additionally, team members pursued Agile-focused certifications such as the PMI Agile Certified Practitioner (PMI-ACP) and Certified ScrumMaster (CSM) to deepen their understanding of Agile methodologies and enhance their ability to drive iterative, value-driven development. Conclusion: A Technical Blueprint for Digital Transformation This experience highlights the critical need to address both methodological and technical challenges in Agile transformations. For the global beauty e-commerce client, success depended on rethinking workflows and reengineering the foundational systems that support them. By implementing modular architectures, automating release processes, and aligning cross-functional teams, the transformation delivered faster product rollouts, improved customer experiences, and a scalable technology foundation for future growth. Organizations embarking on similar journeys should treat Agile transformation as a holistic effort, balancing immediate wins with long-term investments in technical infrastructure. For insights into managing complex transitions, Large-Scale Scrum: More with LeSS by Craig Larman and Bas Vodde provides practical strategies for scaling Agile effectively while addressing organizational challenges.

There’s a common career progression in this technical industry. You come in wet behind the ears as a junior developer, whether front-end, back-end, infrastructure, or even security. After a few years, you lose the “junior” from your title, and after a few more years, you gain a “senior.” You might become a team lead after that with some light duties around oversight and overall team delivery. But then you’re promoted into some role with "manager" in the title — engineering manager or software development manager or something similar. Suddenly, you’re in a position where it’s pretty easy to prove out the Peter principle. Wikipedia summaries this principle, coined in a 1969 book, as follows: “The Peter principle is a concept in management developed by Laurence J. Peter which observes that people in a hierarchy tend to rise to 'a level of respective incompetence': Employees are promoted based on their success in previous jobs until they reach a level at which they are no longer competent, as skills in one job do not necessarily translate to another.” Skills as a successful software engineer in no way prepare you for the duties of management —and not just any kind of management, but people management. There are so many ways this goes wrong that a million and one books, articles, training programs, and Substack posts are out there to help people figure out how to become better managers. So why this article — and why me? Though I’ve worn many hats in my career, for most of it I should have had the title “reluctant manager.” Management?! I’d rather slit my throat. Can’t I just do it all myself? Maybe you’re not like me and have always wished for minions around to do your bidding, but you’ve found they’re a little harder to, well, manage, than you expected. I’m Amanda Kabak, Founder of Verdant Work, and I’ve been in the industry for over 25 years, spending the last decade of that in startups and ending as the CTO of a clean-energy software company. I’ve built and managed teams with limited resources and varying degrees of naivete, but I learned how to lead disparate groups of people to repeated successes in delivery while maintaining good retention and a sunny disposition. This series of articles is specifically geared toward technically minded people who have found themselves in the limelight of team or department management and need some straightforward and pragmatic help to excel in this role as you had done as an individual contributor. What Is Management? One of the core tenets of all effective management is clear communication, so let’s make sure we’re all in agreement as to what we mean when we talk about management. For me, if I’m feeling cheeky, which I usually am, I think of management as the title of this article: "Getting Shit Done Without Doing It Yourself." But, seriously, Merriam Webster so helpfully defines it as “the conducting or supervising of something (such as a business).” What about “facilitating the work and professional maturity of those who report to you”? Is everyone familiar with Maslow’s hierarchy of needs? In management, I focus on the bottom two layers: physical needs and safety, with a touch of the next one, belonging. I have to admit that physical needs and safety are partly metaphors, but they’re really good metaphors as well as being concretely accurate as well. Physical needs: do they have the right equipment? Do they feel comfortable talking to people inside and external to the team? Do they have proper bandwidth and aren’t freezing for sweltering in an out-of-the-way corner of the office? Are they sick and need to take the afternoon off? It’s as easy as that. Ask how they’re doing. Introduce them around. Sit in on a few meetings. Make them log off when they’re hacking up a lung or have brain fog from allergies or sleep deprivation. Now safety, on the other hand, is a little more complex. There’s a physical component to it, yes — but I’m largely concerned with a feeling of security. Can they ask questions? Do they feel empowered to make decisions? Do they fear punishment for doing something wrong? This kind of safety is key to continued performance and growth. Finally, belonging. This one is less concrete than the other two, but the other two enable it to happen, just like in that triangle. If you want a hokey word for it, I would say that we’re looking for synergy: getting more out of individual parts when they’re together. Do you facilitate a feeling of camaraderie on the team? Do team members actively ask for help from each other or you? Do you acknowledge wins and learn together from losses? I’ll touch on this more later, but, remember, you can’t have this if you haven’t met physical needs and safety first. Context Everything in life has a context, which means nothing happens in a vacuum — everything happens within, next to, or during something else. In literature, there are schools of criticism that not only dive deeply into the words on the page (the what) but into what was going on while it was written, the politics in play, the social mores, the epidemics, and wars. This is all context — and guess what? When you give someone a task, there’s a context to it, whether you divulge it or not, and this context is critical for everyone to understand. “What,” alone, is insufficient. Think about the game of Clue, if that’s not dating me too much. Colonel Mustard in the Conservatory with the candlestick. For a story to be complete, it’s not just about what. It’s about who, where, and most importantly, why. What is the business perspective of the task you’re assigning? Who are the customers that need this feature? Why do they need it? How do our competitors handle it? Where does it sit on the larger roadmap? How does it potentially relate to revenue targets? In many cases, users pay your salary, and you and your team are likely one of the most expensive things in the company. Don’t forget them, and don’t let your team forget them. Is everyone clear on how this task relates to others that are happening at the same time? Do they understand where this piece of functionality fits into the larger flow? Are they aware of upstream or downstream integrations they may have to take into account for timing and compatibility? Think about an assembly line: each step is dependent on the step before it is completed successfully and to specification, and the step after this one expects a certain outcome before it can begin. Another kind of context is in your team’s history. It can help to point out things you’ve done before that are similar as well as those that are different. These can provide concrete frames of reference for the work you’re assigning. Document the Details Details are an essential part of good communication. First and most importantly, what is in or out of scope? Let me tell you something: scope is not like Schrodinger’s Cat. You need to know what’s in that box before you open it. If it’s your team member’s job to find out, then that’s a task in and of itself. And, guess what? What is in or out of scope needs to be documented somewhere outside your head, their head, or the business owner’s head. What do I mean by document? I mean something tangible and complete. I don’t care if it’s a Post-It note or a printed and bound novel, though I would caution against both. The key here is that multiple people can point at it and be looking at the same thing. But be cautious about handing over dense blocks of text. Diagrams are awesome, maybe throw in a table or two where it makes sense. Bullet points can help delineate text to make it more digestible; numbering the list facilitates fast referencing for discussion. Look at these two pages. Which do you think is easier for people to digest? Right. The second one. Bullet points are powerful, images are even more so. You know the old adage that an image is worth a thousand words? This is especially true in requirements. You don’t want everyone to have a different picture in their mind; the only way to avoid that is to put the picture on the page. Whatever you do for your documentation and wherever you store it — your GDrive, SharePoint, ADO, Jira, GitHub, the closest conference room — be consistent. Define a set of standards and then use them. Standards not only help organize, but they make the content easier to absorb and can help you ensure everything is complete. If you’ve got a blank space for security concerns in your template, you might remember to ask about it. . . and it might even get implemented. The Definition of Done For any task, you need to know what you’re doing and hopefully why, but you also need to know when you’re finished. Done is this nirvana that never lasts, but for the brief moments we’re there, everything can seem worth it. You could think of it like a home improvement project you’re contracting out. If the contractor wants 50% in the middle of the project and the rest when it’s complete, you’re all going to agree that once the plumbing is roughed in and the tile is up, you’ll pull out your checkbook. That 50% mark has to be defined. The definition of done can be thought of as just another tool of communication. Why are increments of doneness important — besides contractors getting paid? They enforce agreement. If everyone knows where a task is supposed to end before it starts, we can all focus on the work and not on trying to patch up mismatched expectations. Let me say it this way: if someone doesn’t know what’s expected of them, how can they possibly succeed?They enforce process. If 75% of your tasks have a common set of items that must be completed — tests, for example — you can automate these requirements and make your tool require them. People will get used to including them pretty quickly if the PR can’t be made or the build fails. Steps of review and approval should be in place where appropriate.They promote quality. Think of what doneness should look like for this task. What about tests? What about documentation? What about review? Think of all those things that we say we’ll do later “when things slow down.” What if we require them from the start? Because it takes too long? How long does it take for us to fix integration or production bugs or not understand why something was implemented in a certain way?They communicate progress. If something is actually done, meeting all the criteria of doneness, you can. . . call it done! You can communicate that this nugget of business value has been achieved, and you can see where you really are in your project because of it. If something is almost done or kind of done or done except for this one thing, guess what? It’s not done, and you shouldn’t claim credit for it, no matter how much you might want to. It’s the painful truth. Finished, done, is a binary state, which means if it’s not yes, it’s no.They enable a feeling of success. If our work overall is never done, when can we ever have that pizza party? I’m serious. If we are on a hamster wheel of never-ending sprints and continuous releases, when are we supposed to feel satisfied and take a breath? I’m serious about this, too. A long grind induces burnout, which brings about turnover, which loses the company money on retraining and kills days of your time in interviewing and onboarding.They provide moments of reflection. If something goes wrong with a task, everyone knows it by the time you get to done, and you can disseminate that knowledge across the team quickly and make adjustments so it doesn’t happen again — even in the same project, maybe even in the same Sprint. If something goes really right, it’s an opportunity to see it and respond to it right away. So much goodness in true doneness. That’s why one of the most important aspects of this is that you cannot compromise on your definition of done. You just can’t. If the buck stops at you, why would you want to? Because you’re under pressure? Who isn’t? Making sure things are done the right way by people on your team is a lot of how you add value to your company and earn your salary. What makes it easier to do is knowing how critical it is and what happens when you start to compromise. Incremental Conclusion There’s more to management that I’ll cover in the next installation in this series, but let’s review what I’ve covered here before moving on with your day. We defined management as “facilitating the work and professional maturity of those who report to you,” and showed how it could be considered in terms of Maslow’s hierarchy of needs. We then dove into the concept of context, all that stuff that exists around individual tasks that is critical to communicate to your implementers. Documentation, especially in terms of details, came next with some ideas about how to organize and communicate things on the page or screen. Finally (and fittingly), we covered the definition of done and how having one is critical to almost every aspect of delivery. Stay tuned for the next article, which will cover deadlines and protecting your team’s time, understanding the handoff points of your tasks, communicating doneness over time, promoting dialog through questions, and overall mentorship.

History knows a lot of examples when brilliant ideas were conceived in a garage and started in a somewhat chaotic manner. But it also knows just as many examples of equally brilliant ventures failing because of simple mistakes and a general lack of a systematic approach. I suggest you have a look at four basic steps that can get you some decent insurance against chaos. Get yourself and your team through them — and build your project’s foundation layer by layer. And remember: Amat Victoria Curam, Victory Loves Preparation. Question 1: Who Needs Your Project? Identifying your target audience is a critical step that should never be skipped. Without this, your project is at risk of failure even before it starts. You may ask why defining a target audience is of any matter for a developer. The answer is simple and straightforward. Your audience is the fundamental factor that defines everything else, from technology stack to product features. When you know who is going to use your product and how they will use it, you can optimize it accordingly. For instance, if you're building a progressive web app (PWA), the ability to use offline functionality (when there's no internet on the phone) is important, since many users will be using it with an unstable internet connection or even without it. It shouldn't just give you a "white screen": you should at least warn the user that an internet connection is required or give them the option to do something without one. Or, if you know your users’ busiest time slots, you can design a robust system to handle peak traffic without spending too much to make it bulletproof 24/7. All in all, without understanding your audience, you are facing all imaginable risks: from technical setbacks to developing something that sees little demand. So, how can you understand who your users are? Start by analyzing competitors. Look at their audience, their social media, the blogs, and forums where their products are discussed. Notice which demographics engage most, and read their feedback. There may be unmet needs you can address in your own project, potentially drawing users to what you have to offer. If your product is truly unique, there are still ways to identify your target audience. For instance, surveys can gauge interest and help you accumulate valuable data across user groups. Create short, focused surveys with questions about people’s needs, interests, and preferences. Post these on social media, send them via email, or use survey platforms like Google Forms or SurveyMonkey. Run test ad campaigns, targeting different user groups to test their interest and draw their attention. This will show you which audience segments respond best to your ideas. You can also create "personas" – user profiles that include age, gender, interests, profession, goals, and challenges. Personas can help you refine messaging and prioritize features for different segments. Question 2: Why Do THEY Need It and Why Do YOU Need It? Now that you know your target audience, the next step is determining whether they truly need your project. This actually comes down to a simple question: why would they need it? To explore this, ask yourself, your team, and your probable audience several key questions: What challenges do your potential users face daily and what specific problems does your project address for users?Why aren’t existing solutions meeting their needs? Competitors’ products may be inconvenient, expensive, or complicated. Knowing their weaknesses helps you create a unique value proposition.How exactly is your project going to make users’ lives easier and what user experience do you aim to deliver? For example, it could automate routine tasks, improve customer interactions, or provide access to valuable information. Understanding the product’s primary tasks enables you to focus on the core functions that matter most to users. If you’re also managing business or marketing tasks, answering these questions can help you devise a strategy that speaks to your audience. In some cases, it may also help you persuade investors or partners of your project’s potential value. There is also the other side of the goal topic that is equally important for your future project. Besides defining user-oriented goals, you should also ask yourself and your team what the project goals are from your own perspective. In other words, why do you need this project? What are its business goals? Do you plan to keep it small, but beautiful? Or is your ambition to outshine famous unicorns? Is it intended just to generate some income to support yourself and your team, or do you plan to attract additional investments and scale it up? Knowing these objectives is crucial to keep yourself and your team highly committed. It would also help you draft your monetization strategy (if any) and shape your marketing and growth patterns. But most of all, it will give you a roadmap of what is truly important so you can focus on worthy priorities. Question 3: How Are You Going to Develop the Project? Sometimes, a project’s success hinges on choosing the right technologies. Stability, scalability, and maintenance of your product — and especially the development costs — all depend on selecting suitable tools. This makes it essential to determine the technologies you are going to employ. Here’s how to filter down your tech stack selection: Step 1: Define Your Project Goals and Requirements Before choosing technologies, clarify how the product will be used, the expected number of users, data processing and storage needs, and the devices it is going to support. At this point, certain technologies may already seem more suitable than others. Step 2: Assess Your Development Team’s Competencies Choose technologies that align with the team’s expertise. If the team is familiar with specific frameworks or programming languages, leaning toward those can save time and resources. When developers are well-versed in a technology, the likelihood of bugs and delays significantly decreases. Step 3: Consider Project Timelines Even if your team is proficient in a particular technology, other options might allow faster, more affordable development. It’s also essential to account for the project’s future growth: popular, well-supported technologies reduce the risk of issues with outdated solutions. Question 4: Do You Have Everything Prepared? Web project success relies not only on technology and functionality but also on the team’s effectiveness. So, before beginning development, it’s crucial to ensure all technical and organizational resources are ready and configured. Here is a typical checklist that you should run through before you dive into a development frenzy: Set up a task board and workflows. Create a Kanban or Scrum board, define work stages (e.g., Backlog, In Progress, Code Review, Testing, Done), and assign tasks. Make sure everyone knows their roles and task deadlines.Organize chat channels. Create project channels in a messaging app like Slack to facilitate instant communication.Establish repositories on GitHub/GitLab/Bitbucket and set up the basic project structure. Ensure all team members have proper access and understand the branching strategy. Require reviews before merging to minimize errors and maintain code quality. Configure main branches (e.g., ‘main’, ‘develop’, etc.) and add protection policies to prevent direct pushes.Set up Docker images for all project components (frontend, backend, databases, services) and use Docker Compose to streamline local development.Implement CI/CD systems (e.g., Jenkins, GitLab CI/CD, CircleCI). Automate code builds, testing, and deployment. Prepare unit and integration tests to run on every commit. Automate deployments for each development stage.Create a Wiki or an internal knowledge base (e.g., Confluence, GitLab Wiki). Document all project information, architecture, requirements, deployment instructions, and development standards in one location. If possible, automate documentation updates (e.g., generating API documentation with Swagger) to keep your knowledge in line with the development progress. Love and respect are probably the most important keywords for any human life. When it comes to work, it is crucial to love and respect what you do, care for the people you work with, and also for your users and their needs. But you should also never forget about loving and respecting yourself and your own ideas. Good preparation is ultimately a natural tribute to these emotions. Your project is your brainchild, so it deserves to be planted into a properly arranged environment. And you deserve to see it prosper and live happily ever after.

By Filipp Shcherbanich CORE

When we think about software engineers, the focus often lands squarely on technical skills — writing efficient code, solving complex problems, and understanding algorithms. However, this narrow view overlooks a critical element that can make or break a career: personal branding. This oversight is a mistake I made early in my career. I believed my technical abilities alone would lead to success, promotions, and recognition. But over time, I realized that while being skilled at software design and architecture is essential, it is only part of the equation. Why Personal Branding Matters Have you ever been passed over for a promotion or a dream project, only to see someone less skilled take the opportunity? If so, you may have wondered what set that person apart. The difference often lies not in technical expertise but in visibility, reliability, and connections. Even foundational technical books hint at this truth. In The Philosophy of Software Design, John Ousterhout emphasizes the importance of strategic thinking — skills beyond coding. Similarly, Eric Evans' classic Domain-Driven Design delves into strategic DDD, where concepts like ubiquitous language and bounded contexts highlight the importance of human interaction and communication. This reality becomes even more apparent when considering the Agile Manifesto, which prioritizes individuals and interactions over processes and tools. Software is created with, for, and through people. Connections and perceptions matter just as much as code quality because, ultimately, the people you work with shape your career trajectory. What Is Personal Branding? Your brand is the perception others have of you. It's how colleagues, managers, and the broader industry view your reliability, expertise, and approachability. Personal branding involves defining and promoting what you stand for as an individual. It reflects your experiences, skills, values, and unique traits. For software engineers, personal branding is crucial because it influences career growth. Your reputation impacts hiring decisions, promotions, and project allocations. As the ancient saying goes, "Caesar's wife must be above suspicion." Perceptions have shaped decisions for centuries, and this principle holds even in the tech-driven modern world. Benefits of Personal Branding Investing in your brand offers numerous benefits: Improved Credibility: Demonstrating your expertise and knowledge establishes trust among peers and stakeholders.Differentiation: Highlighting what makes you unique sets you apart from others in your field.Lasting Impressions: A well-defined brand ensures you are remembered for the right reasons.Better Connections: Strong branding attracts like-minded professionals and opens doors to valuable opportunities. How to Start Your Branding Today Building a personal brand doesn't require reinventing yourself or creating a façade. Instead, it's about curating and amplifying your authentic self to showcase your strengths. Here's how to begin: Understand Yourself: Reflect on your skills, values, and goals. What do you want to be known for? What are your long-term career aspirations? Defining your brand starts with self-awareness.Audit Your Existing Brand: Search for your online presence. What appears when you do? Are there any inconsistencies in how you are represented across different platforms? This audit can help you identify areas for improvement.Define Your Audience: Personal branding isn't about appealing to everyone. Identify the people who matter most in your career — managers, colleagues, domain experts, or potential employers — and tailor your brand to resonate with them.Leverage Written Content: Write articles, blogs, or social media posts to showcase your knowledge. Sharing your expertise not only boosts your reliability but also expands your reach.Be Active in Your Network: Build relationships with colleagues and industry peers. Being approachable and helpful creates a positive impression, often leading to new opportunities.Maintain Consistency: Whether through LinkedIn, GitHub, or personal blogs, ensure your online presence aligns with your professional goals. Use the same tone, messaging, and visuals across platforms.Remember Non-Technical Audiences: Not everyone influencing your career will understand the technical aspects of your work. Build trust with non-technical stakeholders — managers, HR, and directors — by communicating effectively and showcasing your value.Enhance Word of mouth: Be generous with your knowledge and support. A helpful reputation spreads, creating opportunities through recommendations and referrals. Real-Life Scenarios for Developers Building a Personal Brand As developers, our work often goes unnoticed unless we actively showcase it. Personal branding helps highlight your skills, connect with others, and create career opportunities. Here’s how to start: Open Source Contributions: Imagine you're contributing to a popular open-source project. Your commits and pull requests improve the codebase and showcase your expertise to a global audience. This visibility can lead to job offers, speaking invitations, or collaborations.Tech Blogging: A developer who writes about solving a tricky bug or implementing a complex feature can establish themselves as an expert in that area. For example, a blog post detailing how you optimized database queries in a high-traffic application could resonate with others facing similar challenges.Conference Speaking: Sharing a personal project or a unique approach at a developer conference boosts your confidence and positions you as a thought leader in your field. A talk on how you integrated cutting-edge tools like Kubernetes with CI/CD can inspire others.GitHub Portfolio: Developers often share side projects, frameworks, or libraries on GitHub. Imagine creating a tool that simplifies a common development pain point — like automating documentation generation for APIs. Your GitHub stars and forks become a testament to your innovation.Social Media Engagement: A thread on Twitter about debugging a complex issue in JavaScript or a LinkedIn post about lessons learned while scaling a microservices architecture can attract attention from peers and recruiters.Code Reviews and Mentorship: Providing thoughtful, constructive feedback during code reviews or mentoring junior developers showcases your leadership skills. It builds your internal brand within a team or organization.Live Coding or Tutorials: Hosting live coding sessions on platforms like YouTube or Twitch to solve problems, build apps, or explore new technologies demonstrates your technical skills and communication ability. By embracing scenarios like these, developers can naturally weave personal branding into their daily lives, allowing their expertise and passion to shine authentically and impactfully. Final Thoughts In software engineering, your technical skills are just the foundation. To truly stand out, you must cultivate a personal brand that amplifies your strengths, builds trust, and opens doors. By understanding yourself, creating valuable content, and fostering connections, you can shape how others perceive you and take control of your career narrative. Start today — your future self will thank you.

By Otavio Santana CORE

Implementing agile methodologies during the development of software applications has become an industry norm today. They allow the teams to develop better products through iterative cycles with the help of feedback. However, it should be mentioned that Agile has actually reshaped project management for the better, and at the same time, it is actually rather complex to master its processes. It is capital at its finest, which brings AI into play here. Everyone knows that artificial intelligence has revolutionized the way we execute Agile and makes our processes more efficient, adaptive, and data-driven. It is high time we attempted an understanding of how AI enhances important Agile phases. Improving Forecast Accuracy in Sprint Planning The whole development cycle begins at this stage. Earlier, teams depended on their previous experiences and guesswork to figure out the amount of work they may be required to do in an upcoming sprint. However, this often leads to many mistakes. It causes teams to bypass deadlines or rush their work towards the end of the sprint. AI algorithms are adequate for examining large sets of data. It makes them a unique fit for sprint planning. The following pointers highlight the way AI helps to carry out this process: Historical Data Analysis AI studies previous sprints and keeps track of the time duration of certain tasks. Furthermore, it helps spot problems that team members might not see right away. Predictive Task Estimation AI tools foresee the duration of tasks and the amount of effort required. These are derived from factors such as historical data, customers’ experience, and skills and strengths of the team involved. This approach to sprint planning eliminates much of the guesswork involved in estimations as compared to the previous approach. Capacity Management AI assists teams in making more informed guesses about how much and what amount of throughput and capacity each team member possesses. It incorporated a number of things, which include speed, time when they are not working, and what they are capable of. This means that workloads are more likely to be evenly distributed, and there is less stress. By using AI to fine-tune sprint planning, teams can steer clear of under- or over-committing. This solves a common pain point in Agile practices. Intelligent Backlog Prioritization One of the unending tasks for product owners and development teams is selecting what to include in the product backlog. Previously, most individuals applied their judgment concerning what some of their employees should do next, which features to implement, which bugs to fix, or which enhancements to prioritize. Nevertheless, this process-oriented approach has the negative implication of being biased because of the lack of information or excessive pressure from the stakeholders involved. AI supports teams in making better-informed choices about backlog prioritization: Predictive analytics for feature impact: AI tools analyze customer feedback, market trends, and product performance data to foresee which features will offer the maximum value. It allows teams to give preference to features that match business objectives and customer requirements. Dynamic backlog re-prioritization: AI tools continuously check and adjust priorities as new data comes in. It includes customers' points of view, rivals' moves, or the information numbers reveal. It makes sure the product backlog always aligns with the newest facts. Risk assessment: AI compares what risks may be associated with backlog items, such as how difficult it may be to implement a feature or how prone it is to bugs. This helps teams balance innovation and caution. The automation and objectivity that AI brings to backlog handling help cut out the guesswork. This lets teams zero in on delivering features that make the biggest splash. Improving Collaboration and Productivity With AI-Enhanced Standups Daily stand-ups play a key role in Agile methods by helping teams stay in sync. Stand-ups work well, but they often turn into status reports, which leave little time to tackle problems head-on. AI-powered smart tools can make these stand-ups better by: Tracking issues in real time: Smart tech can give quick updates on code quality, build status, and sprint progress. This gives the team a clear view of the project without going around the room for updates. Spotting roadblocks: By watching ongoing tasks and interactions, smart tech like AI can find potential roadblocks or areas where team members might need help. It can flag these issues before they grow into big problems. Automating meeting summaries: Smart transcription and analysis software can sum up main points from daily check-ins and send them to the team. This helps everyone stay on the same page without having to write things down themselves. By making daily check-ins more actionable and quicker, AI can help teams spend more time solving problems and working together. This makes this key Agile practice even better. Reducing Time and Enhancing Coverage with Automated Testing and QA Testing is a crucial step in Agile that can often slow things down in bigger projects. It is very time-consuming to do manual testing, and even with the more automated test sets, a great deal of work is required to maintain momentum. AI can drastically improve both the speed and quality of testing in Agile: Automated test case generation: AI looks at the code and creates test cases on its own. These cover more edge cases and scenarios than old methods. It ultimately boosts test coverage without people having to step in. Predictive bug detection: Machine learning tools predict areas where various glitches may pop up. They take into account past data, the complexity of the code, and the related history. It allows teams to focus their testing on parts that are more likely to have defects. Self-Healing test scripts: AI fixes test scripts by itself upon the most minute changes in the codebase. This cuts down the time developers need to spend on maintaining tests. By accelerating the testing process and improving bug detection, AI helps Agile teams maintain their velocity without cutting corners on quality. Real-Time Progress Tracking and Reporting Agile teams need constant feedback to fine-tune their strategies. But keeping tabs on progress and making reports takes time and often lacks real-time accuracy. AI can make this job easier by: Automated reporting: AI makes progress reports in real time using current sprint data. This gives product owners and stakeholders a clear view of the project's status. Advanced analytics for velocity tracking: AI-powered tools perform in-depth analysis of the way teams perform. They compare speed across sprints or spot patterns in task completion rates. It helps teams use this data to improve their process. Sentiment analysis of team feedback: NLP gathers team feedback and notes from retrospectives to spot trends. It learns the extent of happiness or satisfaction of the team. Also, it helps leaders manage potential problems head-on. AI-powered reporting tools eliminate the need for manual progress updates. This gives teams more time to focus on work that adds value while keeping everyone across the organization in the loop. Smarter Retrospectives The Agile retrospective plays a key role in continuous improvement. However, retrospectives are often influenced by personal experiences, which results in incomplete or biased insights. AI can change retrospectives by: AI-Powered insights: AI has the ability to give feedback based on data about how sprints are going. It looks at things like code quality, how many bugs there are, and how fast work gets done. This gives teams real facts to talk about when they look back on their work. Sentiment analysis: Tools that use AI analyze the tone in which the team talks to each other, pull requests, or comment in project tools. This helps figure out if the team is happy or frustrated during the sprint. It helps detect problems that people might not say out loud. Actionable recommendations: After teams talk about what happened in the sprint, AI tools can come up with ideas for things to do better next time. It makes sure that teams do not just talk about all the things to improve but also have a plan to make changes. AI has an influence on retrospectives that make them more neutral, practical, and linked to measurable results. This helps teams in continuously refining their Agile practices. AI as a Catalyst for Agile Excellence Introducing AI into Agile opens up new possibilities for the software development teams. While AI improves Agile’s strengths, it also solves some of the inherent issues of Agile at the same time. It helps with the planning of sprints, contributes positively to testing, and is a great way to monitor progress as it happens. When teams use AI, they work more effectively and produce better goods and services. Finally, they place more value on users and stakeholders. As AI tools get better and easier to use, they're going to have a big impact on how Agile works in the future. For developers and teams who always want to get better, combining AI with Agile gives them amazing chances to come up with new ideas and do well!

The LangChain framework is an incredibly powerful tool that significantly accelerates the effective use of LLMs in projects and agent development. The framework provides high-level abstractions that allow developers to start working with models and integrate them into their products right away. However, understanding the core concepts of LangChain, such as the architecture of Runnable, is extremely beneficial for developers building LLM agents and chains, as it provides a structured approach and insight into utilizing the framework. The Basis of LangChain Architecture The Runnable architecture in LangChain is built on the principles of the Command Pattern, a behavioral design pattern that encapsulates requests as objects. This design facilitates parameterization, queuing, and dynamic execution of commands, making Runnables modular, composable, and manageable in various workflows. Runnables are particularly well-suited for workflow management, sequential task execution, handling conditional logic, and interacting with external systems. They deliver flexibility, reusability, and modularity. You can dynamically chain tasks together to create complex behavioral scenarios while maintaining a clean and manageable code structure. One of possible configurations of Runnable chain Most high-level objects in LangChain that perform specific tasks implement the Runnable class. Any objects you plan to include in a chain must also implement the Runnable class in some capacity. Interestingly, Runnable serves as an abstraction for a command, a concrete command, and simultaneously acts as both the invoker and receiver. A notable example is a pipe method available in this class, which is specifically designed for creating chains. This method allows seamless composition of multiple Runnables, making it a cornerstone for structuring and executing workflows within LangChain. In the diagram above, you can see how Runnable operates in conjunction with its various implementations, which we will examine in detail throughout this article. Creating Runnables Practically, there are two ways to create a runnable: through RunnableLambda or by extending the base Runnable class. Using RunnableLambda for Simple Functions The easiest way to create a Runnable is by using RunnableLambda. This class lets you wrap any function as a Runnable, allowing dynamic behavior without the need for custom classes. TypeScript import { RunnableLambda } from "@langchain/core/runnables"; // Define a simple function const toUpperCase = (text: string): string => text.toUpperCase(); // Wrap the function as a Runnable const upperCaseRunnable = RunnableLambda.from(toUpperCase); // Invoke the Runnable const result = await upperCaseRunnable.invoke("hello world"); // Output: "HELLO WORLD" Extending the Runnable Base Class For more advanced use cases, you can extend the Runnable base class. This approach provides full control over the execution lifecycle, including methods like invoke, batch, and stream. TypeScript import { Runnable } from "@langchain/core/runnables"; class GreetUserRunnable extends Runnable<string, string> { lc_namespace = ["GreetUser"]; onStart(data: { input: string }) { console.log(`Starting with input: ${data.input}`); } onEnd(data: { result: string }) { console.log(`Finished with result: ${data.result}`); } onError(error: unknown) { console.error(`Error occurred: ${(error as Error).message}`); } // Custom execution logic async invoke(name: string): Promise<string> { this.onStart({ input: name }); try { const greeting = `Hello, ${name}!`; this.onEnd({ result: greeting }); return greeting; } catch (error) { this.onError(error); throw error; } } } Building Workflows With Runnables The Runnable architecture in LangChain is extended with specialized Runnables grouped by functionality, making it versatile and suitable for a variety of applications. Routing and Branching Runnables that manage execution flow based on conditions or input: RouterRunnable Directs input to specific Runnables based on a key, similar to a switch-case statement. Useful for dynamic task execution based on runtime parameters. TypeScript import { RouterRunnable, RunnableLambda } from "@langchain/core/runnables"; const router = new RouterRunnable({ runnables: { billing: RunnableLambda.from((query: string) => `Billing Department: ${query}`), technical: RunnableLambda.from((query: string) => `Technical Support: ${query}`), general: RunnableLambda.from((query: string) => `General Inquiry: ${query}`), }, }); // Route a billing question const result1 = await router.invoke({ key: "billing", input: "I have a question about my invoice." }); // Output: "Billing Department: I have a question about my invoice." // Route a technical issue const result2 = await router.invoke({ key: "technical", input: "My internet is not working." }); // Output: "Technical Support: My internet is not working." RunnableBranch Executes a specific Runnable from multiple options based on conditional checks, allowing the workflow to adapt to different input scenarios. TypeScript const branch = RunnableBranch.from([ [ (user: { age: number }) => user.age < 18, RunnableLambda.from((user) => `Hey ${user.name}, check out our new teen collection!`), ], [ (user: { age: number }) => user.age >= 18 && user.age < 30, RunnableLambda.from((user) => `Hi ${user.name}, explore our trendy outfits for young adults!`), ], RunnableLambda.from((user) => `Hello ${user.name}, discover our premium range!`), ]); const result = await branch.invoke({ name: "Alice", age: 25 }); // Output: "Hi Alice, explore our trendy outfits for young adults!" Data Manipulation and Assignment Runnables that transform or prepare data for subsequent tasks: RunnableAssign Enhances or modifies the input data by adding new fields or updating existing ones, preparing it for subsequent processing steps. TypeScript import { RunnableAssign, RunnableLambda } from "@langchain/core/runnables"; const getGeolocation = RunnableLambda.from(async (x: { ip: string }) => { // Simulate an API call to get geolocation return { location: `Location for IP ${x.ip}` }; }); const runnableAssign = new RunnableAssign({ getGeolocation }); const res = await runnableAssign.invoke({ name: "John Doe", ip: "192.168.1.1" }); // Output: { name: "John Doe", ip: "192.168.1.1", getGeolocation: { location: "Location for IP 192.168.1.1" } } RunnablePick Selects and extracts specific fields from the input data, allowing focused processing of relevant information. TypeScript import { RunnablePick } from "@langchain/core/runnables"; const orderData = { orderId: "12345", customerEmail: "customer@example.com", items: [{ productId: "A1", quantity: 2 }], totalAmount: 99.99, shippingAddress: "123 Main St", }; const receiptInfoRunnable = new RunnablePick(["orderId", "customerEmail", "totalAmount"]); const res = await receiptInfoRunnable.invoke(orderData); // Output: { orderId: '12345', customerEmail: 'customer@example.com', totalAmount: 99.99 } RunnablePassthrough Passes the input data through without any changes, which is useful for maintaining data integrity within a workflow. TypeScript const chain = RunnableSequence.from([ { question: new RunnablePassthrough(), context: async () => loadContextFromStore(), }, prompt, llm, outputParser, ]); const response = await chain.invoke( "I can pass a single string instead of an object since I'm using `RunnablePassthrough`." ); RunnableMap Applies transformations to each field in a map object, enabling individual processing of key-value pairs. TypeScript const sensorDataRunnable = RunnableMap.from({ temperature: RunnableLambda.from((data: { temp: number }) => `Temperature is ${data.temp}°C`), humidity: RunnableLambda.from((data: { humidity: number }) => `Humidity is ${data.humidity}%`), }); const result = await sensorDataRunnable.invoke({ temp: 22, humidity: 45 }); // Output: { temperature: 'Temperature is 22°C', humidity: 'Humidity is 45%' } Sequence and Workflow Composition Runnables that structure and execute tasks sequentially, enabling the creation of complex workflows: RunnableSequence Chains multiple Runnables in a linear fashion where the output of one becomes the input for the next, forming a step-by-step processing pipeline. TypeScript const imageProcessingChain = RunnableSequence.from([ readImageRunnable, resizeImageRunnable, applyFilterRunnable, saveImageRunnable, ]); const result = await imageProcessingChain.invoke('path/to/input/image.jpg'); RunnableEach Applies a Runnable to each element in a collection, similar to a map function over an array, allowing batch processing. TypeScript import { RunnableEach, RunnableLambda } from "@langchain/core/runnables"; const personalizeEmail = RunnableLambda.from((name: string) => `Dear ${name}, we have an offer for you!`); const sendEmail = emailSendingRunnable; // Assume this is defined elsewhere const emailChain = new RunnableEach({ bound: personalizeEmail.pipe(sendEmail), }); const result = await emailChain.invoke(["Alice", "Bob", "Carol"]); // Emails are sent to Alice, Bob, and Carol. RunnableParallel Executes multiple Runnables simultaneously on the same input, enabling concurrent processing for efficiency. TypeScript import { RunnableLambda, RunnableParallel } from "@langchain/core/runnables"; const calculateMean = RunnableLambda.from((data: number[]) => { return data.reduce((a, b) => a + b, 0) / data.length; }); const calculateMedian = RunnableLambda.from((data: number[]) => { const sorted = data.slice().sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2); return sorted.length % 2 !== 0 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; }); const calculateMode = RunnableLambda.from((data: number[]) => { const frequency: { [key: number]: number } = {}; let maxFreq = 0; let modes: number[] = []; data.forEach((item) => { frequency[item] = (frequency[item] || 0) + 1; if (frequency[item] > maxFreq) { maxFreq = frequency[item]; modes = [item]; } else if (frequency[item] === maxFreq) { modes.push(item); } }); return modes; }); const analysisChain = RunnableParallel.from({ mean: calculateMean, median: calculateMedian, mode: calculateMode, }); const res = await analysisChain.invoke([1, 2, 2, 3, 4]); // Output: { mean: 2.4, median: 2, mode: [2] } Error Handling, Resilience, and Configuration Runnables that enhance robustness with retry mechanisms and fallback options: RunnableBinding Creates a customized Runnable by pre-setting certain parameters or configurations, allowing for reusable components tailored to specific contexts. TypeScript import { RunnableConfig, RunnableLambda } from "@langchain/core/runnables"; const queryDatabase = (query: string, config?: RunnableConfig) => { const dbConfig = config?.configurable?.dbConfig; // Use dbConfig to establish a connection and execute the query return `Executed query on ${dbConfig.host}: ${query}`; }; const runnable = RunnableLambda.from(queryDatabase); // Bind configuration for different environments const prodRunnable = runnable.bind({ configurable: { dbConfig: { host: 'prod.db.example.com' } } }); const testRunnable = runnable.bind({ configurable: { dbConfig: { host: 'test.db.example.com' } } }); const result1 = await prodRunnable.invoke("SELECT * FROM users;"); // Output: "Executed query on prod.db.example.com: SELECT * FROM users;" const result2 = await testRunnable.invoke("SELECT * FROM users;"); // Output: "Executed query on test.db.example.com: SELECT * FROM users;" RunnableRetry Automatically retries a Runnable upon failure according to specified retry policies, enhancing resilience against transient errors. TypeScript import { RunnableLambda } from "@langchain/core/runnables"; const fetchWeatherData = async (location: string): Promise<string> => { // Simulate an API call that might fail if (Math.random() < 0.7) { throw new Error("Network error"); } return `Weather data for ${location}`; }; const fetchWeatherLambda = RunnableLambda.from(fetchWeatherData); // Apply retry logic const fetchWeatherWithRetry = fetchWeatherLambda.withRetry({ stopAfterAttempt: 5 }); try { const res = await fetchWeatherWithRetry.invoke("New York"); console.log(res); } catch (error) { console.error("Failed to fetch weather data after retries:", error.message); } RunnableWithFallbacks Provides alternative Runnables to execute if the primary one fails, ensuring the workflow can continue or degrade gracefully. TypeScript import { RunnableLambda, RunnableWithFallbacks } from "@langchain/core/runnables"; const primaryDataSource = async (id: string): Promise<string> => { // Simulate failure throw new Error("Primary data source is unavailable"); }; const secondaryDataSource = async (id: string): Promise<string> => { return `Data for ${id} from secondary source`; }; const primaryRunnable = RunnableLambda.from(primaryDataSource); const fallbackRunnable = RunnableLambda.from(secondaryDataSource); // Setup with fallback const dataRunnable = primaryRunnable.withFallbacks([fallbackRunnable]); const res = await dataRunnable.invoke("item123"); // Output: "Data for item123 from secondary source" Putting It All Together In the previous sections, we’ve explored individual Runnables and their roles in building modular workflows. Now, let’s see how we can combine these Runnables to create comprehensive, real-world applications. Below are three examples that demonstrate how to integrate multiple Runnables to solve complex problems. Example 1: Intelligent Document Processing Pipeline A company wants to automate the processing of incoming documents like invoices, receipts, and contracts. The goal is to classify the document type, extract relevant data, validate it, and store it in a database. The system should handle errors gracefully and retry operations if transient failures occur. Runnables Used: RunnableSequence, RouterRunnable, RunnableParallel, RunnableRetry, RunnableWithFallbacks, RunnableAssign, RunnableLambda TypeScript import { RunnableSequence, RouterRunnable, RunnableLambda, } from "@langchain/core/runnables"; // Define a unified output type type UnifiedOutput = { type: string; amount?: number; dueDate?: string; client?: string; parties?: string[]; term?: string; total?: number; items?: string[]; }; // Step 1: OCR Processing (simulate with a function) const ocrRunnable = RunnableLambda.from(async (imageBuffer: string) => { // Simulate OCR processing return "Extracted text: Invoice for Acme Corp"; }); // Step 2: Document Classification const classifyDocument = RunnableLambda.from(async (text: string) => { // Simulate document classification if (text.includes("Invoice")) return "invoice"; if (text.includes("Contract")) return "contract"; return "receipt"; }); // Step 3: Data Extraction Runnables for each document type const extractInvoiceData = RunnableLambda.from( async (text: string): Promise<UnifiedOutput> => { // Extract data specific to invoices return { type: "invoice", amount: 1000, dueDate: "2024-12-31", client: "Acme Corp", }; } ); const extractContractData = RunnableLambda.from( async (text: string): Promise<UnifiedOutput> => { // Extract data specific to contracts return { type: "contract", parties: ["Company A", "Company B"], term: "2 years", }; } ); const extractReceiptData = RunnableLambda.from( async (text: string): Promise<UnifiedOutput> => { // Extract data specific to receipts return { type: "receipt", total: 50, items: ["Item1", "Item2"] }; } ); const dataExtractionRouter = new RouterRunnable({ runnables: { invoice: extractInvoiceData, contract: extractContractData, receipt: extractReceiptData, }, }); // Step 5: Data Validation const validateData = RunnableLambda.from(async (data: UnifiedOutput) => { // Perform validation logic if (!data || !data.type) throw new Error("Validation failed: Data is missing or invalid"); return { ...data, isValid: true }; }); // Step 6: Save to Database (simulate with a function) const saveToDatabase = RunnableLambda.from(async (data: UnifiedOutput) => { // Simulate saving to a database return `Data saved: ${JSON.stringify(data)}`; }); // Step 7: Build the workflow sequence const documentProcessingWorkflow = RunnableSequence.from<string, any>([ ocrRunnable, classifyDocument, dataExtractionRouter, validateData, saveToDatabase.withRetry({ stopAfterAttempt: 3 }), ]); // Step 8: Add error handling with fallbacks const workflowWithFallback = documentProcessingWorkflow.withFallbacks({ fallbacks: [ RunnableLambda.from(async () => { return "An error occurred. Please try again later."; }), ], }); // Execute the workflow (async () => { try { const result = await workflowWithFallback.invoke("Document image bytes"); console.log(result); // Expected Output: "Data saved: { type: 'invoice', amount: 1000, dueDate: '2024-12-31', client: 'Acme Corp', isValid: true }" } catch (error: any) { console.error("Failed to process document:", (error as Error).message); } })(); The workflow starts by converting the document image into text using ocrRunnable. The extracted text is classified into a document type (invoice, contract, or receipt). RouterRunnable directs the text to the appropriate data extraction Runnable based on the document type. The extracted data is validated and then saved to the database. The RunnableRetry ensures that saving is retried up to three times in case of transient failures. If any step fails, RunnableWithFallbacks provides a fallback message to handle errors gracefully. Example 2: Personalized Recommendation Engine An e-commerce platform wants to provide personalized product recommendations to users based on their browsing history and preferences. Runnables Used: RunnableParallel, RunnableMap, RunnableBranch, RunnableWithFallbacks TypeScript import { RunnableParallel, RunnableMap, RunnableBranch, RunnableSequence, RunnableLambda, } from "@langchain/core/runnables"; // Step 1: Fetch user data from multiple sources in parallel const fetchUserData = RunnableParallel.from({ browsingHistory: RunnableLambda.from(async (userId) => { // Simulate fetching browsing history return ["Item1", "Item2"]; }), purchaseHistory: RunnableLambda.from(async (userId) => { // Simulate fetching purchase history return ["Item3"]; }), }); // Step 2: Map over the fetched data to process it const processUserData = RunnableMap.from({ browsingHistory: RunnableLambda.from((history: any[]) => { // Process browsing history return history.map((item) => `Processed ${item}`); }), purchaseHistory: RunnableLambda.from((history: any[]) => { // Process purchase history return history.map((item) => `Processed ${item}`); }), }); // Step 3: Define recommendation algorithms const newUserRecommendations = RunnableLambda.from(async (user) => { // Logic for new users return ["Product A", "Product B", "Product C"]; }); const returningUserRecommendations = RunnableLambda.from(async (user) => { // Logic for returning users based on history return ["Product X", "Product Y", "Product Z"]; }); // Step 4: Branch based on user type const recommendationBranch = RunnableBranch.from([ [(user: any) => user.isNew, newUserRecommendations], returningUserRecommendations, ]); // Step 5: Create a fallback recommendation system const defaultRecommendations = RunnableLambda.from(async (user) => { // Default recommendations return ["Default Product 1", "Default Product 2"]; }); const recommendationWithFallback = recommendationBranch.withFallbacks([ defaultRecommendations, ]); // Step 6: Sequence the entire workflow const recommendationWorkflow = RunnableSequence.from([ fetchUserData, processUserData, (data) => ({ ...data, isNew: data.purchaseHistory.length === 0 }), recommendationWithFallback, ]); // Usage const userId = "user123"; const recommendations = recommendationWorkflow.invoke(userId); // Output: Personalized recommendations based on user data The workflow begins by concurrently fetching the user’s browsing history, purchase history, and profile using RunnableParallel. Each piece of data is then processed individually using RunnableMap to prepare it for recommendation generation. The RunnableBranch decides which recommendation algorithm to use based on the user’s profile: If the user is a premium member (isPremiumMember is true), it uses premiumUserRecommendations.If the user has no purchase history (indicating a new user), it uses newUserRecommendations.Otherwise, it defaults to regularUserRecommendations. If any step in the recommendation process fails, RunnableWithFallbacks ensures that the system provides a set of default recommendations, maintaining a good user experience. Finally, RunnableSequence orchestrates the entire workflow, ensuring that each step happens in the correct order. The workflow is executed by invoking it with a userId, and it outputs personalized recommendations based on the user’s data. Example 3: Data Processing Pipeline for Analytics A company needs to process large datasets to generate analytics reports involving data cleaning, transformation, analysis, and visualization. Runnables Used: RunnableSequence, RunnableEach, RunnableRetry, RunnableBinding TypeScript import { RunnableSequence, RunnableEach, RunnableLambda, } from "@langchain/core/runnables"; // Step 1: Define data fetching with retry const fetchData = RunnableLambda.from(async (source) => { // Simulate data fetching, which may fail if (Math.random() < 0.2) { throw new Error("Data fetch error"); } return `Data from ${source}`; }).withRetry({ stopAfterAttempt: 3 }); // Step 2: Data cleaning const cleanData = RunnableLambda.from((data) => { // Perform data cleaning return `Cleaned ${data}`; }); // Step 3: Data transformation const transformData = RunnableLambda.from((data) => { // Transform data return `Transformed ${data}`; }); // Step 4: Data analysis const analyzeData = RunnableLambda.from((data) => { // Analyze data return `Analysis results of ${data}`; }); // Step 5: Data visualization const visualizeData = RunnableLambda.from((analysis) => { // Generate visualization return `Visualization of ${analysis}`; }); // Step 6: Sequence the steps const dataProcessingSequence = RunnableSequence.from([ cleanData, transformData, analyzeData, visualizeData, ]); // Step 7: Process multiple data sources const dataSources = ["Dataset A", "Dataset B", "Dataset C"]; const processAllData = new RunnableEach({ bound: fetchData.pipe(dataProcessingSequence), }); // Usage const reports = processAllData.invoke(dataSources); // Output: Array of visualization results for each data source This workflow handles data processing for multiple datasets from different sources. It begins by defining a fetchData runnable that is bound to specific data sources using RunnableBinding. Each data fetch operation is wrapped with RunnableRetry to handle transient failures by retrying up to three times. The data fetched from each source undergoes a series of processing steps defined by RunnableSequence: Data Cleaning: Removes or corrects erroneous data.Data Transformation: Converts data into a suitable format for analysis.Data Analysis: Performs analytical computations.Data Visualization: Generates visual representations of the analysis. RunnableEach is used to process multiple datasets in parallel. It applies the same processing sequence to each dataset. Conclusion The Runnable architecture in LangChain serves as a powerful foundation for building complex, modular workflows involving large language models (LLMs). Throughout this article, we’ve explored how Runnables can be created and combined to address various challenges: Routing and Branching: Utilizing RouterRunnable and RunnableBranch allows for dynamic execution paths based on runtime conditions.Data Manipulation and Assignment: Tools like RunnableAssign, RunnablePick, and RunnableMap offer flexible data transformation capabilities, preparing inputs for subsequent processing steps.Sequence and Workflow Composition: By chaining tasks using RunnableSequence, RunnableEach, and RunnableParallel, developers can orchestrate processes, whether they require sequential execution or parallel processing.Error Handling and Resilience: With RunnableRetry and RunnableWithFallbacks, workflows gracefully handle errors and provide fallback mechanisms. Runnable promotes a structured approach to building LLM agents and chains. As you integrate LangChain into your projects, consider how Runnables can enhance your workflows, making them more flexible, resilient, and easier to maintain.

In an era where data reigns supreme, the tools and platforms that businesses utilize to harness and analyze their data can make or break their competitive edge. Among these, Databricks stands out as a powerhouse, yet navigating its complexities often feels like deciphering cryptic code. With businesses generating an average of 2.5 quintillion bytes of data daily, the need for a robust, efficient, and cost-effective data cloud has never been more critical. In this post, we demystify Databricks, with a focus on Job Clusters. Readers will gain insight into the platform’s workspace, the pivotal role of workflows, and the nuanced world of compute resources including All-Purpose Compute (APC) clusters vs Jobs Compute clusters. We’ll also shed light on how to avoid costly errors and optimize resource allocation for data workloads. Introduction to Databricks: Components and Configurations Image source Databricks, at its core, is a comprehensive platform that integrates various facets of data science and engineering, offering a seamless experience for managing and analyzing vast datasets. Central to its ecosystem is the Workspace, a collaborative environment where data teams can store files, collaborate on notebooks, and orchestrate data workflows, to name a few capabilities. Workspaces act as the nerve center along with the Unity Catalog, which provides a bridge between workspaces. Together they facilitate an organized approach to data projects and ensure that valuable insights are never siloed. Workflows, available under each Workspace, represent recurring production workloads (aka jobs) which are vital for database production environments. These workflows automate routine data processing tasks, such as machine-learning pipelines, ETL workloads, and data streaming, ensuring reliability and efficiency in data operations. Understanding the significance of workflows is essential for anyone looking to streamline their data processes in Databricks. Databricks Job Clusters Workflows rely on Job Clusters for compute resources. Databricks offers various compute resource options to pick from during the cluster setup process. While the default cluster compute is set to Serverless, you can expose and tweak granular configuration options by picking one of the classic compute options. Serverless Jobs vs Classic Compute Multiple options to pick from during the cluster set up process Databricks recently announced the release of serverless compute across the platforms at the Data + AI 2024 conference. The goal is to provide users with a simplified cluster management experience and reduce compute costs. However, based on our research, Jobs Serverless isn’t globally cheaper than Classic Compute, and there is a price for the convenience that serverless offers. We compared the cost and performance of serverless jobs vs jobs running on classic clusters and found that while short ad-hoc jobs are ideal candidates for serverless, optimized classic clusters outperformed their serverless counterparts by 60% in costs. As for the convenience aspect, it does come at a cost. Serverless compute unburdens you from setting up cluster configuration, but that also means you lose control over job performance and costs. There are no settings to tweak or prices to negotiate with cloud providers. So if you are like us, and want to be able to control the configuration of your data infrastructure this might not be the right option for you. On-Demand Clusters vs. Spot Instances Compute resources form the backbone of Databricks cluster management. The basic classification of compute resources is between on-demand clusters vs Spot instances. Spot instances are considered cost-effective, offering discounts of up to 90% on remnant cloud compute. However, they aren’t the most stable or reliable. This is because the number of workers running can change in the middle of a job, which is dangerous. The job runtime and cost could become highly variable, and sometimes even crash. Overall, on-demand instances are better suited for workloads that cannot be interrupted, workloads with unknown compute requirements and immediate launch and operation needs. Spot instances, on the other hand, are ideal for workloads that can be interrupted and stateful applications with surge usage. Spot vs On-demandOn-demand InstancesSpot InstancesAccessImmediateOnly if there is unused compute capacityPerformanceStable and reliableLimited stability as workers can be pulled during a job runCostKnownVaries. Up to 90% cheaper than on-demand ratesBest forJobs that cannot be interrupted Jobs with unknown compute requirementsJobs that can be interrupted Stateful apps with surge usages Spot vs On-demand Clusters All-Purpose Compute vs Jobs Compute Databricks offers a couple of forms of compute resources, including All-Purpose Compute (APC) clusters, Jobs Compute clusters, and SQL warehouses. Each resource is designed with specific use cases in mind. SQL warehouses, for example, allow users to query information in Databricks using the SQL programming language. APCs and Jobs Compute, on the other hand, are more general compute resources capable of running many different languages such as Python, SQL, and Scala to run your jobs. While APCs are ideal for interactive analysis and collaborative research, Jobs Compute is ideal for scheduled jobs and workflows. Understanding the distinctions and use cases for these compute resources is crucial for optimizing performance and managing costs. Unfortunately, navigating the compute options for your clusters can be difficult. Common errors include the accidental deployment of more expensive compute resources, such as APC cluster when a Jobs Compute cluster would suffice. Such mistakes can have significant financial implications. In fact, we’ve found that APCs can cost up to 50% more than running the same job using Jobs Compute. The difference between APC clusters and Jobs Compute clusters represents a crucial decision point for Databricks users, as they spin up a cluster. Each option is tailored to different stages of the data analysis lifecycle. Knowing these nuances can help you avoid common errors to ensure that your Databricks environment is both effective and cost-efficient. For example, APC clusters can be used for more exploratory research work while Job Clusters are used for mature production scheduled workloads. Photon Workload Acceleration Databricks Photon is a high-performance vectorized query engine that accelerates workloads. Photon can substantially speed up job execution, particularly for SQL-based jobs and large-scale production Spark jobs. Unfortunately, Photon costs about 2x/DBU. Many factors influence whether Photon is the right choice for you, which is why we recommend you A/B test the same job with and without Photon enabled. From our experience, 9 out of 10 organizations opt out of Photon once the results are in. Spark Autoscaling Databricks offers an autoscaling feature to dynamically adjust the number of worker nodes in a cluster based on the workload demands. By dynamically adjusting resources, autoscaling can reduce costs for some jobs. However, due to the spin-up time cost of new workers, sometimes autoscaling can lead to increased costs. In fact, we found that simply applying autoscale across the board can lead to a 30% increase in compute costs. Notebooks for Ad-Hoc Research in Databricks Databricks revolutionizes the data science and data engineering landscape with its notebook concept, which combines code with annotations. It embodies a dynamic canvas where you can breathe life into your projects. Notebooks in Databricks excel in facilitating chunk-based code execution, a feature that significantly enhances debugging efforts and iterative development. By enabling users to execute code in segments, we eliminate the need for running entire scripts for minor adjustments. This accelerates the development cycle and promotes a more efficient debugging process. Notebooks can run on APC clusters and Jobs Compute clusters depending on the task at hand. This choice is paramount when project requirements dictate the need for either exploratory analysis when APC is ideal, or scheduled jobs when Jobs Compute will be the most cost-effective. It’s important to note that All-Purpose Compute clusters are shared environments with a collaborative aspect. Multiple data scientists can simultaneously work on the same project with collaboration facilitated by shared compute resources. As a result, team members can contribute to and iterate on analyses in real time. Such a shared environment streamlines research efforts, making APC clusters invaluable for projects requiring collective input and data exploration. The journey from ad-hoc research in notebooks to the production-ready status of workflows marks a significant transition. While notebooks serve as a sandbox for exploration and development, workflows embody the structured, repeatable processes essential for production environments. This contrast underscores the evolutionary path from exploratory analysis, where ideas and hypotheses are tested and refined, to the deployment of automated, scheduled jobs that operationalize the insights gained. It is this transition from the “playground” of notebooks to the “factory floor” of workflows that encapsulates the full spectrum of data processing within Databricks, bridging the gap between initial discovery and operational efficiency. From Development to Production: Workflows and Jobs The journey from ad-hoc research in notebooks to the deployment of production-grade workflows in Databricks encapsulates a sophisticated transition, pivotal for turning exploratory analyses into automated, recurring processes that drive business operations. At the core of this transition are workflows, which serve as the backbone for scheduling and automating jobs in Databricks. Workflows stand out by their ability to convert manual, repetitive tasks into automated sequences that run based on predefined triggers. These triggers vary widely, from the arrival of new files in a data lake to continuous data streams that require real-time processing. By leveraging workflows, data teams can ensure that their data pipelines are responsive, timely, and efficient, enabling seamless transitions from data ingestion to insight generation. Directed Acyclic Graphs (DAGs) are central to workflows and further enhance the platform by providing a graphical representation of workflows. DAGs in Databricks allow users to visualize the sequence and dependencies of jobs, offering a clear overview of how data moves through various processing stages. This is an essential feature for optimizing and troubleshooting workflows, enabling data teams to identify bottlenecks or inefficiencies within their pipelines. Transitioning data workloads from the exploratory realm of notebooks to the structured environment of production-grade workflows enables organizations to harness the full potential of their data, transforming raw information into actionable insights. Orchestration and Competition: Databricks Workflows vs. Airflow and Azure Data Factory By offering Workflows as a native orchestration tool, Databricks aims to simplify the data pipeline process, making it more accessible and manageable for its users. With that said, it also opens up a complex landscape of competition, especially when viewed against established orchestration tools like Apache Airflow and Azure Data Factory. Comparing Databricks Workflows with Apache Airflow and Azure Data Factory reveals a compelling competitive landscape. Apache Airflow, with its open-source lineage, offers extensive customization and flexibility, drawing users who prefer a hands-on, code-centric approach to orchestration. Azure Data Factory, on the other hand, integrates deeply with other Azure services, providing a seamless experience for users already entrenched in the Microsoft ecosystem. Databricks Workflows promise simplicity and integration, especially appealing to those who already leverage the platform. The key differentiation lies in Databricks’ ability to offer a cohesive experience, from data ingestion to analytics, within a single platform. The strategic implications for Databricks are multifaceted. On one hand, introducing Workflows as an all-in-one solution locks users into its ecosystem. On the other hand, it challenges Databricks to continually add value beyond what users can achieve with other tools. The delicate balance between locking users in and providing unmatched value is critical; users will tolerate a certain degree of lock-in if they perceive they are receiving significant value in return. As Databricks continues to expand its suite of tools with Workflows and beyond, the landscape of data engineering and analytics tools is set for a dynamic evolution. The balance between ecosystem lock-in and value provision will remain a critical factor in determining user adoption and loyalty. The competition between integrated solutions and specialized tools will shape the future of data orchestration, with opportunities for both consolidation and innovation. Conclusion In today’s data-driven world, mastering tools like Databricks is crucial. This guide aims to help simplify the management of Databricks Job Clusters. Our goal is to help you navigate the complexities and optimize resource allocation effectively. Understanding the distinction between compute resources, such as APC clusters for interactive analysis and Jobs Compute clusters for scheduled jobs, is essential. Understanding when to apply Photon and when not to is also critical for cluster optimization. Simply choosing the right compute options for your jobs can reduce your Databricks bill by 30% or more. The journey from exploration and research in notebooks to production-ready automated and scheduled workflows is a significant evolution. Workflows simplify data pipeline orchestration and compete with other data orchestration tools like Apache Airflow and Azure Data Factory. While Airflow offers extensive customization and Azure Data Factory integrates seamlessly with Microsoft services, Databricks Workflows provide a cohesive experience from data ingestion to analytics. The challenge for Databricks is to balance ecosystem lock-in with delivering unmatched value, ensuring it continues to enhance its tools and maintain user loyalty in an evolving data landscape.