DZone Spotlight

Today, I would like to share a management practice we developed at New Relic. It was one of the best things we did as an engineering organization. The practice is called “Mini-Ms." I believe it’s as important a practice for managers as code reviews are for engineers. This post is the first of a series: In this post, we look at how Mini-Ms work and what they are. Next, we learn how to implement Mini-Ms. Then, we discuss variations you may want to consider. Finally, we share a history of Mini-Ms, including where the name came from. What Is a Mini-M? A Mini-M is a group of managers that meet weekly or biweekly. The meeting is a combination support group and a working session. In each session, managers share the challenges they face. The other participants offer support and help problem-solve. Why Are Mini-Ms Effective? What is the value of meeting with other managers and talking about your problems? It may not seem like it is worth the time. Yet, many managers claimed Mini-Ms helped them grow more than anything else. Mini-Ms Help You Develop Skills There are a few reasons for that. Elaine May describes three reasons: “Learning to manage is very different from other types of learning (like engineering, for instance). In management, the concepts are simple and the execution is not. I’ve seen this trip up many engineering managers who read a book or attend a class and find the concepts trivial and therefore easy or not very important. Learning management is more of an apprenticeship versus an intellectual pursuit.” “In a Mini-M, you have many other people to learn from. If you are only learning from a coach, manager, or mentor, their approach may not fit very well for you, but in a Mini-M, you are getting multiple perspectives.” Teaching clarifies your thinking. When you share your experiences and ideas with other managers, you’re forced to articulate your own practice and consider it in a critical way. I have a few more explanations: Skills are unevenly distributed. You might be good at running meetings. I might be good at 1-1s. Another participant might be an effective project manager. During a Mini-M, we all share our experiences with each other. Imagine having to fill in those skills some other way. You would have to study each of those topics or learn each of them the hard way. That would be much slower! The diversity of the group allows the group to grow faster. Mini-Ms meet you where your skills are. Most alternative methods of learning involve assumptions of where your current skills are. For example, a management course tries to teach you a certain set of concepts or practices. But Mini-Ms allow you to expand your skills from wherever they may be. If I am struggling with how to run a particular meeting, I can immediately add a little to my skills during a single Mini-M session. This makes the M-team into a combination of a support group and training session for your management team. Mini-Ms Are a Source of Valuable Support Management is a difficult profession. As hierarchical creatures, being “higher” in the hierarchy separates you from your team members. And other managers are often busy and unavailable. Because Mini-Ms were largely modeled on support groups, your group can become part of your support network. Even outside of the meetings, you might turn to your fellow managers for help. Having a Mini-M helped me to build up a mental map of the skills of the managers around me. This was helpful because I started to understand the skills that I needed to build. When I ran into problems running projects, I felt like I could go to the person I knew was good at it and ask for help. And because we had a relationship from our Mini-M, they were more likely to want to help. In addition, many companies are distributed nowadays. This can result in less accidental interaction with peers, and it can contribute to isolation. Mini-Ms can be a source of connection with other managers. Mini-Ms Help Retain and Attract Managers Because managers feel better supported by their peers, they are more likely to stay at the company or stay in management. I anticipated that benefit. What surprised me was that people outside the company seemed to keep hearing about these Mini-M groups. And they would bring it up during interviews and mention it was why they applied in the first place! Why were they so attracted by Mini-Ms? They demonstrated that we took management seriously as a role. They showed that we valued developing our people, which was a signal that their job as a manager would be better here. They showed we experimented with our organizational practices and would welcome further innovation. Mini-Ms Help Build a Management Culture Along with a support group, the other inspiration for Mini-Ms was a conspiracy. The idea was to bring a few managers together, gripe about the problems in the organization, and come up with solutions. Mini-Ms naturally became the graph along which many management practices spread through the organization. In the Five Dysfunctions of a Team, Lencioni talks about the importance of managers seeing their “first team” as the management team. Mini-Ms helped develop a community of managers who all were focused on improving their skills and their teams. Many of us felt empowered to address the problems we saw in the organization around us. How Does a Mini-M Work? There have been a lot of variations to the way Mini-Ms have run over the years. What follows is a good way to start Mini-Ms. The Organizer of the Mini-Ms assigns each manager to a Mini-M. Each Mini-M has five to eight managers. The Organizer sets the expectation that managers attend. Each group has a Facilitator. They schedule and facilitate meetings. They usually start with a standard meeting format. Here’s an example format: Each person brings a topic: something that was challenging from last week, or something they are facing now. In an in-person meeting, the Facilitator would go around and ask everyone for their topics and put them on a whiteboard. In a distributed setting, all the participants would add their topics at the same time to a shared doc. Everyone gets three votes. They put them on the topics they would like to discuss. The group goes through the topics and discusses them in turn, ordered by the topics that have the most votes. The Facilitator checks in every five minutes to see if the group is finished with the current topic, or wants to continue the discussion. This is based on the Lean Coffee meeting format. The Facilitator sets the expectation that meetings are safe places. Participants discuss sensitive topics and there is an expectation of confidentiality. For example, a manager might talk about the challenges they were having with their manager. When To Use Mini-Ms You should design your Mini-Ms based on the particulars of your company. The size and company culture are particularly important. One precondition to this being successful is that the environment is a supportive one. If the leaders value learning and growth, they are more likely to support Mini-Ms properly. A competitive culture may not be a good fit. I recommend Mini-Ms to companies with product-market fit. Early in the growth of a company, product-market fit is the most important factor in a company’s success. After you’ve found a good fit with the market, organizational effectiveness moves up in importance to become one of the most critical factors for your success. At that point, M-Teams begin to contribute more directly to the success of the company. While you can lay the groundwork for M-Teams earlier, I wouldn’t recommend them in most early-stage companies. You can use Mini-Ms globally, across an organization, or within a smaller group. You can even implement a Mini-M informally, with a smaller group of managers. If you start informally, be sure to read the History section to see how they evolved at New Relic. How Do I Implement Mini-Ms? We get to that in our next post. Thank You Not a lot has been written about Mini-Ms, but there is one post currently on the New Relic blog, and another that has mysteriously vanished. Darin Swanson authored some content that was inspirational for large parts of this article. He and I have worked together on helping other companies implement Mini-Ms, so contact me if you’re interested in help. He also provided feedback and suggestions on drafts of this article. He encouraged me to explain the first team concept more fully, and to describe why pre-product market fit companies may not be a good match. And he suggested I split the content into separate articles. Nic Benders was one of the co-founders of Mini-Ms. He reviewed this post, offered feedback, contributed to the history section, and described some of the design goals for Mini-Ms. Elaine May provided feedback, some of which was so good I just ended up quoting her. She was gracious enough to offer some talk to talk about her experiences setting up or participating in similar programs at other companies, and she talked about her experiences with me in New Relic’s Mini-Ms. Elaine introduced me to the Chatham House Rules, which I incorporated into the post. Bjorn Freeman-Benson was the grandfather of the Mini-M. He shared a lot of his thinking about the principles behind why Mini-Ms were successful and what they were aiming for. He advised me to break up this post into sections and make it easier to get to the implementation section. And shared overall feedback. Merlyn Albery-Speyer helped me improve the section on “when to use Mini-Ms” by pointing out some preconditions for success. He pointed out that the structure became less important after the Mini-M is established. Merlyn pointed out that we tried to keep people from being in the same Mini-M as their managers, or other people reporting to the same manager. He also shared the theory about VPs not being willing to be vulnerable as a possible explanation for why the Mini-Ms never took off in manager-of-manager groups to the same extent they did for frontline managers. Jason Poole shared his experience as an Organizer of Mini-Ms. He pointed out a now mostly disappeared second post on Mini-Ms. He also suggested ways Facilitators could counter unproductive ranting. Marty Matheny shared feedback based on his experience as an Organizer. He helped improve the advice for Organizers. And he pointed out that engineering adjacent departments participated. Chris Hansen pointed out that confidentiality was an issue with whiteboards. He noticed an error that would have been embarrassing. He pointed out the value of M-teams in distributed organizations to counter isolation among managers. He also added some advice for participants. Chris also helped with a point about the value of the first few meetings. Teresa Martyny reviewed a draft of this post and pointed out some redundant assertions I was making. And she recommended I break this into multiple posts or edit for brevity. Natasha Litt was another early Mini-M leader, and she reviewed a draft of the post and contributed to it.

Retrieving the page source of a website under scrutiny is a day-to-day task for most test automation engineers. Analysis of the page source helps eliminate bugs identified during regular website testing, functional testing, or security testing drills. In an extensively complex application testing process, automation test scripts can be written in a way that if errors are detected in the program, then it automatically: Saves that particular page’s source code. Notifies the person responsible for the URL of the page. Extracts the HTML source of a specific element or code-block and delegates it to the responsible authorities if the error has occurred in one particular independent HTML WebElement or code block. This is an easy way to trace and fix logical and syntactical errors in the front-end code. In this article, we first understand the terminologies involved and explore how to get the page source in Selenium WebDriver using Python. What Is an HTML Page Source? In non-technical terminology, it’s a set of instructions for browsers to display info on the screen in an aesthetic fashion. Browsers interpret these instructions in their own ways to create browser screens for the client-side. These are usually written using HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and Javascript. This entire set of HTML instructions that make a web page is called page source, HTML source, or simply source code. Website source code is a collection of source code from individual web pages. Here’s an example of a source code for a basic page with a title, form, image, and submit button. <!DOCTYPE html> <html> <head> <title>Page Source Example - LambdaTest</title> </head> <body> <h2>Debug selenium testing results : LambdaTest</h2> <img loading="lazy" data-fr-src="https://cdn.lambdatest.com/assetsnew/images/debug-selenium-testing-results.jpg" alt="debug selenium testing" width="550" height="500"><br><br> <form action="/"> <label for="debug">Do you debug test results using LambdaTest?</label><br> <input type="text" id="debug" name="debug" value="Of-course!"><br> <br> <input type="submit" value="Submit"> </form> <br><br> <button type="button" onclick="alert('Page Source Example : LambdaTest!')">Click Me!</button> </body> </html> What Is an HTML Web Element? The easiest way to describe an HTML web element would be, “any HTML tag that constitutes the HTML page source code is a web element.” It could be an HTML code block, an independent HTML tag like </br>, a media object on the web page—image, audio, video, a JS function, or a JSON object wrapped within <script> </script> tags. In the above example, <title> is an HTML web element, and the children of body tags are HTML web elements too, i.e., <img>, <button>, etc. How To Get Page Source in Selenium WebDriver Using Python Selenium WebDriver is a robust automation testing tool and provides automation test engineers with a diverse set of ready-to-use APIs. To make Selenium WebDriver get page source, Selenium Python bindings provide us with a driver function called page_source to get the HTML source of the currently active URL in the browser. Alternatively, we can also use the GET function of Python’s request library to load the page source. Another way is to execute JavaScript using the driver function execute_script and make Selenium WebDriver get page source in Python. A unrecommended way of getting page source is using XPath in tandem with the “view-source:” URL. Let’s explore examples for these four ways of how to get page source in Selenium WebDriver using Python. We’ll be using a sample small web page hosted on GitHub for all four examples. This page was created to demonstrate drag and drop testing in Selenium Python using LambdaTest. Get HTML Page Source Using driver.page_source We’ll fetch pynishant.github.io in the ChromeDriver and save its content to a file named page_source.html. This file name could be anything of your choice. Next, we read the file’s content and print it on the terminal before closing the driver: from selenium import webdriver driver = webdriver.Chrome() driver.maximize_window() driver.get("https://pynishant.github.io/") pageSource = driver.page_source fileToWrite = open("page_source.html", "w") fileToWrite.write(pageSource) fileToWrite.close() fileToRead = open("page_source.html", "r") print(fileToRead.read()) fileToRead.close() driver.quit() On successful execution of the above script, your terminal output will show the following page source: Get HTML Page Source Using driver.execute_javascript In the previous example, we have to comment out (or replace) the driver.page_source line and add the following line: driver.execute_script is a Selenium Python WebDriver API to execute JS in a Selenium environment. Here, we execute a JS script that returns an HTML body element. # pageSource = driver.page_source pageSource = driver.execute_script("return document.body.innerHTML;") The output code looks like this: As you can observe, it only returns the innerHTML of the body element. Like the last output, we do not get the whole page source. To get the entire document, we execute document.documentElement.outerHTML. The execute_script line now looks like this: pageSource = driver.execute_script("return document.documentElement.outerHTML;") This gives us precisely the output we got using the driver.page_source. Fetch Page Source Using Python’s Request Library in Selenium WebDriver This method has nothing to do with Selenium but you can check the “What Is Selenium?” article, it’s a purely Pythonic way to get a webpage source. Here, we use Python’s request library to make a get request to the URL and save the request’s response, i.e., page source to an HTML file and print on the terminal. Here is the script: import requests url = 'https://pynishant.github.io/' pythonResponse = requests.get(url) fileToWrite = open("py_source.html", "w") fileToWrite.write(pythonResponse.text) fileToWrite.close() fileToRead = open("py_source.html", "r") print(fileToRead.read()) fileToRead.close() This method can be used to quickly store a webpage source code without loading the page in the Selenium-controlled browser. Similarly, we can use the urllib Python library to fetch the HTML page source. Get HTML Page Source Using the “view-source” URL This is rarely required, but you can append the target URL with view-source and load it in the browser window to load the source code and save it in manual testing: Programmatically, to take source code of screenshots in Python Selenium (if required), you can load the page using: driver.get("view-source:https://pynishant.github.io/") Get HTML Page Source in Selenium Python WebDriver Using XPath The fourth method to make Selenium WebDriver get a page source is to use XPath for saving it. Here, instead of page_source or executing JavaScript, we identify the source element, i.e., <html> and extract it. Comment out the previous page source fetching logic and replace it with the following: # pageSource = driver.page_source pageSource = driver.find_element_by_xpath("//*").get_attribute("outerHTML") In the above script, we are using a driver method, find_element_by_xpath, to locate the web page’s HTML element. We enter the document using source nod:"//*" and get its “outer HTML,” which is the document itself. The output looks the same as we got earlier using driver.page_source. How To Retrieve HTML Source of WebElement in Selenium To get the HTML source of a WebElement in Selenium WebDriver, we can use the get_attribute method of the Selenium Python WebDriver. First, we grab the HTML WebElement using driver element locator methods like (find_element_by_xpath or find_element_by_css_selector). Next, we apply the get_attribute() method on this grabbed element to get it’s HTML source. Suppose, from pynishant.github.io, and we want to grab and print the source code of the div with id “div1.” The code for this looks like this: from selenium import webdriver driver = webdriver.Chrome() driver.maximize_window() driver.get("https://pynishant.github.io/") elementSource = driver.find_element_by_id("div1").get_attribute("outerHTML") print(elementSource) driver.quit() Here’s the output: Similarly, to get the children or innerHTML of a WebElement: driver.find_element_by_id("some_id_or_selector").get_attribute("innerHTML") There is an alternative way of doing this and achieving same result: elementSource = driver.find_element_by_id("id_selector_as_per_requirement") driver.execute_script("return arguments[0].innerHTML;", elementSource) How To Retrieve JSON Data from an HTML Page Source in Python Selenium WebDriver Modern applications are built with multiple APIs at play. And often, these API dynamically change the content of HTML elements. JSON objects have emerged as an alternative to XML response types. So, it has become essential for a pro Selenium Python tester to handle JSON objects, especially those embedded in <script> HTML tags. Python provides us with an in-built JSON library to experiment with JSON objects. To demonstrate with an example, we load “https://www.cntraveller.in/” in Selenium driver and look-out for SEO schema contained in <script type=”application/ld+json”> </script> to verify that logo URL is included in the “JSON” schema. By the way, if you feel confused, this “SEO schema” is useful to get web pages ranked on google. It has nothing to do with code-logic or testing. We’re using it just for demonstration. We’ll be using LambdaTest for this demo: from selenium import webdriver import json import re username = "hustlewiz247" accessToken = "1BtTGpkzkYeOKJiUdivkWxvmHQppbahpev3DpcSfV460bXq0GC" gridUrl = "hub.lambdatest.com/wd/hub" desired_cap = { 'platform' : "win10", 'browserName' : "chrome", 'version' : "71.0", "resolution": "1024x768", "name": "LambdaTest json object test ", "build": "LambdaTest json object test", "network": True, "video": True, "visual": True, "console": True, } url = "https://"+username+":"+accessToken+"@"+gridUrl print("Initiating remote driver on platform: "+desired_cap["platform"]+" browser: "+desired_cap["browserName"]+" version: "+desired_cap["version"]) driver = webdriver.Remote( desired_capabilities=desired_cap, command_executor= url ) # driver = webdriver.Chrome() driver.maximize_window() driver.get("https://www.cntraveller.in/") jsonSource = driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('text') jsonSource = re.sub(";","",jsonSource) jsonSource = json.loads(jsonSource) if "logo" in jsonSource: print("\n logoURL : " + str(jsonSource["logo"])) else: print("JSON Schema has no logo url.") try: if "telephone" in jsonSource: print(jsonSource["telephone"]) else: print("No Telephone - here is the source code :\n") print(driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('outerHTML')) except Exception as e: print(e) driver.quit() The output contains logoURL and webElement source: Code Breakdown The following three lines import required libraries: Selenium WebDriver, Python’s JSON, and re library to handle JSON objects and use regular expressions: from selenium import webdriver import json import re Next, we configure our script for running it successfully on LambdaTest’s cloud. It took me less than thirty seconds to get started (maybe because I had prior experience with the platform). But even if you are a first-timer, it would take less than one minute. Register on LambdaTest’s official website, login using Google, and click on “Profile” to copy your username and access token: username = "your_username_on_lambdaTest" accessToken = "your lambdaTest access token" gridUrl = "hub.lambdatest.com/wd/hub" desired_cap = { 'platform' : "win10", 'browserName' : "chrome", 'version' : "71.0", "resolution": "1024x768", "name": "LambdaTest json object test ", "build": "LambdaTest json object test", "network": True, "video": True, "visual": True, "console": True, } url = "https://"+username+":"+accessToken+"@"+gridUrl We launch the driver in full-screen mode and load the cntraveller home page with the following line of code: driver = webdriver.Remote( desired_capabilities=desired_cap, command_executor= url ) # driver = webdriver.Chrome() driver.maximize_window() driver.get("https://www.cntraveller.in/") Now, we locate JSON objects containing script using the XPath locator and delete the unnecessary semicolons to load the string in JSON format properly: jsonSource = driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('text') jsonSource = re.sub(";","",jsonSource) jsonSource = json.loads(jsonSource) And then, we check if the logo URL is present. If present, we print it: if "logo" in jsonSource: print("\n logoURL : " + str(jsonSource["logo"])) else: print("JSON Schema has no logo url.") Also, we check if the telephone detail is present. If not, we print the source code of the WebElement: try: if "telephone" in jsonSource: print(jsonSource["telephone"]) else: print("No Telephone - here is the source code :\n") print(driver.find_element_by_xpath("//script[contains(text(),'logo') and contains(@type, 'json')]").get_attribute('outerHTML')) except Exception as e: print(e) Lastly, we quit the driver: driver.quit() How To Get Page Source as XML in Selenium WebDriver If you’re loading an XML-rendered website, you may want to save the XML response. Here’s a working solution for making Selenium get XML page source: drive.execute_script(‘return document.getElementById(“webkit-xml-viewer-source-xml”).innerHTML’) Conclusion You can use any of the above-demonstrated methods and leverage the agility and scalability of LambdaTest Selenium Grid cloud to automate your test processes. It lets you execute your test cases on 3000+ browsers, operating systems, and their versions. Also, you can integrate the automation testing flow with modern CI/CD tools and adhere to the best continuous testing practices. Happy Testing!

Mind maps are used for exploring, clarifying, explaining, or planning. They are an effective way to gather and depict information. This could be information that we want to learn or knowledge that we want to share. We may want to focus on certain points, such as abstracting a number or details. Alternatively, we may want to plan our work ahead or explain how things work in detail. Brainstorming and finding connections between different ideas, solving problems, and creating new ideas, mind maps are a useful tool in our professional or personal lives. They go well beyond software development and they can be used in any human endeavor that requires critical thinking and decision making. A mind map could be anything that organizes our thoughts — drawings, diagrams, tables, keywords, pictures, graphs, Wikipages, and so on. It can be done with a pen and a paper, a marker and a board, or using mind mapping tools like Coggle, MindMeister, Ayoa, MindNote, XMind, etc. A number of software groups that I’ve been working with have been reusing mind maps to release features. Product owners, developers, and testers have found an effective way to avoid missing use cases and identifying edge cases. This helped toward requirements clarification and disambiguation, testability and completeness. Reusing mind maps has led to fruitful discussions, educated decisions, and information exchange between teams. Development Method Used There exist different development methods, like waterfall, agile, lean, extreme programming, prototyping, rapid application development, and others. There are also different flavors for different methods, whilst hybrid combinations could also exist. We will not focus on which method, flavor, or combination is better, but to avoid confusion, we must clarify that this article assumes an agile-like whole-team approach. Development teams contain (at least) developers and testers, and the team’s work is divided into missions. Each mission contains features to be released. For each mission, a product owner is assigned that focuses on business decisions for customers. Mind Maps for Different Roles in a Software Group Product Owner The product owner helps customers define functional and non-functional requirements. After all, this role tries to make the most out of development teams in each mission by fulfilling all requirements. This implies that all team members are pursuing a common goal, establishing priorities so that they work on the highest-valued functionality. Decisions are also made that lead to a good return on investment per mission. What is the value delivered per feature? Which feature is more important for any given set of features? Which feature should be developed first, which second and so on. What did the customer request? A mind map from a product owner could be per feature, or it could involve multiple features or the entire mission. This depends on the feature size and on the scope of the mind map. Size To analyze and explain large features that contain a lot of functionality, it will require more space on a board. When many large features are put in the same mind map, things may get difficult to understand. When a mind map contains tons of information, it is easy to miss details, and people usually get discouraged using it. However, when the scope of a mind map is to depict the functionality delivered during an entire mission, if large features are involved, then careful decisions will need to be made on what to abstract. Scope A mind map depicting interdependencies at a business level between different features will probably include only the interdependent features. A mind map focusing on a feature and its sub-features will probably abstract all other features. A mind map focusing on the security aspects of the delivered features will probably depict only the features involved in the security use cases. This is the context that needs to be clarified as far as a product owner is concerned. Developer Developers need clarity and completeness about what should be implemented. They decide how to write the implementation code and their tests. Based on mind maps from the product owner, they can create their own mind maps. It’s up to the developer what information should be depicted (the scope of the mind map). Is it used to clarify code interdependencies? Is it used to explain how things work? Is it used to depict what has been unit tested and what not? Is it used to request clarifications from product owners? Is it used to depict specific areas in the code that require attention from testers? Entity-Relationship Mind Maps Entities and their relationships are a basic way of analyzing software systems. Entities could be specific to the domain in which we are working, such as invoices for accounting systems, items in inventory systems, nodes in network management systems, or friends in social networks. Exploring entities and their relationships in a mind map has helped development and product teams achieve alignment on what should be developed. It has also helped testers identify edge cases and brainstorm with developers about performance bottlenecks. State-Transition Mind Maps Identifying events that trigger transitions between states is another way of analyzing software systems. For example, a log-in event may have triggered the transition from a logged-out state into an authorization state and then a logged-in state. States can be directly controlled by the UI and easy to identify like logged-in/logged-out. They can also be subtle, triggered by the passage of time (e.g., timers in our software system have expired) and are uncontrollable by the UI, like authorization. Mind maps depicting states and their transitions can be helpful in implementing and debugging. They can also help to identify edge cases and reproduce sporadic bugs. Tester This role tries to make sure that bugs are found and fixed before releasing features to the customers. After the release, testers try to verify that the evolution of features follows certain quality criteria. Testers create and execute (UI or API) tests from requirements. Mind maps from product owners as well as from the developers could be very beneficial. A product owner will include functionality that needs to be released. This is excellent for test-case design in black-box testing. Some development teams created mind maps to depict what has been tested at a unit level. This indicated to the testers what has been tested and what not. Testers may augment development mind maps to make sense of what needs to be tested. Here lies grey-box testing information. There were cases when developers shared mind maps depicting areas in the UI that needed attention from testers. This was another major source of grey-box testing information. Based on this kind of feedback from developers, testers could focus their testing efforts. Ecosystem Mind Maps An ecosystem includes the environment in which our software lives, all the interfaces into our software, and all the external dependencies. A mind map of the entire ecosystem could give a clear picture for investigating connections, dependencies, and risks associated with the parts of our system that are outside our software’s control. One team used ecosystem mind maps that depicted how their software connected to the outside world. The focus was on the interfaces, users, and connections to integrated systems. Another team used a deployment-based ecosystem mind map. It was focused on where the components that make up their system, such as databases, configuration files, and executables, live in a production deployment. Ecosystem mind maps helped testers to devise their own positive and negative testing mind maps. They also helped product owners to get a bigger picture of their software and consider use cases that would have been missed on requirements. Developers found edge cases that needed special attention on their code. Wrapping Up Mind maps are reusable, tailorable, and can be mixed and matched according to our needs. They can be an effective tool for exchanging information and boosting confidence between teams to release features. After all, it’s all about aligning with the common goal of making customers happy.

The record feature has arrived in the latest LTS version, Java 17! Records allow the making of an immutable class without a boilerplate. That is awesome! The question is: how can we use it? In general, we saw a couple of samples with DTO, but we can do more than that. In this tutorial and video, we'll explore design capabilities with a record exceeding DTO. DTO We won't focus on it here, but it is worth mentioning that it is a good sample of record but not a unique case. It does not matter if you use Spring, MicroProfile, or Jakarta. Currently, we have several samples cases that I'll list below: MapStruct Jakarta JSON-B Spring Value Objects or Immutable Types In the DDD, value objects represent a concept from your problem domain. Those classes are immutable, such as Money, email, etc. So, once both value objects records are firm, you can use them. In our first sample, we'll create an email that needs validation only: Java public record Email (String value) { } As with any value object, you can add methods and behavior, but the result should be a different instance. Imagine we'll create a Money type, and we want to create the add operation. Thus, we'll add the method to check if those are the same currency and then result in a new instance: Java public record Money(Currency currency, BigDecimal value) { Money add(Money money) { Objects.requireNonNull(money, "Money is required"); if (currency.equals(money.currency)) { BigDecimal result = this.value.add(money.value); return new Money(currency, result); } throw new IllegalStateException("You cannot sum money with different currencies"); } } The Money was a sample, mainly because Java has a specification with JavaMoney and a famous library, Joda-Money, where you can use it. The point is when you need to create a Value object or an immutable type record that can fit perfectly on it. Immutable Entities But wait? Did you say immutable entities? Is that possible? It is not usual, but it happens, such as when the entity holds a historic transitional point. Can an entity be immutable? If you check Evan's definition of an entity in Chapter 5: An entity is anything that has continuity through a life cycle and distinctions independent of attributes essential to the application's user. The entity is not about be mutable or not, but it is related to the domain; thus, we can have immutable entities, but again, it is not usual. There is a discussion at Stackoverflow about this question. Let's create an entity, Book, where this entity has an ID, title, and release as a year. What happens if you want to edit a book? We don't: we need to create a new edition. Therefore, we'll also add the edition field. Java public record Book(String id, String title, Year release, int edition) {} Ok, but we also need validation. Otherwise, this book will have inconsistent data on it. It does not make sense to have null values on the id, title, and release as a negative edition. With a record, we can use the compact constructor and put validations on it: Java public Book { Objects.requireNonNull(id, "id is required"); Objects.requireNonNull(title, "title is required"); Objects.requireNonNull(release, "release is required"); if (edition < 1) { throw new IllegalArgumentException("Edition cannot be negative"); } } We can overwrite equals, hashCode, and toString methods if we wish. Indeed, let's overwrite the equalshashCode contracts to operate on the id field: Java @Override public boolean equals(Object o) { if (this == o) { return true; } if (o == null || getClass() != o.getClass()) { return false; } Book book = (Book) o; return Objects.equals(id, book.id); } @Override public int hashCode() { return Objects.hashCode(id); } To make it easier to create this class or when you have more complex objects, you can either create a factory method or define builders. The code below shows the builder creation on the book record method: Java Book book = Book.builder().id("id").title("Effective Java").release(Year.of(2001)).builder(); At the end of our immutable entity with a record, we'll also include the change method, where we need to change the book to a new edition. In the next step, we'll see the creation of the second edition of Effective Java. Thus, we cannot change the fact that there was a first edition of this book once; this historical part is part of our library business. Java Book first = Book.builder().id("id").title("Effective Java").release(Year.of(2001)).builder(); Book second = first.newEdition("id-2", Year.of(2009)); Currently, JPA cannot support immutable for compatibility reasons, but we can explore it on NoSQL APIs such as Eclipse JNoSQL and Spring Data MongoDB. We covered many of those topics; therefore, let's move on to another design pattern to represent the form of our code design. State Implementation There are circumstances where we need to implement a flow or a state inside the code. The state design pattern explores an e-commerce context where we have an order, and we need to keep the chronological flow of an order. Naturally, we want to know when it was requested, delivered, and finally received from the user. The first step is interface creation. To make it smooth, we'll use String to represent products, but you know we'll need an entire object for it: Java public interface Order { Order next(); List<String> products(); } With this interface ready for use, let's create an implementation that follows its flows and returns the products. We want to avoid any change to the products. Thus, we'll overwrite the products methods from the record to produce a read-only list. Java public record Ordered(List<String> products) implements Order { public Ordered { Objects.requireNonNull(products, "products is required"); } @Override public Order next() { return new Delivered(products); } @Override public List<String> products() { return Collections.unmodifiableList(products); } } public record Delivered(List<String> products) implements Order { public Delivered { Objects.requireNonNull(products, "products is required"); } @Override public Order next() { return new Received(products); } @Override public List<String> products() { return Collections.unmodifiableList(products); } } public record Received(List<String> products) implements Order { public Received { Objects.requireNonNull(products, "products is required"); } @Override public Order next() { throw new IllegalStateException("We finished our journey here"); } @Override public List<String> products() { return Collections.unmodifiableList(products); } } We have the state implemented; let's change the Order interface. First, we'll create a static method to start an order. Then, to ensure that we won't have a new intruder state, we'll block the new order state implementation and only allow the ones we have; therefore, we'll use the sealed interface feature. Java public sealed interface Order permits Ordered, Delivered, Received { static Order newOrder(List<String> products) { return new Ordered(products); } Order next(); List<String> products(); } We made it! We'll test the code with a list of products. As you can see, we have our flow exploring the capabilities of records. Java List<String> products = List.of("Banana"); Order order = Order.newOrder(products); Order delivered = order.next(); Order received = delivered.next(); Assertions.assertThrows(IllegalStateException.class, () -> received.next()); The state with an immutable class allows you to think about transactional moments, such as an entity, or generate an event on an event-driven architecture. Video Check out more video info to know more about the record:

Docker is a containerization technology that allows developers to package and deploy applications in lightweight, portable containers. These containers are isolated from the host operating system, which makes them portable across different environments and eliminates the “works on my machine” problem. Docker is a popular platform for creating and managing containerized applications; however, several alternatives for Docker can also be used for this purpose. Podman, Kubernetes, Openshift, LXD, Docker Swarm, BuidKit, and Mesos are some of the popular Docker alternatives available in the market today. In this article, we’ll discuss the following three Docker hub alternatives: Podman, Containerd, and LXD. So, let’s Begin! Podman Developed by RedHat, Podman is a daemonless, open-source, Linux-native container engine that is considered one of the best alternatives for Docker. Podman is used to build, run, and manage Linux OCI containers and container images. A container engine is an all-in-one software that is responsible for creating, running, and managing containers. A container engine provides an API or command-line interface for interacting with the containers, allowing developers to create, start, stop, and manage containers. Examples of container engines include Docker, Podman, and CRI-O. Podman uses the lib pod library, which provides a higher-level API for managing pods and containers. It also provides built-in support for rootless containers and improved security features. Advantages of Podman Easy to use: Podman has a simple and intuitive command-line interface that is similar to Docker’s command-line interface, making it easy for users who are already familiar with Docker to start using Podman. Compatible with Kubernetes: Podman can be used in confluence with Kubernetes, which means it can be used to run containers on a cluster adn locally. Support for multiple container formats: Podman supports both OCI and Docker container formats, which means it can run containers created using either format. Support for Cgroups v2: Podman supports Cgroups v2, which is a new version of the Linux kernel’s control group (cgroup) mechanism that provides more fine-grained control over resource allocation. Network support namespaces: Podman supports network namespaces, which allows you to use different network configurations for different containers. Differences Between Podman and Docker Docker and Podman are container engines, but there are some key differences between the two. Docker and Docker hub alternatives, such as Podman, are widely used and supported in the industry, and it depends on the specific use case and requirements of which one to use. Here are some of the key differences between Docker and Podman: Daemonless: Podman does not require a daemon to run containers, whereas Docker uses a daemon to manage containers. This means that Podman can run containers directly without the need for an additional service running in the background. Rootless: Podman can run containers without needing root access, whereas Docker requires root access to manage the container daemon. This makes Podman more secure, as it limits the potential attack surface. Image storage: Podman stores images in the local file system, whereas Docker uses a centralized image registry. This means that, with Podman, it is not necessary to have an internet connection to use local images. Networking: Docker uses its own networking stack, while Podman uses the host’s networking stack. CLI: Both have similar command line interfaces, so it’s easy to switch between them. Overall, Docker and Podman are powerful tools for containerization. For these two, and any other Docker alternatives, the ultimate choice between them often comes down to personal preference and specific use case requirements. Containerd Next on the list of Docker alternatives is Containerd. Containerd is a high-level, lightweight container runtime that provides a consistent and stable interface for running containers. Designed to be used as a daemon process that runs on a host system, it manages the lifecycle of containers by starting and stopping them, as well as providing other features, such as image management and storage. Containerd is also designed to work with other container orchestration tools, such as Kubernetes, to manage the scaling and scheduling of containers in a cluster. Advantages of Containerd Lightweight: Containerd is designed to be lightweight and fast, which means it has a small footprint and uses minimal resources. This makes it well-suited for use in high-performance and resource-constrained environments. Consistency: Containerd provides a consistent and stable interface for running containers, which makes it easier to manage and orchestrate them at scale. Flexibility: Containerd can be used with a variety of container orchestration tools, such as Kubernetes and Docker Swarm, which allows for greater flexibility in terms of how containers are managed and scaled. Plugins: Containerd has a modular design and support for plugins, which allows for easy customization and extension of its functionality. Security: Containerd provides a secure and isolated environment for running containers, and it has built-in support for image signing and validation. Support: Containerd is an open-source project with a large and active community, which means it has a wide range of support and resources available. Differences Between Containerd and Docker Containerd and Docker are container runtimes, but they have some key differences. Let’s take a look at these: Design: Containerd is designed to be a lightweight and minimal container runtime, while Docker is a more fully-featured container platform that includes additional components such as a built-in container registry and a management API. Functionality: Containerd focuses on providing a stable and consistent interface for running containers, while Docker provides a more comprehensive set of features such as image management and orchestration. Deployment: Containerd is intended to be used as a daemon process that runs on a host system, while Docker is typically deployed as a standalone service. Architecture: Containerd has a modular architecture that is designed to work with other container orchestration tools, while Docker has its own built-in orchestration features. Support: Containerd is an open-source project that is backed by a large and active community, while Docker is a commercial product that is supported by the company behind it. Plugins: Containerd has a pluggable architecture, which means it can be extended or customized using plugins, while Docker does not have a similar feature. Security: Containerd provides a secure and isolated environment for running containers and has built-in support for image signing and validation, while Docker does not have this feature by default. LXD Now, we’ll discuss one of the most commonly used alternatives of Docker in the list of Docker hub alternatives. LXD (Linux Containers Daemon) is a container hypervisor for Linux. It allows multiple isolated Linux systems (containers) to run on a single host, providing a lightweight alternative to virtual machines. LXD uses Linux kernel features, such as control groups and namespaces, to provide isolation, while also providing a simple and user-friendly command-line interface for managing containers. LXD is designed to work with existing Linux distributions and tools and supports a wide range of container images and formats, including Docker. It also provides advanced features like live migration, storage management, and network management. Developed and maintained by Canonicals, LXD is one of the well-known Docker hub alternatives and is the default container hypervisor for Ubuntu 20.04 and later versions. Advantages of LXD There are several advantages to using LXD as a container hypervisor. LXD is one of the most known Docker desktop alternatives available in the industry today. Take a look at the advantages of LXD below: Lightweight and fast: LXD uses Linux kernel features, such as control groups and namespaces, which are more lightweight and efficient than traditional virtualization methods. This results in faster startup times and lower resource overhead for containers. Easy to use: LXD provides a simple and user-friendly command-line interface for managing containers, making it easy to create, start, stop, and manage containers. Compatible with existing Linux distributions and tools: LXD is designed to work with existing Linux distributions and tools, and supports a wide range of container images and formats, including Docker. Advanced features: LXD provides advanced features, such as live migration, storage management, and network management, which allows you to move running containers between hosts without interruption, manage storage resources, and network interfaces within containers. Security: LXD uses AppArmor and Seccomp to provide additional security to the containers. Networking: LXD provides easy-to-use networking features to manage the container’s network interfaces, assign IP addresses, and create virtual networks. Scalability: LXD can run thousands of containers on a single host, making it highly scalable for large-scale deployments. High availability: LXD supports clustering features with HAproxy, allowing the creation of a highly available environment with automatic failover. Differences Between LXD and Docker LXD and Docker are containerization technologies, but they have some key differences. The decision of choosing Docker desktop alternatives should be made based on the use case and business requirements. Use case: LXD is a container hypervisor, while Docker is a container runtime. This means that LXD provides an additional layer of abstraction, allowing multiple isolated Linux systems (containers) to run on a single host, while Docker focuses on running individual containers. Containerization: LXD provides a more complete, system-level containerization experience, while Docker is more focused on application-level containerization. Design: LXD is designed to work with existing Linux distributions and tools and supports a wide range of container images and formats, including Docker. Docker, on the other hand, is focused on its own container format and ecosystem. Security Integration: LXD uses AppArmor and Seccomp to provide additional security to the containers, while Docker uses namespaces and control groups to isolate containers. Networking: LXD provides easy-to-use Networking features to manage the container's network interfaces and assign IP addresses, and create virtual networks, while Docker uses virtual networks based on IP addresses and network interfaces provided by the host. Overall, while Docker and LXD are powerful containerization technologies, they are designed to solve different problems and have different use cases. Depending on the use case, these alternatives of Docker can be used. How To Choose the Best Docker Alternative When choosing a Docker alternative, it is important to consider the following factors: Compatibility: Make sure the alternative is compatible with your existing infrastructure and technologies. Features: Evaluate the features offered by the alternative and see if they align with your needs. Support: Consider the level of support offered by the alternative and its community. Performance: Consider the performance of the alternative in terms of resource usage and scalability. Security: Evaluate the security features offered by the alternative and see if they meet your requirements. Cost: Consider the cost of using the alternative and compare it to other options. Conclusion So these were some of the popular alternatives for Docker. Each of these Docker alternatives has its own strengths and weaknesses, so it's important to analyze the pros and cons of each and study your business requirements before choosing any of these alternatives of Docker.

The amount of data that needs to be processed, filtered, connected, and stored is constantly growing. Companies that can process the data quickly have an advantage. Ideally, it should happen in real-time. Event-driven architectures are used for this. Such architectures allow exchanging messages in real-time, storing them in a single database, and using distributed systems for sending and processing messages. In this article, we will talk about how event-driven architecture differs from other popular message-driven architecture. In addition, we will learn the most popular event-driven architecture patterns, and their main advantages and disadvantages. Event-Driven vs. Message-Driven Architecture An event is some action that took place at a certain point in time. It is generated by the service and does not have specific recipients. Any system component can be an event consumer. A message is a fixed packet of data that is sent from one service to another. An event is a type of message that signals a change in the state of the system. In an event-driven architecture, the component that generates the event tells other components where it will be stored. That way, any component can save, process, and respond to an event, and the producer won’t know who the consumer is. In message-driven systems, components that create messages send them to a specific address. After sending a message, the component immediately receives control and does not wait for the message to be processed. In a message-driven architecture, if a similar message needs to be sent to multiple recipients, the sender must send it to each recipient separately. In contrast, in an event-driven architecture, a producer generates an event once and sends it to a processing system/ After that, the event can be consumed by any number of subscribers connecting to that system. Event-Driven Patterns There are different approaches to implementing an event-driven architecture. Often, when designing a program, several approaches are used together. In this section, we will talk about the most popular patterns that allow you to implement an event-driven architecture, their advantages, and their application areas. Global Event Streaming Platform It is vital for today’s companies to respond to events in real-time. Customers expect the business to respond immediately to various events. So, there is a need to develop such software architectures that will meet modern business requirements and will be able to process data as event streams, and not only data that is in a state of rest. A great solution is to use a global event streaming platform. It allows you to process business functions as event streams. In addition, this platform is fault-tolerant and scalable. All events that occur in the system are recorded in the data streaming platform once. External systems read these events and process them in real time. Event streaming platforms consist of a diverse set of components. Their creation requires significant resources and engineering experience. Such a pattern is quite popular and is used in many industries. Central Event Store The central event store ensures the publication and storage of events in a single database. It is a single endpoint for events of various types. This allows applications and services to respond to events in real-time without delays or data loss. Applications can easily subscribe to a variety of events, reducing development costs. The central event store is used for a variety of purposes: Publication of changes for consumer services. Reconstruction of past states and conducting business analytics. Search for events. Saving all program changes as a sequence of events. A single endpoint for notification based on application state changes. Monitoring system status. Using a central event store, you can create new applications and use existing events without having to republish them. Event-First and Event Streaming Applications Event streaming allows you to receive data from various sources, such as applications, databases, various sensors, Internet devices, etc., process, clean, and use them without first saving them. This method of event processing provides fast results and is very important for companies that transfer large amounts of data and require to receive information quickly. Key benefits of event streaming platforms: Improving the customer experience: Customers can instantly learn about changes in the status of their orders, which improves their experience and, accordingly, increases the company’s income. Risk reduction: Systems that use streaming events allow the detection of fraud on the Internet, can stop a suspicious transaction, or block a card. Reliability: Event streaming platforms allow for robust systems that can effectively handle subscriber failures. Feedback in real time: Users can see the results of their operations immediately after their execution, without waiting a minute. Event streaming systems require less infrastructure and data to support, so they are simple and fast to build. Applications that use this architecture pattern are used in various industries, such as financial trading, risk and fraud detection in financial systems, the Internet of Things, retail, etc. CQRS Command Query Responsibility Segregation (CQRS) is the principle of separating data structures for reading and writing information. It is used to increase the performance, security, and scalability of the software. The application of CQRS is quite useful in complex areas where a single model for reading and writing data is too complex. When it is separated, it is greatly simplified. This is especially noticeable when the number of reading and writing operations is significantly different. Different databases can be used for reading and storing data. In this case, they must be synchronized. To do this, the recording model must publish an event every time it updates the database. Advantages of CQRS: Independent scaling of reading and write workloads. Separate optimized data schemes for reading and writing. The use of more flexible and convenient models thanks to the separation of concerns. Ability to avoid complex queries by storing the materialized view in the read database. The CQRS template is useful for the following scenarios: In systems where many users simultaneously access the same data. When data read performance should be configured separately from data write performance. This is especially important when the number of reads greatly exceeds the number of writes. When the read-and-write models are created by different development teams. With frequent changes in business rules. In cases where the logic of the system and the user interface are quite simple, this template is not recommended. Event Sourcing An event sourcing stores the system state as a sequence of events. Whenever the system state changes, a new event is added to the event list. However, this system state can be restored by reprocessing events in the future. All events are stored in the event store, which is an event database. A good example of such an architecture is a version control system. Its event store is a log of all commits, and a working copy of the source tree is the system state. The main advantages of event search: Events can be published each time the system state changes. Ensuring a reliable change audit log. The ability to determine the state of the system at any time. The ability to easily transition from a monolithic application to a microservice architecture through the use of loosely coupled objects that exchange events. However, to reconstruct the state of business objects, you need to send standard requests, which is difficult and inefficient. Therefore, the system must use Command Request Responsibility Distribution (CQRS) to implement requests. This means that applications must process the final agreed-upon data. Automated Data Provisioning This pattern provides fully self-service data provisioning. The user must determine what data and in what format he needs, as well as where it should be stored. For example, in a database, distributed cache, microservice, etc. The selected repository can be used together with the central repository. The system provides the client with infrastructure along with pre-loaded data and manages the flow of events. The processor processes and filters data streams according to user requirements. The use of cloud infrastructure makes such a system faster and more practical. Systems that use automated data provisioning are used in finance, retail, and the Internet, both on-premises and in the cloud. Advantages and Disadvantages of Event-Driven Architecture Even though event-driven architecture is quite popular now, is developing rapidly, and helps to solve many business problems, there are also some disadvantages of using this approach. In this section, we list the main advantages and disadvantages of event-driven architecture. Advantages Autonomy: The loose coupling of components that use this architecture allows event producers and consumers to function independently of each other. This connection allows you to use different programming languages and technologies to develop different components. In addition, producers and consumers can be added and removed from the system without affecting other participants. Fault tolerance: Events are published immediately after they occur. Various services and programs subscribe to these events. If the consumer shuts down, events continue to be published and queued. When the consumer reconnects, it will be able to handle these events. Real-time user interaction for a better user experience. Economy: The consumer receives the message immediately after the producer has published it, eliminating the need for constant polling to verify the event. This reduces CPU consumption and network bandwidth usage. Disadvantages Error handling is difficult. Because event producers and consumers can be many and loosely connected, it is difficult to trace the actions between them and identify the root cause of a malfunction. Inability to predict events occurring in different periods. Since events are asynchronous, the order in which they occur cannot be predicted. Additionally, there may be duplicates, each of which may require a contextual response. This requires additional testing time and deep system analysis to prevent data loss. The weak connection between the systems that generate the events and the systems that receive them can be a point of vulnerability that attackers can exploit. Conclusion Event-driven architectures have been around for a long time and were used to pass messages. In connection with modern business needs, they are actively developing and offering better opportunities for connection and management. Event-driven architectures are indispensable for real-time event processing, as required by many modern systems. To implement such an architecture, there are several different patterns, each of which has its advantages and application features. When designing a system, you need to clearly define its functions and applications to choose the right pattern. Although event-driven architecture is widely used and has many advantages, its features can sometimes have a bad effect on the system, which must be taken into account during its design.

I'm stating the obvious here. ChatGPT, released just eight weeks ago, has taken the whole world by storm. Microsoft is rumored to have invested $10B in it, and Sathya Nadella expects this to transform every Microsoft Product. Eventually, this should even come to SQL Server, the product I used to ship to Microsoft in the 90s. SQL itself is entering its 50th year and has continuously evolved to reign over all languages. Can SQL do ChatGPT? Let’s see. Examples here are from Couchbase. Couchbase has SQL for JSON, called N1QL, aka SQL++. Also, this article is different from the other 71 articles I've written here. You'll see it if you read long enough! "Every product of Microsoft will have the same AI capabilities to completely transform the product." Sathya Nadella Summary Couchbase N1QL (Non-First Normal Form Query Language), aka SQL++, is a SQL-like language for querying JSON data stored in Couchbase. The CURL() function in Couchbase N1QL allows you to make HTTP requests to external services. It can be used to invoke ChatGPT by sending a request to its API endpoint. A JavaScript UDF (User-Defined Function) in Couchbase is a function written in JavaScript that can be executed as part of an N1QL query. It can also be used to invoke ChatGPT by making an HTTP request to its API endpoint. To connect ChatGPT with Couchbase N1QL, you need to know the API endpoint of ChatGPT and the authentication credentials required to access it. The syntax for invoking ChatGPT from Couchbase N1QL using the CURL() function may look like this: SQL SELECT CURL("https://api.openai.com/v1/engines/davinci/completions", { "request": "POST", "headers": [ "Authorization: Bearer ChangeToYourKeyElseWontWork", "Content-Type: application/json" ], "data": '{ "prompt": "Write a song on SQL using the style of Taylor Swift songs.", "max_tokens": 200 }' }) AS result ; Here's the response: JSON [ { "result": { "choices": [ { "finish_reason": "length", "index": 0, "logprobs": null, "text": " Where Swift sneakily commands, “…here you go / I wrote it on my phone so you could Google it,” you should write: “…here’s my query / I’ll drink my coffee and then will spool it.” Or, “…Here’s what I’m thinking / I pilfered my colleague’s project.” Only you can aspire to provide the official Taylor Swift SQL spoof song (don’t forget to play some chord tinkling).\n\n4. Invent an official buzzword for lambda expressions. Start a campaign to make “lambda-eme” or variants thereof the standard. Try to get lambda-me in all the common tools available (lamellar, lambmaster, lambator, lambda café, lambananas, and so on).\n\n5. Write about a process that took you too long in just 5 minutes. Make fun of" } ], "created": 1675103061, "id": "cmpl-6eT7FnvGFN9HViONQnDhv5uabM6AO", "model": "davinci", "object": "text_completion", "usage": { "completion_tokens": 200, "prompt_tokens": 13, "total_tokens": 213 } } } ] To pass parameters to ChatGPT, you can modify the data field in the above example. For example: JSON "data": '{ "prompt": "Write a song on SQL using the style of Taylor Swift songs.", "temperature": 0.5, "max_tokens": 200 }' Here Are the Questions the Rest of the Article Will Try to Answer: What is Couchbase N1QL, and how does it work? What is the CURL() function, and how is it used in Couchbase N1QL? What is a JavaScript UDF in Couchbase, and how is it different from CURL()? How to connect ChatGPT with Couchbase N1QL using the CURL() function and/or JavaScript UDF? What is the syntax for invoking ChatGPT from Couchbase N1QL using CURL() function and/or JavaScript UDF? How to pass parameters to ChatGPT from Couchbase N1QL using CURL() function and/or JavaScript UDF? What are the best practices for error handling and debugging when invoking ChatGPT from Couchbase N1QL using CURL() function and/or JavaScript UDF? What are the limitations and considerations when using CURL() function and/or JavaScript UDF to invoke ChatGPT from Couchbase N1QL? Couchbase N1QL and CURL() Couchbase N1QL (Non-First Normal Form Query Language) is a SQL-like language for querying JSON data stored in Couchbase. It provides various functions for transforming and manipulating data, and one of these functions is the CURL() function. The CURL() function in Couchbase N1QL allows you to make HTTP requests to external services from within an N1QL query. This can be useful for integrating Couchbase with other systems, such as web APIs, or for fetching data from the web for further processing in Couchbase. The syntax for using the CURL() function in Couchbase N1QL is as follows: SELECT CURL(<url>, <options>); SELECT RAW list FROM CURL("https://api.github.com/users/sitaramv/repos") AS list LIMIT 1; In this example, the CURL() function is sending a GET request to the URL, which returns the repos for the user sitaramv. The result of the function is stored in the response variable list, which can be further processed using other N1QL functions. The <options> JSON object can contain several properties, such as a method for specifying the HTTP method (GET, POST, PUT, etc.), headers for setting custom HTTP headers, and data for sending data in the body of the request. Here is an example of how the CURL() function can be used to post data to an external JSON select imagetext from curl("https://vision.googleapis.com/v1/images:annotate?key=PUT YOUR KEY HERE", {"request": "POST", "header":"Content-Type: application/json", "data": '{ "requests": [ { "image": { "source": { "imageUri": "http://www.couchbase.com/blog/wp-content/uploads/2018/01/Screen-Shot-2018-01-21-at-6.50.38-PM.png" } }, "features": [ { "type": "TEXT_DETECTION" } ] } ] }'}) AS imagetext In this example, the CURL() function is sending a POST request to the URL with a JSON payload in the body of the request. The custom header Content-Type: application/jsonis set to indicate that the data in the request body is in JSON format. The CURL() function in Couchbase N1QL provides a convenient way to connect Couchbase to external systems and fetch data from the web. With the ability to specify the HTTP method, headers, and request body, it offers a lot of flexibility and can be used in a variety of scenarios. Couchbase N1QL (Non-First Normal Form Query Language) provides several functions for transforming and manipulating data, and one of these functions is the JavaScript User-Defined Function (UDF). A JavaScript UDF is a custom function written in JavaScript that can be used within an N1QL query to perform more complex operations. The JavaScript UDF in Couchbase N1QL allows you to write custom logic in JavaScript, which can then be executed as part of an N1QL query. This provides a lot of flexibility, as you can write JavaScript code to perform complex operations that cannot be achieved using the built-in N1QL functions. Here is an example of how a JavaScript UDF can be used in Couchbase N1QL: SQL curl -v -X POST \ http://localhost:8093/evaluator/v1/libraries/mysamples \ -u Administrator:password \ -H 'content-type: application/json' \ -d 'function square(val) { \ return val * val; \ } CREATE FUNCTION square(val) LANGUAGE JAVASCRIPT AS "square" AT "mysamples"; SELECT square(5) as result; In this example, a JavaScript UDF named square is created that takes a single argument val and returns its square. The square function is defined using the CREATE FUNCTION statement, and the JavaScript code for the function is enclosed in a pair of $$. The LANGUAGE javascript option specifies that the code for the function is written in JavaScript. The JavaScript UDF can be used within an N1QL query by calling the function and passing the necessary arguments. In this example, the square function is called with argument value 5, and the result is stored in the result variable. The JavaScript UDF can itself issue N1QL statements to operate on the data. SQL curl -v -X POST http://localhost:8093/evaluator/v1/libraries/p1 -u Administrator:password -H 'content-type: application/json' -d 'function ptc1(a, b) { var qi0 = START TRANSACTION; var acc = []; for (const row of qi0) { acc.push(row); } var qi1 = INSERT INTO b VALUES(UUID(), {"radius": $a, "area": $b}) RETURNING meta().id,* ; for (const row of qi1) { acc.push(row); } var a2 = a * a; var b2 = b * b; var qi2 = INSERT INTO b VALUES(UUID(), {"radius": $a2, "area": $b2}) RETURNING meta().id,* ; for (const row of qi2) { acc.push(row); } var qi9 = COMMIT ; for (const row of qi9) { acc.push(row); } return acc; }' create or replace function ptc1(x, y) language javascript as "ptc1" at "p1" ; execute function ptc1(4, 16); The JavaScript UDF in Couchbase N1QL is different from the CURL() function in several ways. The CURL() function is used to make HTTP requests to external services and retrieve data from the web, whereas the JavaScript UDF allows you to write custom logic in JavaScript and execute it as part of an N1QL query. JavaScript itself can invoke CURL() function. In the example below, we write a sample JavaScript() function called ChatGPT(), which invokes the ChatGPT API. Additionally, the CURL() function returns the data from the external service, whereas the JavaScript UDF can return any value that can be expressed in JavaScript, such as numbers, strings, objects, arrays, etc. In conclusion, the JavaScript UDF in Couchbase N1QL provides a powerful way to extend the capabilities of N1QL by writing custom logic in JavaScript. It offers a lot of flexibility and can be used to perform complex operations that cannot be achieved using the built-in N1QL functions. Invoking OpenAI's ChatGPT from Couchbase N1QL can be achieved using either the CURL() function or a JavaScript User-Defined Function (UDF). Both methods have their own syntax and are briefly discussed below. Using the CURL() function: The CURL() function can be used to make HTTP requests to external services, such as OpenAI's API for ChatGPT. The syntax for invoking ChatGPT from Couchbase N1QL using the CURL() function is as follows: JSON SELECT CURL("https://api.openai.com/v1/engines/davinci/completions", { "request": "POST", "headers": [ "Authorization: Bearer ChangeThisToYourKey", "Content-Type: application/json" ], "data": '{ "prompt": "Hello. How are you doing today?", "temperature": 0.9, "max_tokens": 200 }' }) AS result ; In this example, the CURL() function is used to make a POST request to the OpenAI API endpoint for ChatGPT. The headers option is used to set the content type as application/json, and the data option is used to specify the request payload, which includes the prompt for ChatGPT, the temperature, and the maximum number of tokens to generate. The response from the API is stored in the response variable. Using a JavaScript UDF: Another way to invoke ChatGPT from Couchbase N1QL is by using a JavaScript UDF. The syntax for invoking ChatGPT from Couchbase N1QL using a JavaScript UDF is as follows: JavaScript curl -v -X POST http://localhost:8093/evaluator/v1/libraries/cglib -u Administrator:password -d 'function chatGPT3(prompt) { var chaturl = "https://api.openai.com/v1/engines/davinci/completions" dstr = "{\"prompt\": \"" + prompt + "\",\"temperature\": 0.5 , \"max_tokens\": 200}" var options = { "request" : "POST", headers: [ "Authorization: Bearer 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", "Content-Type: application/json" ], "data": dstr }; var query = SELECT CURL($chaturl, $options); var acc = []; for (const row of query) acc.push(row); return acc; }' In the cbq shell, do the following: DROP FUNCTION chatGPT3; CREATE FUNCTION chatGPT3(prompt) language javascript as "chatGPT3" at "cglib"; select chatGPT3("Write an essay on Lincoln."); Here's the result of that query: JSON cbq> select chatGPT3("Write an essay on Lincoln."); { "requestID": "6acb9a20-93f0-41c2-bdc4-fe28107d85a9", "signature": { "$1": "json" }, "results": [ { "$1": [ { "$1": { "choices": [ { "finish_reason": "length", "index": 0, "logprobs": null, "text": "\n\n“Lincoln,” wrote Carl Sandburg, “was a self-made man in the best sense of the word.”\n\nWhat does Sandburg mean by this statement?\n\nWhat advantages did Lincoln have in his youth?\n\nWhat disadvantages did he have?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\nWhat events from his youth helped to make him a self-made man?\n\n" } ], "created": 1675310533, "id": "cmpl-6fL5Zu1MGjNorxBOwyg3H1ODzzjSD", "model": "davinci", "object": "text_completion", "usage": { "completion_tokens": 200, "prompt_tokens": 6, "total_tokens": 206 } } } ] } ], "status": "success", "metrics": { "elapsedTime": "7.179456323s", "executionTime": "7.179347355s", "resultCount": 1, "resultSize": 1537, "serviceLoad": 1 } } cbq> The actual essay chatGPT wrote was (don't ask me to explain this...!) "text": "\n\n“Lincoln,” wrote Carl Sandburg, “was a self-made man in the best sense of the word.” What does Sandburg mean by this statement? What advantages did Lincoln have in his youth? What disadvantages did he have? What events from his youth helped to make him a self-made man? What events from his youth helped to make him a self-made man? What events from his youth helped to make him a self-made man? What events from his youth helped to make him a self-made man? In this example, a JavaScript UDF named chatGPTe is created that takes prompt as the argument. The JavaScript code for the UDF uses the request library to make a POST request to the OpenAI API endpoint for ChatGPT. The options for the request include the URL, the method, the headers, and the request payload. Once the API call is made, the response from the API is stored in the response variable. The UDF then returns the response, which can be used in an N1QL query by calling the chatGPT function and passing the necessary arguments. In conclusion, you can use either the CURL() function directly or via JavaScript function. What are the limitations and considerations when using CURL() function and/or JavaScript UDF to invoke ChatGPT from Couchbase N1QL? Invoking ChatGPT from Couchbase N1QL using CURL() function and JavaScript UDF can offer a lot of benefits for your application. However, it is important to be aware of the limitations and considerations in order to ensure a smooth and successful integration. Here are some of the most important things to keep in mind when using these methods: CURL() Function Limitations: The CURL() function in N1QL is limited to making GET and POST requests only. If you need to make other types of requests, such as PUT or DELETE, you will need to use a JavaScript UDF. The CURL() function is synchronous and will block the N1QL query until the response is received. This can impact performance, especially if the response is large or slow. Data Size Limitations: Both the CURL() function and JavaScript UDF have limitations on the amount of data that can be passed or received. You may need to chunk your data into smaller pieces and make multiple requests if the data is very large. The response from ChatGPT may also be large, so it is important to consider this when planning your integration. You may need to process the response in smaller chunks and store the results in Couchbase rather than trying to load the entire response into memory. Security Considerations: When using the CURL() function or JavaScript UDF to make requests to ChatGPT, you will be passing sensitive information, such as API keys and other credentials, over the network. It is important to consider the security of this information and take steps to encrypt or protect it as necessary. The response from ChatGPT may also contain sensitive information, so it is important to secure the data stored in Couchbase and protect it from unauthorized access. By keeping these limitations and considerations in mind, you can ensure a smooth and successful integration of ChatGPT with Couchbase N1QL using CURL() function and JavaScript UDF. If you encounter any issues or challenges, it is always a good idea to seek assistance from a knowledgeable professional or the Couchbase community for guidance. Epilogue Unlike my other 71 articles on DZone, I just wrote the prologue and the epilogue of this article. The rest of the article was written by ChatGPT. ChatGPT's ability to write human-consumable content is pretty impressive. I did the edits, wrote correct examples, and tested them. ChatGPT's code generation still needs improvement. It's a whole new world.

When we combine the cloud with IaC tools like Terraform and continuous deployment we get the almost magical ability to create resources on demand. For all its benefits, however, the cloud has also introduced a set of difficulties, one of which is estimating cloud costs accurately. Cloud providers have complex cost structures that are constantly changing. AWS, for example, offers 536 types of EC2 Linux machines. Many of them have similar names and features. Take for example "m6g.2xlarge" and "m6gd.2xlarge" — the only difference is that the second comes with an SSD drive, which will add $60 dollars to the bill. Often, making a mistake in defining your infrastructure can cause your bill to balloon at the end of the month. It’s so easy to go above budget. We can set up billing alerts, but there are no guarantees that they will work. Alerts can happen during the weekend or be delayed, making us shoot past our budget in a few hours. So, how can we avoid this problem and use the cloud with confidence? Enter Infracost Infracost is an open-source project that helps us understand how and where we’re spending our money. It gives a detailed breakdown of actual infrastructure costs and calculates how changes impact them. Basically, Infracost is a git diff for billing. Infracost has two versions: a VSCode addon and a command line program. Both do the same thing: parse Terraform code, pull the current cost price points from a cloud pricing API, and output an estimate. You can use Infracost pricing API for free or host your own. The paid tier includes a cloud dashboard to track changes over time. We can see the estimates right in the IDE: Real-time cost estimation on VSCode. Or as comments in pull requests or commits: Cost change information in the PR. Setting Up Infracost To try out Infracost, we’ll need the following: An Infracost API key: You can get one by signing up for free at Infracost.io. The Infracost CLI installed in your machine Some Terraform files Once the CLI tool is installed, run infracost auth login to retrieve the API key. Now we’re ready to go. The first command we’ll try is infracost breakdown. It analyzes Terraform plans and prints out a cost estimate. The --path variable must point to the folder containing your Terraform files. For example, imagine we want to provision an "a1.medium" EC2 instance with the following: provider "aws" { region = "us-east-1" skip_credentials_validation = true skip_requesting_account_id = true } resource "aws_instance" "myserver" { ami = "ami-674cbc1e" instance_type = "a1.medium" root_block_device { volume_size = 100 } } At current rates, this instance costs $28.62 per month to run: $ infracost breakdown --path . Name Monthly Qty Unit Monthly Cost aws_instance.myserver ├─ Instance usage (Linux/UNIX, on-demand, a1.medium) 730 hours $18.62 └─ root_block_device └─ Storage (general purpose SSD, gp2) 100 GB $10.00 OVERALL TOTAL $28.62 If we add some extra storage (600GB of EBS), the cost increases to $155.52, as shown below: $ infracost breakdown --path . Name Monthly Qty Unit Monthly Cost aws_instance.myserver ├─ Instance usage (Linux/UNIX, on-demand, a1.medium) 730 hours $18.62 ├─ root_block_device │ └─ Storage (general purpose SSD, gp2) 100 GB $10.00 └─ ebs_block_device[0] ├─ Storage (provisioned IOPS SSD, io1) 600 GB $75.00 └─ Provisioned IOPS 800 IOPS $52.00 OVERALL TOTAL $155.62 Infracost can also calculate usage-based resources like AWS Lambda. Let's see what happens when we swap the EC2 instance for serverless functions: provider "aws" { region = "us-east-1" skip_credentials_validation = true skip_requesting_account_id = true } resource "aws_lambda_function" "my_lambda" { function_name = "my_lambda" role = "arn:aws:lambda:us-east-1:account-id:resource-id" handler = "exports.test" runtime = "nodejs12.x" memory_size = 1024 } Running infracost breakdown yields a total cost of 0 dollars: $ infracost breakdown --path . Name Monthly Qty Unit Monthly Cost aws_lambda_function.my_lambda ├─ Requests Monthly cost depends on usage: $0.20 per 1M requests └─ Duration Monthly cost depends on usage: $0.0000166667 per GB-seconds OVERALL TOTAL $0.00 That can’t be right unless no one uses our Lambda function, which is precisely what the tool assumes by default. We can fix this by providing an estimate via a usage file. We can create a sample usage file with this command: $ infracost breakdown --sync-usage-file --usage-file usage.yml --path . We can now provide estimates by editing usage.yml. The following example consists of 5 million requests with an average runtime of 300 ms: resource_usage: aws_lambda_function.my_lambda: monthly_requests: 5000000 request_duration_ms: 300 We’ll tell Infracost to use the usage file with --usage-file to get a proper cost estimate: $ infracost breakdown --path . --usage-file usage.yml Name Monthly Qty Unit Monthly Cost aws_lambda_function.my_lambda ├─ Requests 5 1M requests $1.00 └─ Duration 1,500,000 GB-seconds $25.00 OVERALL TOTAL $26.00 That’s much better. Of course, this is accurate as long as our usage file is correct. If you’re unsure, you can integrate Infracost with the cloud provider and pull the utilization metrics from the source. Git Diff for Cost Changes Infracost can save results in JSON by providing the --format json and --out-file options. This gives us a file we can check in source control and use as a baseline. $ infracost breakdown --path . --format json --usage-file usage.yml --out-file baseline.json We can now compare changes by running infracost diff. Let’s see what happens if the Lambda execution time goes from 300 to 350 ms: $ infracost diff --path . --compare-to baseline.json --usage-file usage.yml ~ aws_lambda_function.my_lambda +$4.17 ($26.00 → $30.17) ~ Duration +$4.17 ($25.00 → $29.17) Monthly cost change for TomFern/infracost-demo/dev Amount: +$4.17 ($26.00 → $30.17) Percent: +16% As you can see, the impact is a 16% increase. Integrating Infracost With CI/CD We’ve seen how this tool can help us estimate cloud costs. That’s valuable information, but what role does Infracost take in continuous integration? To answer that, we must understand what infracost comment does. The comment command takes a JSON file generated by infracost diff and posts its contents directly into GitHub, Bitbucket, or GitLab. Thus, by running Infracost inside CI, we make relevant cost information available to everyone on the team. Infracost comments on the cost difference in a GitHub commit. If you want to learn how to configure CI/CD to run Infracost on every update, check out this tutorial: How to Run Infracost on Semaphore. Working With Monorepos You will likely have separate Terraform files for each subproject if you work with a monorepo. In this case, you should add an infracost config file at the project's root. This allows you to specify the project names and where Terraform and usage files are located. You can also set environment variables and other options. version: 0.1 projects: - path: dev usage_file: dev/infracost-usage.yml env: NODE_ENV: dev - path: prod usage_file: prod/infracost-usage.yml env: AWS_ACCESS_KEY_ID: ${PROD_AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${PROD_AWS_SECRET_ACCESS_KEY} NODE_ENV: production When the config file is involved, you must replace the --path argument with --config-file in all your commands. Establishing Policies One more trick Infracost has up its sleeve is enforcing policies. Policies are rules that evaluate the output of infracost diff and stop the CI pipeline if a resource goes over budget. This feature allows managers and team leads to enforce limits. When the policy fails, the CI/CD pipeline stops with an error, preventing the infrastructure from being provisioned. When a policy is in place, Infracost warns us if any limits are exceeded. Infracost implements policies using Open Policy Agent (OPA), which uses the Rego language to encode policy rules. Rego has a ton of features, and it’s worth digging in to learn it thoroughly, but for our purposes, we only need to learn a few keywords: deny[out] defines a new policy rule that fails if the out object has failed: true msg: defines the error message shown when the policy fails. out: defines the logic that makes the policy pass or fails. input: references the contents of the JSON object generated with infracost diff The following example shows a policy that fails when the total budget exceeds $1,000: # policy.rego package infracost deny[out] { # define a variable maxMonthlyCost = 1000.0 msg := sprintf( "Total monthly cost must be less than $%.2f (actual diff is $%.2f)", [maxMonthlyCost, to_number(input.totalMonthlyCost)], ) out := { "msg": msg, "failed": to_number(input.totalMonthlyCost) >= maxMonthlyCost } } This is another example that fails if the cost difference is equal to or greater than $500. package infracost deny[out] { # maxDiff defines the threshold that you require the cost estimate to be below maxDiff = 500.0 msg := sprintf( "Total monthly cost diff must be less than $%.2f (actual diff is $%.2f)", [maxDiff, to_number(input.diffTotalMonthlyCost)], ) out := { "msg": msg, "failed": to_number(input.diffTotalMonthlyCost) >= maxDiff } } You can experiment and try several examples online on the OPA playground. To enforce a policy, you must add the --policy-path option in any of the infracost comment commands like this: curl -fsSL https://raw.githubusercontent.com/infracost/infracost/master/scripts/install.sh | sh checkout infracost diff --path . --usage-file usage.yml --compare-to baseline.json --format json --out-file /tmp/infracost-diff-commit.json infracost comment github --path=/tmp/infracost-diff-commit.json --repo=$SEMAPHORE_GIT_REPO_SLUG --commit=$SEMAPHORE_GIT_SHA --github-token=$GITHUB_API_KEY --policy-path policy.rego --behavior=update Conclusion The power to spin up resources instantly is a double-edged knife: a typo in a Terraform file can be a costly mistake. Staying proactive when managing our cloud infrastructure is essential to sticking to the budget and avoiding nasty surprises at the end of the month. If you’re already automating deployment with continuous deployment and managing services with Terraform, you may as well add Infracost to the mix to make more informed decisions and impose spending limits. Setting this up takes only a few minutes and can save thousands of dollars down the road.

This is the last part of the debugging series. To learn the rest, you’ll need to get the book “Practical Debugging at Scale: Cloud Native Debugging in Kubernetes and Production” or the course. One of the most frequently asked questions I receive is: can we do these things in VS Code? The answer is, unfortunately, no. But I elaborate on the debugging capabilities of VS Code in this video: “16 Missing Features in the VS Code Debugger” on YouTube. I’ll do a blog post that covers that next week. Below is the last video in the series: Transcript Welcome back to the ninth part of debugging at scale, where we really know the quality of your code. Remote debugging doesn’t always deal with a remote machine. We often need it when debugging into Kubernetes or Docker. We’ll delve more into that later, but for now, we’ll discuss the basic mechanics. How to connect, how to make it slightly less vulnerable to security attacks, and then we’ll discuss the problems of remote debugging. The Connection and Command Line We’ll start with a discussion around the connection. We first need to run the process that we’ll connect to remotely. To do that, we need to run a command similar to this one. Notice that this is a simplified version. In many cases, the argument should be embedded in configuration files. When you inspect your maven or gradle files, you might see many of the arguments listed here. This is how these things work under the hood. Let’s go over the command and break it down piece by piece to see that we understand it correctly. The first part is the launch of the Java command line. This is pretty obvious. We need quotes in bash since there’s a star at the end of the line, and bash wants to expand it. Without this quote, the command won’t work properly. Agent lib is the system that loads the native library wiring directly into the virtual machine, and JDWP is the Java Debug Wire Protocol. This is the underlying networking protocol used to communicate between the debugger and the running process. It’s a high-level protocol, that means it can be implemented on top of various transports. Typically, it’s implemented over TCP sockets, but it’s the same protocol we used to debug devices directly. You don’t need to know too much about JDWP, but the concept is simple. You send commands and can query the system. That’s what the IDE does for you. When you add a breakpoint, the IDE sends a JDWP command to add a breakpoint at the given location. When the breakpoint is hit, JDWP sends back an event to the IDE, indicating that the IDE can then query the details about the current environment, stack, variables, etc. In this case, we transfer the details via a server socket. We can use dt_shmem, which stands for shared memory, as the wire protocol. This is faster and useful for processes that have access to a shared memory area. This is actually pluggable, and you can build your own JDWP transport. This isn’t useful usually but speaks to the power and flexibility of the API. We can optionally suspend the virtual machine on launch if you want to debug something right from the start. I’ve set this to no, which means the VM will start running right away. If you set it to yes with the letter “y,” the VM will pause on launch and wait for the JDWP connection. This is the address and port we are listening on. In this case, I allow anyone to connect on port 5005. I can limit this to localhost only by changing the star character. This is probably the better approach. Although, it won’t make the protocol fully secure. This is the rest of the command, the class we’re running. Typically, you would have something more substantial here. In this case, I’m just running the PrimeMain class. To start debugging, we need to edit the run configuration in intellij. Connecting from IntelliJ/IDEA Next, we need to locate a configuration for remote debugging. Once I select that, we can add it. Notice it’s pre-configured with the defaults, such as port 5005. I give the new run configuration a name, and we’re ready to go with debugging the app. Notice there are many options to tune here, but we don’t need any of them. Also, check out this area right here. Seems familiar? That’s the exact line we discussed before. The IDE is showing us how to set up the command line for the remote process. This lets us verify that we entered everything correctly. We now have a new debug remote run configuration. We can switch to a different configuration from the same location. But when we want to do remote debugging, we need to toggle it here. Next, we need to press the debug button to run this command. We are now instantly connected to the running process. Once that is done, this feels and acts like any debugger instance launched from within the IDE. I can set a breakpoint, step over, inspect variables, etc., so why do it? In some cases, running the server locally in the IDE is impractical. A good example would be debugging a container on your own machine. That might not be trivial. Security Implications of JDWP Calling JDWP insecure is inaccurate. That would be like putting your house keys and home address wrapped in a nice gift wrapping with an itemized list of your valuables sorted by value in front of your house. This is an open door. An open door isn’t a security vulnerability. It’s an open door! JDWP is very insecure when used remotely. Locally, on your own machine, it isn’t a problem, but it has almost no security protections. There’s no solution for that. But there’s a very partial workaround of tunneling it over SSH. This is relatively trivial. Just use this command to open a tunnel between the remote machine to your local machine. For both sides, it will seem like local debugging. So the example I showed before (of connecting to a local host server) would work perfectly with this remote host as SSH will move all the packets back and forth securely. We can’t SSH into a Kubernetes container, but we can port forward, which is almost identical. We can do something similar to this command to forward the port from the given pod to the local machine and vice versa. Same idea as the SSH tunneling but appropriate to the Kubernetes world. Dangers of Remote Debugging In this final section, I want to talk about the dangers of remote debugging in production. Breakpoints break, seems obvious. That’s what they’re here to do. But if we run on a server, we block it completely by mistake. We can use trace points. As I said, they’re great. But they are no replacement to breakpoints, and an accidental click in the gutter can literally stop your server in its tracks. JDWP effectively allows remote code execution. Lets you access all the bytecode of the app, which is effectively the same as giving access to your full server source code. It lets attackers do almost anything since it wasn’t designed with security in mind. We need to relaunch the application with debugging enabled. That means killing the running process and starting it over again. Disconnecting existing users, etc. That isn’t great. Some operations in the debugger require more than one step in terms of the protocol. As a result, you could send a request to the debugger, lose your connection, and the debugger could be stuck in a problematic state. This is an inherent limitation of the JDWP protocol and can’t be worked around in a standard debugger. The problem is that even unintentional actions can demolish a server. A simple conditional breakpoint that invokes a method as part of the condition can demolish server performance and crash it. JDWP effectively allows remote code execution. Lets you access all the bytecode of the app, which is effectively the same as giving access to your full source code. It lets attackers do almost anything since it wasn’t designed with security in mind. Imagine placing a breakpoint where the user password is passed to authentication. If JDWP is open for your server, a member of your team might use that, and you will never know. There’s no tracking at all! 60% of security hacks happen from within the organization. If your company does remote debugging, they have no way of knowing whether an employee used that to manipulate the application state or siphon user details. There’s no tracking or anything. This can be in violation of various rules and regulations since it might expose personal user data. Remote debugging into production can trigger liability risks. I discuss some of the solutions for those problems both in the low-level tooling and in higher-level observability solutions. This is covered in the book and in the full course. Final Word With this, we finished the first part of the course. If you want to check out the full course; go to “debugagent.com” to learn more. The next video covers the strategies for debugging and the science of debugging. If you have any questions, please use the comments section below. Thank you!

Have you ever needed to make some HTTP requests in your Jenkins pipeline? How will you do that? The first that comes to mind is to use the simple CURL command. It's a powerful tool, and it will be fine for simple cases like the usual GET request. But what about more complicated issues? What if you need to execute POST with a huge JSON body and many headers? CURL command will be obscure and not maintainable. Also, it will be hell with escaping quotes and special symbols to make it works in the pipeline. While last interacting with this hell, I've thought about an alternative and found that - a powerful plugin, HTTP Request. To use the HTTP Request Plugin, you will need to install it in your Jenkins. This can be done through the Jenkins plugin manager, which allows you to browse and install plugins from the Jenkins interface. Once the plugin is installed, you can use it in pipelines with the keyword httpRequest. Main features: Request types GET, HEAD, POST, PUT, PATCH, DELETE Expected response code or string content Basic/Form authentication Connection timeout Add custom headers I've prepared a few examples to compare and contrast both approaches. These examples illustrate the key differences between executing HTTP requests in the pipeline and demonstrate how HTTP Request Plugin may be better suited for specific situations. By examining these examples, you can better understand each tool's strengths and weaknesses and determine which is the best fit for your needs. GET Request CURL stage('Execute Request') { steps { script { sh "curl https://dummyjson.com/products" } } } Plugin stage('Execute Request') { steps { httpRequest "https://dummyjson.com/products" } } Custom Headers CURL stage('Execute Request') { steps { script { sh "curl --location --request GET 'https://dummyjson.com/auth/products/1' \\\n" + "--header 'Content-Type: application/json' \\\n" + "--header 'Authorization: Bearer YOUR_TOKEN' \\\n" + "--data-raw ''" } } } Plugin stage('Execute Request') { steps { httpRequest customHeaders: [[name: 'Content-Type', value: 'application/json'], [name: 'Authorization', value: 'Bearer YOUR_TOKEN']], url: 'https://dummyjson.com/auth/products/1' } } POST Request With Complex Payload CURL stage('Execute Request') { steps { script { sh "curl --location --request POST 'https://dummyjson.com/carts/add' \\\n" + "--header 'Content-Type: application/json' \\\n" + "--data-raw '{\n" + " \"userId\": \"1\",\n" + " \"products\": [\n" + " {\n" + " \"id\": \"23\",\n" + " \"quantity\": 5\n" + " },\n" + " {\n" + " \"id\": \"45\",\n" + " \"quantity\": 19\n" + " }\n" + " ]\n" + "}'" } } } Plugin stage('Execute Request') { steps { httpRequest contentType: 'APPLICATION_JSON', httpMode: 'POST', requestBody: '''{ "userId": "1", "products": [ {"id": "23","quantity": 5}, {"id": "45","quantity": 19} }''', url: 'https://dummyjson.com/carts/add' } } Response Code and Content Validation CURL stage('Execute Request') { steps { script { def (String response, int code) = sh(script: "curl -s -w '\\n%{response_code}' https://dummyjson.com/products", returnStdout: true) .trim() .tokenize("\n") if(code != "200") { throw new Exception("Status code doesn't match. Received: $code" ) } if(!response.contains("userId")){ throw new Exception("Very strange message\n" + response) } } } } Plugin stage('Execute Request') { steps { httpRequest url:"https://dummyjson.com/products", validResponseCodes:'200', validResponseContent:'userId' } } The validation rule will fail to build with the following: hudson.AbortException: Fail: Status code 500 is not in the accepted range: 200 hudson.AbortException: Fail: Response doesn't contain expected content 'userId' Conclusion HTTP Request Plugin is a valuable tool for Jenkins users. It allows you to easily send HTTP requests as part of your Jenkins jobs, enabling you to automate tasks and integrate Jenkins with other tools and services. The plugin's user-friendly interface and additional functionality for handling responses make it easy to use and powerful. If you are looking to expand the capabilities of your Jenkins instance and streamline your workflow, the Jenkins HTTP Request Plugin is definitely worth considering. Feel free to read the official documentation and visit the GitHub repository.