Language Resources

DZone's Featured Languages Resources

Mastering Coroutine Execution: Yielding, Flow, and Practical Use Cases in Unity

By Denis Kondratev

After laying the groundwork in our previous article on the basics of Unity's coroutines, we're now ready to delve deeper into the mechanics that drive coroutine execution. This article aims to explore two key aspects that make coroutines a powerful tool in Unity: the concept of yielding and the coroutine's relationship with Unity's main game loop. Yielding is a cornerstone of coroutine functionality, allowing a coroutine to pause its execution and yield control to other routines. This feature enables you to write asynchronous code that can wait for specific conditions to be met, such as time delays or external data, before resuming its execution. We'll explore the different types of yield statements available in Unity, like yield return null and yield return new WaitForSeconds(), and discuss their implications on coroutine behavior. Moreover, understanding how coroutines fit into Unity's main game loop is crucial for leveraging their full potential. Unlike standard methods that execute all their code at once, coroutines have the ability to pause and resume, interleaving their execution with the main game loop. This allows for more flexible and efficient code, especially in scenarios like animations, AI behaviors, and timed events. To illustrate these concepts, we'll provide Unity C# code examples that demonstrate how yielding works and how coroutines are executed in relation to the main game loop. By the end of this article, you'll have a deeper understanding of coroutine mechanics, setting the stage for our discussion on practical use cases and advanced coroutine patterns in Unity. So, let's dive in and unravel the intricacies of coroutine execution in Unity. Yielding Execution One of the most powerful features of coroutines in Unity is the ability to yield execution. This means that a coroutine can pause its operation, allowing other functions or coroutines to run, and then resume from where it left off. This is particularly useful for breaking up tasks that would otherwise block the main thread, making your game unresponsive. The concept of yielding is central to how coroutines function. When a coroutine yields, it effectively says, "I have reached a point where I can pause, so go ahead and run other tasks." This is done using the yield keyword in C#, followed by a return statement that specifies the condition under which the coroutine should resume. Here's a simple example that uses yield return null, which means the coroutine will resume on the next frame: C# using System.Collections; using UnityEngine; public class SimpleYieldExample : MonoBehaviour { IEnumerator Start() { Debug.Log("Coroutine started: " + Time.time); yield return null; Debug.Log("Coroutine resumed: " + Time.time); } } In this example, the coroutine starts and logs the current time. It then yields, allowing other functions and coroutines to execute. On the next frame, it resumes and logs the time again, showing that it paused for approximately one frame. Different Types of Yield Statements Unity provides several types of yield statements, each with its own use case: yield return null: Pauses the coroutine until the next frame yield return new WaitForSeconds(float seconds): Pauses the coroutine for a specified number of seconds yield return new WaitForEndOfFrame(): Pauses the coroutine until the end of the frame, after all graphical rendering is done yield return new WaitForFixedUpdate(): Pauses the coroutine until the next fixed frame rate update function Each of these yield statements serves a different purpose and can be crucial for various tasks like animations, loading, or any time-sensitive operations. Understanding the concept of yielding and the different types of yield statements available can significantly enhance your ability to write efficient and effective coroutines in Unity. In the next section, we'll explore how these coroutines fit into Unity's main game loop, providing a more holistic understanding of coroutine execution. Coroutine Execution Flow Understanding how coroutines operate within Unity's main game loop is crucial for mastering their behavior and capabilities. While it's easy to think of coroutines as separate threads running in parallel, they are actually executed within Unity's main game loop. However, their ability to pause and resume sets them apart and allows for more complex and flexible behavior. How Coroutines Run in Conjunction With Unity's Main Game Loop Coroutines in Unity are not separate threads but are instead managed by Unity's main game loop. When a coroutine yields, it essentially steps out of the game loop temporarily, allowing other game processes to take place. It then re-enters the loop either in the next frame or after a specified condition is met. Here's a simplified example to demonstrate this: C# using System.Collections; using UnityEngine; public class CoroutineFlowExample : MonoBehaviour { void Start() { StartCoroutine(MyCoroutine()); } IEnumerator MyCoroutine() { Debug.Log("Coroutine started at frame: " + Time.frameCount); yield return null; Debug.Log("Coroutine resumed at frame: " + Time.frameCount); } } In this example, the coroutine starts and logs the current frame count. It then yields, stepping out of the game loop. On the next frame, it resumes and logs the frame count again. You'll notice that the frame count will have incremented, indicating that the game loop continued while the coroutine was paused. An Illustration or Example Showing the Flow of Execution in Coroutines To further illustrate how a coroutine's execution is interleaved with the main game loop, consider the following pseudo-code that represents a simplified Unity game loop: Plain Text Game Loop: 1. Update Physics 2. Run Coroutines 3. Render Frame 4. Repeat Now, let's say we have a coroutine that performs some logic, waits for 2 seconds, and then continues: C# IEnumerator MyWaitingCoroutine() { Debug.Log("Logic Part 1: Frame " + Time.frameCount); yield return new WaitForSeconds(2); Debug.Log("Logic Part 2: Frame " + Time.frameCount); } In this scenario, "Logic Part 1" would execute during the "Run Coroutines" step of the game loop. The coroutine would then yield, waiting for 2 seconds. During this time, the game loop would continue to cycle through its steps, updating physics and rendering frames. After approximately 2 seconds, the coroutine would resume, executing "Logic Part 2" during the "Run Coroutines" step. Understanding this interleaved execution is key to mastering coroutines in Unity. It allows you to write code that is both efficient and easy to manage, as you can break up tasks into smaller parts without blocking the main game loop. In the next section, we'll explore some practical use cases where this capability is particularly beneficial. Use Cases for Coroutines Coroutines are a versatile tool in Unity, capable of handling a wide range of scenarios that require asynchronous or time-dependent behavior. Their ability to pause and resume makes them particularly useful for tasks that are too complex or time-consuming to be executed in a single frame. In this section, we'll explore some common use cases where coroutines shine and provide practical Unity C# examples to demonstrate their utility. Timed Events Coroutines are excellent for managing events that need to happen after a certain amount of time has passed. For example, you might want to delay a game character's action or trigger an event after a countdown. C# IEnumerator TriggerTimedEvent() { yield return new WaitForSeconds(5); Debug.Log("Timed event triggered!"); } In this example, the message "Timed event triggered!" will be logged after a 5-second delay. Animations Coroutines can also be used to control animations, especially those that require precise timing or sequencing. C# IEnumerator AnimateObject(Vector3 targetPosition) { Vector3 startPosition = transform.position; float journeyLength = Vector3.Distance(startPosition, targetPosition); float startTime = Time.time; float speed = 1.0f; float distanceCovered = (Time.time - startTime) * speed; float fractionOfJourney = distanceCovered / journeyLength; while (fractionOfJourney < 1) { distanceCovered = (Time.time - startTime) * speed; fractionOfJourney = distanceCovered / journeyLength; transform.position = Vector3.Lerp(startPosition, targetPosition, fractionOfJourney); yield return null; } } Here, the object will move from its current position to a target position, interpolating its position over time. AI Behaviors Coroutines can be used to manage complex AI behaviors, such as decision-making processes that occur over multiple frames. C# IEnumerator AIDecisionMaking() { Debug.Log("AI thinking..."); yield return new WaitForSeconds(2); Debug.Log("AI made a decision!"); } In this example, the AI "thinks" for 2 seconds before making a decision, represented by the log statements. Showcase Some Practical Examples in Unity Consider a game where a player's health regenerates over time. A coroutine can manage this efficiently: C# IEnumerator RegenerateHealth() { while (true) { if (playerHealth < 100) { playerHealth++; Debug.Log("Health: " + playerHealth); } yield return new WaitForSeconds(1); } } In this example, the player's health increases by 1 every second until it reaches 100, at which point the coroutine will still run but won't increase the health. Understanding these practical applications of coroutines can significantly improve the way you approach problem-solving in Unity. Whether it's managing time-dependent events, controlling animations, or implementing complex AI behaviors, coroutines offer a flexible and efficient way to achieve your goals. In the next article, we'll delve deeper into best practices, performance considerations, and more advanced coroutine patterns. Conclusion As we've explored in this article, understanding the mechanics of coroutine execution is not just an academic exercise; it's a practical skill that can significantly enhance your Unity projects. Coroutines offer a robust and flexible way to manage asynchronous and time-dependent tasks, from simple timed events and animations to more complex AI behaviors. For instance, we've seen how you can use coroutines to manage health regeneration in a game: C# IEnumerator RegenerateHealth() { while (true) { if (playerHealth < 100) { playerHealth++; Debug.Log("Health: " + playerHealth); } yield return new WaitForSeconds(1); } } This example demonstrates that coroutines can be an effective way to handle game mechanics that are dependent on time or other asynchronous events. The yield return new WaitForSeconds(1); line is a powerful yet straightforward way to introduce a delay, allowing other game processes to continue running smoothly. But this is just scratching the surface. As you become more comfortable with coroutines, you'll find that they can be used for much more than simple delays and animations. They can manage complex state machines for AI, handle user input in a non-blocking manner, and even manage resource-intensive tasks by spreading the workload over multiple frames. In the next article, we'll delve deeper into the world of coroutines, exploring best practices to optimize your usage of this feature. We'll look at performance considerations, such as how to avoid common pitfalls that can lead to frame rate drops. We'll also explore advanced coroutine patterns, like nested coroutines, and how to manage multiple coroutines efficiently. By mastering coroutines, you're adding a powerful tool to your Unity development toolkit. Whether you're developing a simple mobile game or a complex virtual reality experience, coroutines can help you create more efficient and responsive games. So stay tuned for our next piece, where we'll take your coroutine skills to the next level. More

The Easy Way To Deploy Multilanguage Apps to Kubernetes

By Sylvain Kalache

Kubernetes can be intricate to manage, and companies want to leverage its power while avoiding its complexity. A recent survey found that 84% of companies don’t see value in owning Kubernetes themselves. To address this complexity, Cloud Foundry introduced open-source Korifi, which preserves the classic Cloud Foundry experience of being able to deploy apps written in any language or framework with a single cf push command. But the big difference is that this time, apps are pushed to Kubernetes. In this tutorial, we’ll explore how to use Korifi to deploy web applications written in different languages: Ruby, Node.js, ASP.NET, and PHP. I will also provide insights into Korifi’s functioning and basic configuration knowledge, helping you kick-start your multi-cloud, multitenant, and polyglot journey. Ruby For all the examples in this tutorial, I will use sample web applications that you can download from this GitHub repository, but feel free to use your own. You can also find instructions on installing Korifi in this article, which guides you through the easiest way to achieve that by running two Bash scripts that will set everything up for you. Once you have Korifi installed and have cloned a Ruby sample application, go into the root folder and type the following command: Shell cf push my-ruby-app That’s it! That is all you need to deploy a Ruby application to Kubernetes. Keep in mind that while the first iteration of cf push will take some time as Korifi needs to download a number of elements (I will explain this in the next paragraph); all subsequent runs will be much faster. At any point, if you want to check the status of a Korifi app, you can use the cf app command, which, in the case of our Ruby app, would be: Shell cf app my-ruby-app Node.js Before deploying a Node.js application to Kubernetes using Korifi, let me explain how it works under the hood. One of the key components at play here is Cloud Native Buildpacks. The concept was initially introduced in 2011 and adopted by PaaS providers like Google App Engine, GitLab, Deis, and Dokku. This project became a part of the CNCF in 2018. Buildpacks are primarily designed to convert an application’s source code into an OCI image, such as a Docker image. This process unfolds in two steps: first, it scans the application to identify its dependencies and configures them for seamless operation across diverse clouds. Then, it assembles an image using a Builder, a structured amalgamation of Buildpacks, a foundational build image, a lifecycle, and a reference to a runtime image. Although you have the option to construct your own build images and Buildpacks, you can also leverage those provided by established entities such as Google, Heroku, and Paketo Buildpacks. In this tutorial, I will exclusively use ones provided by Paketo — an open-source project that delivers production-ready Buildpacks for popular programming languages. Let’s briefly demonstrate what Korifi does by manually creating a Buildpack from a Node.js application. You can follow the installation instructions here to install the pack CLI. Then, get into the root folder of your application and run the following command: Shell pack build my-nodejs-app --builder paketobuildpacks/builder:base Your Node.js OCI image is available; you can check this by running the command: Shell docker images Once the Docker image is ready, Korifi utilizes Kubernetes RBAC and CRDs to mimic the robust Cloud Foundry paradigm of orgs and spaces. But the beauty of Korifi is that you don’t have to manage any of that. You only need one command to push a Node.js application to Kubernetes: Shell cf push my-nodejs-app That’s it! ASP.NET Now, let’s push an ASP.NET application. If you run cf push my-aspnet-app, the build will fail, and you will get the following error message: Shell BuildFail: Check build log output FAILED 2023-08-11T19:12:58.11+0000 [STG/] OUT ERROR: No buildpack groups passed detection. 2023-08-11T19:12:58.11+0000 [STG/] OUT ERROR: failed to detect: buildpack(s) failed with err These logs tell us that Korifi may not know a valid Buildpack to package an ASP.NET application. We can verify that by running the following command: Shell cf buildpacks You should get the following output, and we can see that there are no .NET-related buildpacks. Shell position name stack enabled locked filename 1 paketo-buildpacks/java io.buildpacks.stacks.jammy true false paketo-buildpacks/java@9.18.0 2 paketo-buildpacks/go io.buildpacks.stacks.jammy true false paketo-buildpacks/go@4.4.5 3 paketo-buildpacks/nodejs io.buildpacks.stacks.jammy true false paketo-buildpacks/nodejs@1.8.0 4 paketo-buildpacks/ruby io.buildpacks.stacks.jammy true false paketo-buildpacks/ruby@0.39.0 5 paketo-buildpacks/procfile io.buildpacks.stacks.jammy true false paketo-buildpacks/procfile@5.6.4 To fix that, first, we need to tell Korifi which Buildpack to use for an ASP.NET application by editing the ClusterStore: Shell kubectl edit clusterstore cf-default-buildpacks -n tutorial-space Make sure to replace tutorial-space with the value you used during your Korifi cluster configuration. Add the line – image: gcr.io/paketo-buildpacks/python; your file should look like this: Shell spec: sources: - image: gcr.io/paketo-buildpacks/java - image: gcr.io/paketo-buildpacks/nodejs - image: gcr.io/paketo-buildpacks/ruby - image: gcr.io/paketo-buildpacks/procfile - image: gcr.io/paketo-buildpacks/go - image: gcr.io/paketo-buildpacks/python Then we need to tell Korifi in which order to use Buildbacks by editing our ClusterBuilder: Shell kubectl edit clusterbuilder cf-kpack-cluster-builder -n tutorial-space Add the line – id: paketo-buildpacks/dotnet-core at the top of the spec order list. your file should look like this: Shell spec: sources: - image: gcr.io/paketo-buildpacks/java - image: gcr.io/paketo-buildpacks/nodejs - image: gcr.io/paketo-buildpacks/ruby - image: gcr.io/paketo-buildpacks/procfile - image: gcr.io/paketo-buildpacks/go - image: gcr.io/paketo-buildpacks/python If everything was done right, you should see the .NET Core Paketo Buildpack in the list output by the cf buildpacks command. Finally, you can simply run cf push my-aspnet-app to push your ASP.NET application to Kubernetes. PHP We need to follow the same process for PHP with the Buildpack paketo-buildpacks/php that needs to be added to the ClusterStore and ClusterBuilder. For anyone using Korifi version 0.9.0 released a few days ago, the issue that I am about to discuss has been fixed. But in case you are using an older version, running cf push my-php-app will fail and return the following error message: Shell [APP/] OUT php: error while loading shared libraries: libxml2.so.2: cannot open shared object file: No such file or directory The OCI image is missing the libxml library, which is required by PHP, this is probably due to the builder not supporting PHP. To check that, let’s look what builder Korifi is using by running this command: Shell kubectl describe clusterbuilder cf-kpack-cluster-builder | grep 'Run Image' Which will output the following: Shell Run Image: index.docker.io/paketobuildpacks/run-jammy-base@sha256:4cf369b562808105d3297296efea68449a2ae17d8bb15508f573cc78aa3b3772a As you can see, Korifi currently uses Paketo Jammy Base, which, according to its Github repo description, does not support PHP. You also can check that by looking at the builder’s builder.toml file or by running the command pack builder suggest, which will return the output: Shell Suggested builders: [...] Paketo Buildpacks: paketobuildpacks/builder-jammy-base Ubuntu 22.04 Jammy Jellyfish base image with buildpacks for Java, Go, .NET Core, Node.js, Python, Apache HTTPD, NGINX and Procfile Paketo Buildpacks: paketobuildpacks/builder-jammy-buildpackless-static Static base image (Ubuntu Jammy Jellyfish build image, distroless-like run image) with no buildpacks included. To use, specify buildpack at build time. Paketo Buildpacks: paketobuildpacks/builder-jammy-full Ubuntu 22.04 Jammy Jellyfish full image with buildpacks for Apache HTTPD, Go, Java, Java Native Image, .NET, NGINX, Node.js, PHP, Procfile, Python, and Ruby [...] While Jammy Base does not support PHP, the Jammy Full builder does. There are multiple ways to get Korifi to use another builder, I will just cover one way in this tutorial. This way assumes that we used the easy way to install Korifi with the deploy-on-kind.sh script. You need to go to Korifi source code and edit the file scripts/assets/values.yaml so that the fields clusterStackBuildImage and clusterStackRunImage are set to paketobuildpacks/build-jammy-full by running this command: Shell sed -i 's/base/full/g' scripts/assets/values.yaml` Then, run the scripts/deploy-on-kind.sh script. That’s it! Korifi will use the Jammy full builder, and Korifi will be able to deploy your PHP application with a cf push my-php-app command. Summary Hopefully, now you’ve experienced just how easy it is to use Korifi to deploy applications to Kubernetes written in Ruby, Node.js, ASP.NET, and PHP. You can stay tuned with the Korifi project by following Cloud Foundry X account and joining the Slack workspace. More

Python “Magic” Methods (Part 2)

By Nicolas Fränkel

CORE

Building LangChain Applications With Amazon Bedrock and Go: An Introduction

By Abhishek Gupta

CORE

HTML Layout

By Madhuri Hammad

Using C# as a Scripting Language

At AINIRO.IO we've just created a new release of Magic, where the most important feature is the ability to dynamically compile C# code and load the resulting IL code into the AppDomain, almost turning C# into an "interpreted language" due to an execution model that is more similar to PHP and JavaScript than traditionally compiled languages. This has a lot of benefits, especially for Business Process Workflows, since it allows you to use C# in a dynamic runtime, where you've got dynamic actions that are executed from Hyperlambda being a high-level execution orchestration runtime. Below is some example code illustrating the idea: C# using System; using magic.node; using magic.node.extensions; using magic.signals.contracts; [Slot(Name = "foo")] public class Foo : ISlot { public void Signal(ISignaler signaler, Node input) { input.Value = $"Hello {input.GetEx()}, najs to meet you"; } } The point about the above code, of course, is that it implements the ISlot interface, which allows me to interact with it from Hyperlambda, as illustrated below. C# foo:Thomas Hansen The above Hyperlambda, of course, will invoke my C# slot, passing in "Thomas Hansen," and my C# slot, of course, will do some simple string concatenation, returning the result to the caller. If you save the above C# code as "/etc/csharp/foo.cs", you can execute the following Hyperlambda code to dynamically compile the file and execute the slot. C# // Loading file. io.file.load:/etc/csharp/slot.cs // compiling file into an assembly. system.compile references .:netstandard .:System.Runtime .:System.ComponentModel .:System.Private.CoreLib .:magic.node .:magic.node.extensions .:magic.signals.contracts code:x:@io.file.load assembly-name:foo.dll // Loading assembly as plugin now that we've created it. system.plugin.load:x:@system.compile // Invoking dynamically created C# slot. .name:John Doe foo:x:@.name // Unloading plugin. system.plugin.unload:foo.dll Notice that the above [system.compile] never saves the assembly but returns it as a byte[]. To save the compiled code, you can use, for instance [io.file.save.binary]. In the video below, I am demonstrating some features related to this and showing you how you can almost treat C# as if it's a 100% dynamic scripting language due to the dynamic nature of the process. This has a lot of advantages, especially related to BPW or Business Process Workflows, where you've got tons of smaller building blocks or composables you need to orchestrate together dynamically without having to go through an entire process of deployment and more rigid processes. This allows you to dynamically orchestrate C# snippets together, where Hyperlambda becomes the orchestration tool, loosely coupling building blocks of C# code together that somehow perform a larger task. Due to the dynamic nature of Hyperlambda again, allowing you to build anything from scheduled tasks to HTTP endpoints, this has a lot of really interesting advantages for more complex domains, where the end state of your system is in constant flux, possibly due to integrating with hundreds of different parts, where each part is a separate application, often changing over time, making statically compiled code sub-optimal. Statically compiled code is amazing, and you should, of course, prefer it when you can — However, there are problem domains it is fundamentally incompatible with — Workflows being one example. Now, with the ability to compile C# code on the fly in Hyperlambda, this is no longer a problem, and you can use statically compiled C# as much as you wish for such problems. As long as you obey the Hyperlambda interface being the ISlot interface, allowing Hyperlambda to orchestrate your code together.

By Thomas Hansen

CORE

The Best Programming Languages for Kids

Teaching kids to program is not just about technicalities or computers. It is about unlocking their potential and instilling some of the most crucial life skills that help them survive in the ever-evolving technological world. In this article, we will find out the importance of teaching coding to kids as well as list some of the best programming languages that can be introduced at an early age. Why Teach Kids to Code? Coding has become one of the most crucial skills to have in the 21st century. It is no longer considered as a skill for professionals or experts. Here are some of the major advantages of teaching coding to kids can have Logical thinking: Coding not only involves solving a problem or developing an idea but also encourages questioning, assumptions, and listing out step-by-step plans of action. It promotes logical thought processes to handle any situation or challenge. Creativity: Coding offers an opportunity for kids to explore and express their imaginations and ideas via a digital platform. Kids have the freedom to choose any functionalities and endless possibilities while addressing an issue or building games. Problem-solving: Coding often requires kids to break down complex issues or projects, analyze them, find possible solutions, and implement them. This makes kids systematic thinkers looking for the best solutions. Future-proofing: Coding skills undoubtedly open the door to a number of exciting career opportunities. From developer to data scientist, analyst to cybersecurity, machine learning to AI, the job market is endless. Best Programming Languages for Kids There are hundreds of programming languages available today, and choosing the best programming languages can be a challenging task. Here is a list of 4 top programming languages in 2024 that are highly recommended for kids to learn. 1. JavaScript (Scripting Programming Language) JavaScript is one of the most used programming languages. Interactivity: JavaScript makes interactivity of web pages and applications possible. It allows users to take real-time actions and enhances functionality by creating a user-friendly experience via animations, buttons, and more. Real-world applications: JavaScript has a wide real-world application, and most websites and applications use it, including Facebook, Google Maps, and Gmail. Easy to start: JavaScript doesn't have hard-to-grasp syntax and a dedicated platform to master. Kids can start it even with a simple text editor and relatively simple lines of code to develop exciting projects. 2. Python Programming Language Python is among the most popular programming languages today. It is a high-level and versatile language that is used for web development, data analysis, machine learning, automation, and more. Clear and readable: The major advantage of using Python is that it doesn't have a complex syntax to adopt. The syntaxes are very easy to read and write, which reduces coding errors drastically and makes it ideal for a beginner-core programming language. Educational resources: Python is an open-source language and has a community of its own. It also has a huge library and enough resources available, be it in terms of tutorials, guides, or sample projects, that allows new learners to create, clear doubts, and get ideas to start with. Problem-solving: Just like any other programming language, python makes it possible to solve or develop a wide range of exciting projects with much ease. 3. CSS (Cascading Style Sheets) CSS stands for Cascading Style Sheets. It is a styling sheet that contains information regarding the layout, colors, fonts, and visual presentation of a webpage: CSS is responsible for making a webpage attractive and appealing to the users. It allows the designer to try and experiment with different aesthetics of a web page. CSS offers an excellent opportunity for kids to explore their creativity by developing color schemes, animations, and a more engaging user interface. CSS is responsible for making the web content adaptable to different screen sizes and devices without compromising the quality and user experience. 4. HTML (HyperText Markup Language) HTML stands for Hypertext Markup Language. It is a lightweight coding language that is very easy to learn and use. Structure: It is responsible for structuring and organizing various content or elements on the web pages, e.g., title, heading, paragraphs, body, table, etc. Semantics: HTML uses tags to identify or mark the contents on the web pages. These tags help the search engines understand the content and make it accessible to the user. For example: HTML <header>, <nav>, <article>, <footer> Integration: HTML allows various features or components to be easily incorporated on a webpage. This basically serves as a backbone to a web page — e.g., CSS, Widgets, CMS, etc. The top programming languages mentioned above have been carefully chosen, and mastering them will give kids exposure to a broad field of programming. This is due to the fact that the list includes languages that help in developing fully working websites or applications, including designing, presentations, functionality, and accessibility. Additionally, these languages are beginner-friendly, making them a great option for newcomers.

By Lekh raj

Transitioning From Groovy to Kotlin for Gradle Android Projects

The world of Android app development is constantly evolving, and so are the tools and languages used to build these apps. Gradle, a popular build system, has been an integral part of Android development for years. In the past, Gradle build scripts for Android projects were written in Groovy, but with the introduction of Kotlin, developers now have the option to write their build scripts in a more modern and concise language. In this article, we'll explore the transition from Groovy to Kotlin for Gradle Android projects and discuss the benefits and steps involved in making this shift. Why Transition to Kotlin for Gradle? Modern language: Kotlin is a modern, statically typed language that offers features not present in Groovy, making build scripts more concise and expressive. It is designed to be fully interoperable with Java, which is crucial for Android development. Type safety: Kotlin is known for its strong type safety, reducing the likelihood of runtime errors. With Groovy, you might encounter runtime issues due to dynamic typing. Improved tooling support: The Android Studio IDE has excellent support for Kotlin, making it easier to write, read, and maintain Gradle scripts. Code completion, refactoring, and error checking are some of the benefits you'll experience when using Kotlin. Conciseness: Kotlin's concise syntax can lead to shorter, more readable code. This is particularly beneficial in build scripts, which often involve complex logic. Transitioning to Kotlin Step-By-Step Here's a step-by-step guide on how to transition from Groovy to Kotlin for Gradle Android projects: 1. Check Kotlin Version Ensure that you have a recent version of Kotlin installed. You can do this by adding the Kotlin DSL plugin to your project. You can find the latest version on the Kotlin website Kotlin Gradle Plugin Portal. Kotlin plugins { kotlin("jvm") version "latest_version_here" } Replace "latest_version_here" with the actual version number you obtained from the Kotlin website or Gradle plugin portal. This ensures that you're using the most up-to-date version of the Kotlin plugin for your Gradle Android project. 2. Convert Gradle Files Start by converting your project's build.gradle files to Kotlin DSL files (.kts). You can do this by renaming the files or by selecting the "Convert to Kotlin DSL" option in Android Studio. Groovy (build.gradle) Groovy android { compileSdkVersion 30 defaultConfig { applicationId "com.example.myapp" minSdkVersion 21 targetSdkVersion 30 } } Kotlin (build.gradle.kts) Kotlin android { compileSdkVersion(30) defaultConfig { applicationId = "com.example.myapp" minSdkVersion(21) targetSdkVersion(30) } } 3. Update Build Script Modify your build.gradle.kts script to use Kotlin syntax. You'll notice that variable declarations, function definitions, and other aspects of the script will differ from Groovy. Be prepared for a bit of a learning curve if you're new to Kotlin. 4. Dependencies and Plugins Ensure that any third-party dependencies and Gradle plugins used in your project are compatible with Kotlin DSL. Most widely used libraries and plugins already support Kotlin, but it's essential to verify this. Groovy (build.gradle): Groovy apply plugin: 'kotlin-android' apply plugin: 'kotlin-android-extensions' dependencies { implementation 'com.android.support:appcompat-v7:28.0.0' implementation 'com.google.code.gson:gson:2.8.6' } Kotlin (build.gradle.kts): Kotlin plugins { kotlin("android") kotlin("android.extensions") } dependencies { implementation 'com.android.support:appcompat-v7:28.0.0' implementation 'com.google.code.gson:gson:2.8.6' } 5. Using Kotlin's Extension Functions Kotlin allows you to define extension functions to make your Gradle build scripts more concise and expressive. Here's an example of defining an extension function to configure ProGuard rules: Kotlin fun ProguardFiles.getDefaultProguardFile(name: String) = getDefaultFile("${name}.pro") android { buildTypes { release { proguardFiles(getDefaultProguardFile("proguard-android.txt"), getDefaultProguardFile("proguard-rules.pro")) } } } This extension function simplifies the code by encapsulating the logic of getting the default ProGuard file. 6. Test and Debug After converting your build scripts, thoroughly test your build process. Be on the lookout for errors or unexpected behavior, as syntax differences can lead to issues. 7. Migration in Stages It's often a good idea to transition gradually. Start with a small, less critical module or subproject before migrating your entire project. This allows you to get comfortable with the new syntax and identify potential issues. 8. Leverage Kotlin Features As you migrate, take advantage of Kotlin's features. For example, you can use Kotlin's powerful extension functions to make your build scripts more concise and expressive. 9. Continuous Learning Kotlin is a rich language with many features. Continue to learn and explore how you can improve your Gradle scripts by leveraging Kotlin's capabilities. Benefits and Future-Proofing Transitioning to Kotlin for your Gradle Android projects may require some effort, but it's a worthwhile investment. The benefits of improved tooling, type safety, and conciseness can significantly enhance your development process. Furthermore, as Kotlin continues to gain traction in the Android development community, transitioning your Gradle scripts to Kotlin is a step toward future-proofing your projects. In conclusion, the transition from Groovy to Kotlin for Gradle Android projects can lead to more robust and maintainable build scripts. Embracing Kotlin's modern features and improved tooling support can make your development process more efficient and less error-prone. It's a step forward in keeping your Android projects up-to-date with the latest technologies and best practices.

By Volha Kurcheuskaya

AWS NoSQL Performance Lab Using Python

In most financial firms, online transaction processing (OLTP) often relies on static or infrequently updated data, also called reference data. Reference data sources don’t always require ACID transaction capabilities, rather need support for fast read queries often based on simple data access patterns, and event-driven architecture to ensure the target systems remain up-to-date. NoSQL databases emerge as ideal candidates to meet these requirements, and cloud platforms such as AWS offer managed and highly resilient data ecosystems. In this article, I am not going to determine which AWS NoSQL database is better: the concept of a better database only exists within a specific purposeful context. I will share a coding lab to measure the performance of AWS-managed NoSQL databases such as DynamoDB, Cassandra, Redis, and MongoDB. Performance Testing I will start by defining the performance test case, which will concurrently insert a JSON payload 200 times and then read it 200 times. JSON Payload The base/parent class in base_db.py implements the test case logic of executing 10 concurrent threads to create and read 200 records. Python #imports ..... class BaseDB: def __init__(self, file_name='instrument.json', threads=10, records=20): ................................... def execute(self): create_threads = [] for i in range(self.num_threads): thread = threading.Thread( target=self.create_records, args=(i,)) create_threads.append(thread) thread.start() for thread in create_threads: thread.join() read_threads = [] for i in range(self.num_threads): thread = threading.Thread(target=self.read_records, args=(i,)) read_threads.append(thread) thread.start() for thread in read_threads: thread.join() self.print_stats() Each thread executes the write/read routine in the create_records and read_records, respectively. Notice that these functions do not include any database-specific logic, but rather, measure the performance of each read-and-write execution. Python def create_records(self, thread_id): for i in range(1, self.num_records + 1): key = int(thread_id * 100 + i) start_time = time.time() self.create_record(key) end_time = time.time() execution_time = end_time - start_time self.performance_data[key] = {'Create Time': execution_time} def read_records(self, thread_id): for key in self.performance_data.keys(): start_time = time.time() self.read_record(key) end_time = time.time() execution_time = end_time - start_time self.performance_data[key]['Read Time'] = execution_time Once the test case is executed, the print_stats function prints the execution metrics such as the read/write mean and the standard deviation (stdev) values, which indicate database read/write performance and consistency (smaller stdev implies more consistent execution performance). Python def print_stats(self): if len(self.performance_data) > 0: # Create a Pandas DataFrame from performance data df = pd.DataFrame.from_dict(self.performance_data, orient='index') if not df.empty: df.sort_index(inplace=True) # Calculate mean and standard deviation for each column create_mean = statistics.mean(df['Create Time']) read_mean = statistics.mean(df['Read Time']) create_stdev = statistics.stdev(df['Create Time']) read_stdev = statistics.stdev(df['Read Time']) print("Performance Data:") print(df) print(f"Create Time mean: {create_mean}, stdev: {create_stdev}") print(f"Read Time mean: {read_mean}, stdev: {read_stdev}") NoSQL Code Unlike relational databases that support standard SQL, each NoSQL database has its own SDK. The child test case classes for each NoSQL database only need to implement a constructor and create_record/read_recod functions that contain proprietary database SDK to instantiate a database connection and to create/read records in a few lines of code. DynamoDB Test Case Python import boto3 from base_db import BaseDB class DynamoDB (BaseDB): def __init__(self, file_name='instrument.json', threads=10, records=20): super().__init__(file_name, threads, records) dynamodb = boto3.resource('dynamodb', region_name='us-east-1') table_name = 'Instruments' self.table = dynamodb.Table(table_name) def create_record(self, key): item = { 'key': key, 'data': self.json_data } self.table.put_item(Item=item) def read_record(self, key): self.table.get_item(Key={'key': key}) if __name__ == "__main__": DynamoDB().execute() AWS Setup To execute these performance test cases in an AWS account, you should follow these steps: Create an EC2 IAM role with privileges to access the required AWS data services. Launch an EC2 instance and assign the newly created IAM role. Create each NoSQL database instance. IAM Role DynamoDB Table Cassandra Keyspace/Table Please note the DB host and credentials were hardcoded and removed in the mongo_db.py and redis_db.py modules and will need to be updated with the corresponding database connection setting for your AWS account. To connect to DynamoDB and Cassandra, I opted to use the Boto3 session credentials temporarily assigned to the db_performnace_iam_role IAM Role. This code will run in any AWS account in the East 1 region without any modification. Python class CassandraDB(BaseDB): def __init__(self, file_name='instrument.json', threads=10, records=20): super().__init__(file_name=file_name, threads=threads, records=records) self.json_data = json.dumps( self.json_data, cls=DecimalEncoder).encode() # Cassandra Keyspaces configuration contact_points = ['cassandra.us-east-1.amazonaws.com'] keyspace_name = 'db_performance' ssl_context = SSLContext(PROTOCOL_TLSv1_2) ssl_context.load_verify_locations('sf-class2-root.crt') ssl_context.verify_mode = CERT_REQUIRED boto_session = boto3.Session(region_name="us-east-1") auth_provider = SigV4AuthProvider(session=boto_session) cluster = Cluster(contact_points, ssl_context=ssl_context, auth_provider=auth_provider, port=9142) self.session = cluster.connect(keyspace=keyspace_name) Connect to the EC2 instance (I used the Session Manager), and run the following Shell script to perform these tasks: Install Git. Install Pythion3. Clone the GitHub performance_db repository. Install and activate the Python3 virtual environment. Install 3rd party libraries/dependencies. Execute each test case. Shell sudo yum install git sudo yum install python3 git clone https://github.com/dshilman/db_performance.git sudo git pull cd db_performance python3 -m venv venv source ./venv/bin/activate sudo python3 -m pip install -r requirements.txt cd code sudo python3 -m dynamo_db sudo python3 -m cassandra_db sudo python3 -m redis_db sudo python3 -m mongo_db You should see the following output for the first two test cases: (venv) sh-5.2$ sudo python3 -m dynamo_dbPerformance Data: Create Time Read Time1 0.336909 0.0314912 0.056884 0.0533343 0.085881 0.0313854 0.084940 0.0500595 0.169012 0.050044.. ... ...916 0.047431 0.041877917 0.043795 0.024649918 0.075325 0.035251919 0.101007 0.068767920 0.103432 0.037742 [200 rows x 2 columns]Create Time mean: 0.0858926808834076, stdev: 0.07714510154026173Read Time mean: 0.04880355834960937, stdev: 0.028805479258627295Execution time: 11.499964714050293 (venv) sh-5.2$ sudo python3 -m cassandra_dbPerformance Data: Create Time Read Time1 0.024815 0.0059862 0.008256 0.0069273 0.008996 0.0098104 0.005362 0.0058925 0.010117 0.010308.. ... ...916 0.006234 0.008147917 0.011564 0.004347918 0.007857 0.008329919 0.007260 0.007370920 0.004654 0.006049 [200 rows x 2 columns]Create Time mean: 0.009145524501800537, stdev: 0.005201661271831082Read Time mean: 0.007248317003250122, stdev: 0.003557610695674452Execution time: 1.6279327869415283 Test Results DynamoDB Cassandra MongoDB Redis Create mean: 0.0859stdev: 0.0771 mean: 0.0091stdev: 0.0052 mean: 0.0292std: 0.0764 mean: 0.0028stdev: 0.0049 Read mean: 0.0488stdev: 0.0288 mean: 0.0072stdev: 0.0036 mean: 0.0509std: 0.0027 mean: 0.0012stdev: 0.0016 Exec Time 11.45 sec 1.6279 sec 10.2608 sec 0.3465 sec My Observations I was blown away by Cassandra’s fast performance. Cassandra support for SQL allows rich access pattern queries and AWS Keyspaces offer cross-region replication. I find DynamoDB's performance disappointing despite the AWS hype about it. You should try to avoid the cross-partition table scan and thus must use an index for each data access pattern. DynamoDB global tables enable cross-region data replication. MongoDB has a very simple SDK, is fun to use, and has the best support for the JSON data type. You can create indexes and run complex queries on nested JSON attributes. As new binary data formats are emerging, MongoDB may lose its appeal. Redis performance is amazingly fast, however, at the end of the day, it’s a key/value cache even if it supports complex data types. Redis offers powerful features such as pipelining and scripting to further improve query performance by passing code to Redis to execute on the server side. Conclusion In conclusion, choosing the AWS-managed NoSQL database for your enterprise reference data platform depends on your specific priorities. If performance and cross-region replication are your primary concern, AWS Cassandra stands out as a clear winner. DynamoDB integrates well with other AWS services such as Lambda and Kinesis and therefore is a great option for AWS native or serverless architecture. For applications requiring robust support for JSON data types, MongoDB takes the lead. However, if your focus is on fast lookup or session management for high availability, Redis proves to be an excellent option. Ultimately, the decision should align with your organization's unique requirements. As always, you can find the code in the GitHub repo linked earlier in this article (see Shell script task #3 above). Feel free to contact me if you need help running this code or with the AWS setup.

By David Shilman

CORE

SQL Data Manipulation Language (DML) Operations: Insert, Update, Delete

Imagine yourself as the master conductor of a symphony orchestra, and the database is your musical score. With SQL's DML operations, you hold the baton that allows you to add, modify, and remove the harmonious notes that compose your database. Yes, you read that right! This may sound like a dream, but SQL (Structured Query Language) turns this dream into reality. SQL, as we all know, is not just any ordinary tool; it’s a game-changing tool that has the ability to transform how we manage and manipulate relational databases. With the power of SQL, you open up a world of endless possibilities for handling and controlling your valuable data. SQL includes Data Manipulation Language (DML), a powerful set of operations like Insert, Update, and Delete, that controls access to data and to the database. It allows you to mold and shape your data in various ways. With just a few lines of code, you can easily and quickly add new data, modify or update existing data, and remove unnecessary data from database tables. Whether you're an experienced professional or a novice, understanding these SQL operations will give you the ability to turn your data to meet your specific requirements. In this article, we will explore various DML operations and understand how they help achieve effective data management. If you looking for more query questions in SQL, check out SQL Query Interview Questions and Answers that help you prepare for Interviews. By the end, you'll have a better understanding of how these SQL operations effectively covert or alter your data in a meaningful way. So, let's dive in together and discover what data manipulation language holds in SQL! SQL Data Manipulation Language (DML) Operations Let’s embark on an adventure through the captivating landscape of SQL Data Manipulation Language (DML) operations: Insert, Update, and Delete. Command Description INSERT Adding new records to a table UPDATE Modifying existing data DELETE Removing records NOTE: DML commands, like DELETE, are not automatically saved in the database, which means changes can be rolled back. To prevent loss of information, we use the COMMIT command to make the changes permanent in the database tables. To make all changes permanent, you can use the COMMIT statement at the end of your DML command. Now, let’s try to understand each of the DML commands stated above in more detail: 1. Insert Operation It allows users to add new data into rows of a database table. Insert operation is important as it allows us to fill tables with essential data. Syntax: SQL INSERT INTO table_name (column1, column2, column3, ...) VALUES (value1, value2, value3, ...); We can also specify the value of data to be inserted without including the column names. SQL INSERT INTO table_name VALUES (value1, value2, .... valuen); Here: Table_name represents the name of the table into which data will be inserted. "column1, column2, column3, …” represent the specific columns to which values will be inserted. "value1, value2, value3, ..." represent actual values that will be inserted into the specified columns. SQL Query Example: Let’s consider a table called "Employees" with columns "EmployeeID," "FirstName," "LastName", “Email”, “PhoneNo”, and “Salary”. To insert a new employee record, we can use the following SQL statement: SQL INSERT INTO Employees (EmployeeID, FirstName, LastName, Email, Country, Salary) VALUES (1, 'Navya', ‘John', 'navyajohn@email.com', 'USA', 50000.00), (2, 'Kunal', 'Swami', 'kunalswami@email.com', 'UK', 44000.00), (3, 'Jai', 'Doneriya', 'jaidoneriya@email.com', 'India', 35000.00); Output: 2. Update Operation It allows users to modify or update existing data within a table. It generally offers the flexibility to update or modify the values of one or more records in the table. Syntax: SQL UPDATE table_name SET column1 = value1, column2 = value2, ... WHERE condition; Here: "table_name" represents the name of the table to be updated. "column1 = value1, column2 = value2, ..." specifies the columns to be updated along with their new values. "WHERE" clause is optional. The update will be applied to all records in the table unless the user specifies a "WHERE" clause. SQL Query Example: Suppose want to update the FirstName of the employee with EmployeeID 1 to 'Smith' and Email with ‘smithjohn@gmail.com’. The UPDATE statement would look like this: SQL UPDATE Employees SET LastName = 'Smith', Email= ‘smithjohn@gmail.com’ WHERE EmployeeID = 1; Output: 3. Delete Operation It allows users to remove/delete records or sets of records from the database tables depending upon the condition specified in the WHERE clause. Syntax: SQL DELETE FROM table_name WHERE condition; Here: "table_name" represents the name of the table from which records should be deleted. "WHERE" clause is used to determine which records should be deleted. Without a "WHERE" clause, all records from the specified table will be removed. SQL Query Example: Suppose we want to delete all employees whose EmployeeID is greater than and equal to 2. The DELETE Command would look like: SQL DELETE FROM Employees WHERE EmployeeID >= 2; Output: Other Operations Discover powerful commands and techniques that will enhance your SQL skills. Let's dive in! 1. Select Statement The Select statement is the most widely used DML command to retrieve data from a table. It allows you to select specific columns you want to retrieve, apply filters, and even sort the results in the desired way. The data returned is stored in a new table called the result set. Syntax: SQL SELECT column1, column2 FROM table_name WHERE condition ORDER BY column ASC/DESC; 2. Join Operation Join operations allow you to combine data from multiple tables based on a common column. It enables you to fetch related data and create meaningful connections between tables. Syntax: SQL SELECT column1, column2 FROM table1 JOIN table2 ON table1.column = table2.column; 3. Aggregate Functions Aggregate functions perform calculations on sets of values and return a single value. They are useful for obtaining summarized information from a table, such as finding the sum, average, maximum, or minimum value. Syntax: SQL SELECT aggregate_function(column) FROM table_name; 4. Subqueries Subqueries are queries embedded within another query. They allow you to use the result of one query as a condition in another query, providing advanced filtering and data manipulation capabilities. Syntax: SQL SELECT column1 FROM table_name WHERE column2 IN (SELECT column3 FROM table2 WHERE condition); 5. Group By The Group By clause is used to group rows based on one or more columns. It is often combined with aggregate functions to perform calculations on each group separately. Syntax: SQL SELECT column1, aggregate_function(column2) FROM table_name GROUP BY column1; 6. Having Clause The Having clause works in conjunction with the Group By clause. It allows you to apply filters to the grouped data based on specific conditions. Syntax: SQL SELECT column1, aggregate_function(column2) FROM table_name GROUP BY column1 HAVING condition; 7. Distinct The Distinct keyword retrieves only distinct (unique) values from a column, as it effortlessly eliminates duplicates from the result set returned by the SQL Query. It ensures that the result set contains only one instance of each value. Syntax: SQL SELECT DISTINCT column FROM table_name; 8. Drop-Table The Drop Table statement is used to delete an entire table from a database. This operation permanently removes the table and all its associated data. Syntax: SQL DROP TABLE table_name; Conclusion All DML operations mentioned above, like Insert, Update, Delete, and more, are essential for managing and manipulating data in SQL databases. Users can add new data, update/modify existing data, and remove unnecessary data from the tables using DML Operations. Therefore, having a deep understanding of the syntax and functionality of DML operations is essential for users to effectively control and maintain data integrity. Remember, becoming proficient in DML operations offers you powerful capabilities for data management, simplifying the handling of data-driven applications and ensuring the reliability of information in relational databases.

By Madhuri Hammad

Python “Magic” Methods: Part 1

Java was the first language I used professionally and is the scale by which I measure other languages I learned afterward. It's an OOP statically-typed language. Hence, Python feels a bit weird because of its dynamic typing approach. For example, Object offers methods equals(), hashCode(), and toString(). Because all other classes inherit from Object, directly or indirectly, all objects have these methods by definition. Conversely, Python was not initially built on OOP principles and is dynamically typed. Yet, any language needs cross-cutting features on unrelated objects. In Python, these are specially-named methods: methods that the runtime interprets in a certain way but that you need to know about. You can call them magic methods. The documentation is pretty exhaustive, but it needs examples for beginners. The goal of this post is to list most of these methods and provide these examples so that I can remember them. I've divided it into two parts to make it more digestible. Lifecycle Methods Methods in this section are related to the lifecycle of new objects. object.__new__(cls[, ...]) The __new()__ method is static, though it doesn't need to be explicitly marked as such. The method must return a new object instance of type cls; then, the runtime will call the __init__() (see below) method on the new instance. __new__() is meant to customize instance creation of subclasses of immutable classes. Python class FooStr(str): #1 def __new__(cls, value): return super().__new__(cls, f'{value}Foo') #2 print(FooStr('Hello')) #3 Inherit from str. Create a new str instance, whose value is the value passed to the constructor, suffixed with Foo. Print HelloFoo. object.__init__(self[, ...]) __init__() is the regular initialization method, which you probably know if you've read any basic Python tutorial. The most significant difference with Java is that the superclass __init__() method has no implicit calling. One can only wonder how many bugs were introduced because somebody forgot to call the superclass method. __init__() differs from a constructor in that the object is already created. Python class Foo: def __init__(self, a, b, c): #1 self.a = a #2 self.b = b #2 self.c = c #2 foo = Foo('one', 'two', 'three') print(f'a={foo.a}, b={foo.b}, c={foo.c}') #3 The first parameter is the instance itself. Initialize the instance. Print a=one, b=two, c=three. object.__del__(self) If __init()__ is akin to an initializer, then __del__() is it's finalizer. As in Java, finalizers are unreliable, e.g., there's no guarantee that the interpreter finalizes instances when it shuts down. Representation Methods Python offers two main ways to represent objects: one "official" for debugging purposes, and the other "informal." You can use the former to reconstruct the object. The official representation is expressed via the object.__repr__(self). The documentation states that the representation must be "information-rich and unambiguous." Python class Foo: def __init__(self, a, b, c): self.a = a self.b = b self.c = c def __repr__(self): return f'Foo(a={foo.a}, b={foo.b}, c={foo.c})' foo = Foo('one', 'two', 'three') print(foo) #1 Print Foo(a=one, b=two, c=three). My implementation returns a string, though it's not required. Yet, you can reconstruct the object with the information displayed. The object.__str__(self) handles the unofficial representation. As its name implies, it must return a string. The default calls __repr__(). Aside from the two methods above, the object.__format__(self, format_spec) method returns a string representation of the object. The second argument follows the rules of the Format Specification Mini-Language. Note that the method must return a string. It's a bit involved so I won't implement it. Finally, the object.__bytes__(self) returns a byte representation of the object. Python from pickle import dumps #1 class Foo: def __init__(self, a, b, c): self.a = a self.b = b self.c = c def __repr__(self): return f'Foo(a={foo.a}, b={foo.b}, c={foo.c})' def __bytes__(self): return dumps(self) #2 foo = Foo('one', 'two', 'three') print(bytes(foo)) #3 Use the pickle serialization library. Delegate to the dumps() method. Print the byte representation of foo. Comparison Methods Let's start with similarities with Java: Python has two methods, object.__eq__(self, other) and object.__hash__(self), that work in the same way. If you define __eq__() for a class, you must define __hash__() as well. Contrary to Java, if you don't define the former, you must not define the latter. Python class Foo: def __init__(self, a, b): self.a = a self.b = b def __eq__(self, other): if not isinstance(other, Foo): #1 return false return self.a == other.a and self.b == other.b #2 def __hash__(self): return hash(self.a + self.b) #3 foo1 = Foo('one', 'two') foo2 = Foo('one', 'two') foo3 = Foo('un', 'deux') print(hash(foo1)) print(hash(foo2)) print(hash(foo3)) print(foo1 == foo2) #4 print(foo2 == foo3) #5 Objects that are not of the same type are not equal by definition. Compare the equality of attributes. The hash consists of the addition of the two attributes. Print True. Print False. As in Java, __eq__()__ and __hash__() have plenty of gotchas. Some of them are the same, others not. I won't paraphrase the documentation; have a look at it. Other comparison methods are pretty self-explanatory: Method Operator object.__lt__(self, other) < object.__le__(self, other) `` object.__ge__(self, other) >= object.__ne__(self, other) != Python class Foo: def __init__(self, a): self.a = a def __ge__(self, other): return self.a >= other.a #1 def __le__(self, other): return self.a <= other.a #1 foo1 = Foo(1) foo1 = Foo(1) foo2 = Foo(2) print(foo1 >= foo1) #2 print(foo1 >= foo2) #3 print(foo1 <= foo1) #4 print(foo2 <= foo2) #5 Compare the single attribute. Print True. Print False. Print True. Print True. Note that comparison methods may return something other than a boolean. In this case, Python will transform the value in a boolean using the bool() function. I advise you not to use this implicit conversion. Attribute Access Methods As seen above, Python allows accessing an object's attributes via the dot notation. If the attribute doesn't exist, Python complains: 'Foo' object has no attribute 'a'. However, it's possible to define synthetic accessors on a class, via the object.__getattr__(self, name) and object.__setattr__(self, name, value) methods. The rule is that they are fallbacks: if the attribute doesn't exist, Python calls the method. Python class Foo: def __init__(self, a): self.a = a def __getattr__(self, attr): if attr == 'a': return 'getattr a' #1 if attr == 'b': return 'getattr b' #2 foo = Foo('a') print(foo.a) #3 print(foo.b) #4 print(foo.c) #5 Return the string if the requested attribute is a. Return the string if the requested attribute is b. Print a. Print getattr b. Print None. For added fun, Python also offers the object.__getattribute__(self, name). The difference is that it's called whether the attribute exists or not, effectively shadowing it. Python class Foo: def __init__(self, a): self.a = a def __getattribute__(self, attr): if attr == 'a': return 'getattr a' #1 if attr == 'b': return 'getattr b' #2 foo = Foo('a') print(foo.a) #3 print(foo.b) #4 print(foo.c) #5 Return the string if the requested attribute is a. Return the string if the requested attribute is b. Print getattr a. Print getattr b. Print None. The dir() function allows returning an object's list of attributes and methods. You can set the list using the object.__dir__(self)__ method. By default, the list is empty: you need to set it explicitly. Note that it's the developer's responsibility to ensure the list contains actual class members. Python class Foo: def __init__(self, a): self.a = 'a' def __dir__(self): #1 return ['a', 'foo'] foo = Foo('one') print(dir(foo)) #2 Implement the method. Display ['a', 'foo']; Python sorts the list. Note that there's no foo member, though. Descriptors Python descriptors are accessors delegates, akin to Kotlin's delegated properties. The idea is to factor a behavior somewhere so other classes can reuse it. In this way, they are the direct consequence of favoring composition over inheritance. They are available for getters, setters, and finalizers, respectively: object.__get__(self, instance, owner=None) object.__set__(self, instance, value) object.__delete__(self, instance) Let's implement a lazy descriptor that caches the result of a compute-intensive operation. Python class Lazy: #1 def __init__(self): self.cache = {} #2 def __get__(self, obj, objtype=None): if obj not in self.cache: self.cache[obj] = obj._intensiveComputation() #3 return self.cache[obj] class Foo: lazy = Lazy() #4 def __init__(self, name): self.name = name self.count = 0 #5 def _intensiveComputation(self): self.count = self.count + 1 #6 print(self.count) #7 return self.name foo1 = Foo('foo1') foo2 = Foo('foo2') print(foo1.lazy) #8 print(foo1.lazy) #8 print(foo2.lazy) #9 print(foo2.lazy) #9 Define the descriptor. Initialize the cache. Call the intensive computation. Conclusion This concludes the first part of Python magic methods. The second part will focus on class, container, and number-related methods.

By Nicolas Fränkel

CORE

Generative AI Apps With Amazon Bedrock: Getting Started for Go Developers

This article is an introductory guide for Go developers who want to get started building Generative AI applications using Amazon Bedrock, which is a fully managed service that makes base models from Amazon and third-party model providers accessible through an API. We will be using the AWS Go SDK for Amazon Bedrock, and we'll cover the following topics as we go along: Amazon Bedrock Go APIs and how to use them for tasks such as content generation How to build a simple chat application and handle streaming output from Amazon Bedrock Foundation Models Code walkthrough of the examples The code examples are available in this GitHub repository. Before You Begin You will need to install a recent version of Go, if you don't have it already. Make sure you have configured and set up Amazon Bedrock, including requesting access to the Foundation Model(s). As we run the examples, we will be using the AWS Go SDK to invoke Amazon Bedrock API operations from our local machine. For this, you need to: Grant programmatic access using an IAM user/role. Grant the below permission(s) to the IAM identity you are using: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "bedrock:*", "Resource": "*" } ] } Note on AWS Go SDK Authentication If you have used the AWS Go SDK before, you will be familiar with this. If not, please note that in the code samples, I have used the following to load the configuration and specify credentials for authentication: cfg, err := config.LoadDefaultConfig(context.Background(), config.WithRegion(region)) When you initialize an aws.Config instance using config.LoadDefaultConfig, the AWS Go SDK uses its default credential chain to find AWS credentials. You can read up on the details here, but in my case, I already have a credentials file in <USER_HOME>/.aws which is detected and picked up by the SDK. Amazon Bedrock Client Types The Amazon Bedrock Go SDK supports two client types: The first one, bedrock.Client can be used for control plane-like operations such as getting information about base foundation models, or custom models, creating a fine-tuning job to customize a base model, etc. The bedrockruntime.Client in the bedrockruntime package is used to run inference on the Foundation models (this is the interesting part!). Listing Amazon Bedrock Foundation Models To start off, let's take a look at a simple example of the control plane client to list foundation models in Amazon Bedrock (error handling, logging omitted): region := os.Getenv("AWS_REGION") if region == "" { region = defaultRegion } cfg, err := config.LoadDefaultConfig(context.Background(), config.WithRegion(region)) bc := bedrock.NewFromConfig(cfg) fms, err := bc.ListFoundationModels(context.Background(), &bedrock.ListFoundationModelsInput{ //ByProvider: aws.String("Amazon"), //ByOutputModality: types.ModelModalityText, }) for _, fm := range fms.ModelSummaries { info := fmt.Sprintf("Name: %s | Provider: %s | Id: %s", *fm.ModelName, *fm.ProviderName, *fm.ModelId) fmt.Println(info) } We create a bedrock.Client instance and use it to get the supported Foundation Models in Amazon Bedrock using ListFoundationModels API. Clone the GitHub repository, and change it to the correct directory: git clone https://github.com/build-on-aws/amazon-bedrock-go-sdk-examples cd amazon-bedrock-go-sdk-examples go mod tidy Run this example: go run bedrock-basic/main.go You should see the list of supported foundation models. Note that you can also filter by provider, modality (input/output), and so on by specifying it in ListFoundationModelsInput. Invoke Model for Inferencing (With bedrockruntime APIs) Let's start by using Anthropic Claude (v2) model. Here is an example of a simple content generation scenario with the following prompt: <paragraph> "In 1758, the Swedish botanist and zoologist Carl Linnaeus published in his Systema Naturae, the two-word naming of species (binomial nomenclature). Canis is the Latin word meaning "dog", and under this genus, he listed the domestic dog, the wolf, and the golden jackal." </paragraph> Please rewrite the above paragraph to make it understandable to a 5th grader. Please output your rewrite in <rewrite></rewrite> tags. To run the program: go run claude-content-generation/main.go The output might differ slightly in your case, but should be somewhat similar to this: <rewrite> Carl Linnaeus was a scientist from Sweden who studied plants and animals. In 1758, he published a book called Systema Naturae where he gave all species two word names. For example, he called dogs Canis familiaris. Canis is the Latin word for dog. Under the name Canis, Linnaeus listed the pet dog, the wolf, and the golden jackal. So he used the first word Canis to group together closely related animals like dogs, wolves and jackals. This way of naming species with two words is called binomial nomenclature and is still used by scientists today. </rewrite> Here is the code snippet (minus error handling etc.). //... brc := bedrockruntime.NewFromConfig(cfg) payload := { Prompt: fmt.Sprintf(claudePromptFormat, prompt), MaxTokensToSample: 2048, Temperature: 0.5, TopK: 250, TopP: 1, } payloadBytes, err := json.Marshal(payload) output, err := brc.InvokeModel(context.Background(), &bedrockruntime.InvokeModelInput{ Body: payloadBytes, ModelId: aws.String(claudeV2ModelID), ContentType: aws.String("application/json"), }) var resp Response err = json.Unmarshal(output.Body, &resp) //..... We get the bedrockruntime.Client instance, and create the payload containing the request we need to send Amazon Bedrock (this includes the prompt as well). The payload is JSON formatted and its details are well documented here: Inference parameters for foundation models. Then we include the payload in the InvokeModel call. Note the ModelId in the call that you can get from the list of Base model IDs. The JSON response is then converted to a Response struct. Note that this "workflow" (preparing payload with prompt, marshaling payload, model invocation, and un-marshaling) will be common across our examples (and most likely in your applications) going forward with slight changes as per the model/use case. You can also try an information extraction scenario using this prompt: <directory> Phone directory: John Latrabe, 800-232-1995, john909709@geemail.com Josie Lana, 800-759-2905, josie@josielananier.com Keven Stevens, 800-980-7000, drkevin22@geemail.com Phone directory will be kept up to date by the HR manager." <directory> Please output the email addresses within the directory, one per line, in the order in which they appear within the text. If there are no email addresses in the text, output "N/A". To run the program: go run claude-information-extraction/main.go Chat: A Canonical GenAI Example We can't have a GenAI article without a chat application, right? Continuing with the Claude model, let's look at a conversational example. While you can exchange one-off messages, this example shows how to exchange multiple messages (chat) and also retain the conversation history. Since it's a simple implementation, the state is maintained in memory. To run the application: go run claude-chat/main.go # If you want to log messages being exchanged with the LLM, # run the program in verbose mode go run claude-chat/main.go --verbose Here is an output of a conversation I had. Notice how the last response is generated based on previous responses, thanks to the chat history retention: Using the Streaming API In the previous chat example, you would have waited for a few seconds to get the entire output. This is because the process is completely synchronous - invoke the model and wait for the complete response. InvokeModelWithResponseStream API allows us to adopt an asynchronous approach - also referred to as Streaming. This is useful if you want to display the response to the user or process the response as it's being generated; this provides a "responsive" experience to the application. To try it out, we use the same prompt as in the content generation example since it generates a response long enough for us to see streaming in action. <rewrite> Carl Linnaeus was a scientist from Sweden who studied plants and animals. In 1758, he published a book called Systema Naturae where he gave all species two word names. For example, he called dogs Canis familiaris. Canis is the Latin word for dog. Under the name Canis, Linnaeus listed the pet dog, the wolf, and the golden jackal. So he used the first word Canis to group together closely related animals like dogs, wolves and jackals. This way of naming species with two words is called binomial nomenclature and is still used by scientists today. </rewrite> Run the application: go run streaming-claude-basic/main.go You should see the output being written to the console as the parts are being generated by Amazon Bedrock. Let's take a look at the code. Here is the first part: business as usual. We create a payload with the prompt (and parameters) and call the InvokeModelWithResponseStream API, which returns a bedrockruntime.InvokeModelWithResponseStreamOutput instance. //... brc := bedrockruntime.NewFromConfig(cfg) payload := Request{ Prompt: fmt.Sprintf(claudePromptFormat, prompt), MaxTokensToSample: 2048, Temperature: 0.5, TopK: 250, TopP: 1, } payloadBytes, err := json.Marshal(payload) output, err := brc.InvokeModelWithResponseStream(context.Background(), &bedrockruntime.InvokeModelWithResponseStreamInput{ Body: payloadBytes, ModelId: aws.String(claudeV2ModelID), ContentType: aws.String("application/json"), }) //.... The next part is different compared to the synchronous approach with InvokeModel API. Since the InvokeModelWithResponseStreamOutput instance does not have the complete response (yet), we cannot (or should not) simply return it to the caller. Instead, we opt to process this output bit by bit with the processStreamingOutput function. The function passed into it is of the type type StreamingOutputHandler func(ctx context.Context, part []byte) error which is a custom type I defined to provide a way for the calling application to specify how to handle the output chunks - in this case, we simply print to the console (standard out). //... _, err = processStreamingOutput(output, func(ctx context.Context, part []byte) error { fmt.Print(string(part)) return nil }) //... Take a look at what the processStreamingOutput function does (some parts of the code omitted for brevity). InvokeModelWithResponseStreamOutput provides us access to a channel of events (of type types.ResponseStream) which contains the event payload. This is nothing but a JSON formatted string with the partially generated response by the LLM; we convert it into a Response struct. We invoke the handler function (it prints the partial response to the console) and make sure we keep building the complete response as well by adding the partial bits. The complete response is finally returned from the function. func processStreamingOutput(output *bedrockruntime.InvokeModelWithResponseStreamOutput, handler StreamingOutputHandler) (Response, error) { var combinedResult string resp := Response{} for event := range output.GetStream().Events() { switch v := event.(type) { case *types.ResponseStreamMemberChunk: var resp Response err := json.NewDecoder(bytes.NewReader(v.Value.Bytes)).Decode(&resp) if err != nil { return resp, err } handler(context.Background(), []byte(resp.Completion)) combinedResult += resp.Completion //.... } resp.Completion = combinedResult return resp, nil } Responsive Chat Application, Thanks To Streaming API Now that you have understood the how and why of handling streaming responses, our simple chat app is the perfect candidate for using this! I will not walk through the code again. I've updated the chat application to use the InvokeModelWithResponseStream API and handle the responses as per the previous example. To run the new version of the app: go run claude-chat-streaming/main.go So far we used the Anthropic Claude v2 model. You can also try the Cohere model example for text generation. To run: go run cohere-text-generation/main.go Image Generation Image generation is another bread-and-butter use case of Generative AI! This example uses the Stable Diffusion XL model in Amazon Bedrock to generate an image given a prompt and other parameters. To try it out: go run stablediffusion-image-gen/main.go "<your prompt>" # for e.g. go run stablediffusion-image-gen/main.go "Sri lanka tea plantation" go run stablediffusion-image-gen/main.go "rocket ship launching from forest with flower garden under a blue sky, masterful, ghibli" You should see an output JPG file generated. Here is a quick walkthrough of the code (minus error handling, etc.). The output payload from the InvokeModel call result is converted to a Response struct which is further deconstructed to extract the base64 image (encoded as []byte) and decoded using encoding/base64 and write the final []byte into an output file (format output-<timestamp>.jpg). //... brc := bedrockruntime.NewFromConfig(cfg) prompt := os.Args[1] payload := Request{ TextPrompts: []TextPrompt{{Text: prompt}, CfgScale: 10, Seed: 0, Steps: 50, } payloadBytes, err := json.Marshal(payload) output, err := brc.InvokeModel(context.Background(), &bedrockruntime.InvokeModelInput{ Body: payloadBytes, ModelId: aws.String(stableDiffusionXLModelID), ContentType: aws.String("application/json"), }) var resp Response err = json.Unmarshal(output.Body, &resp) decoded, err := resp.Artifacts[0].DecodeImage() outputFile := fmt.Sprintf("output-%d.jpg", time.Now().Unix()) err = os.WriteFile(outputFile, decoded, 0644) //... Notice the model parameters (CfgScale, Seed and Steps); their values depend on your use case. For instance, CfgScale determines how much the final image portrays the prompt: use a lower number to increase randomness in the generation. Refer to the Amazon Bedrock Inference Parameters documentation for details. Create Embeddings From Text Text embeddings represent meaningful vector representations of unstructured text such as documents, paragraphs, and sentences. Amazon Bedrock currently supports the Titan Embeddings G1 - Text model for text embeddings. It supports text retrieval, semantic similarity, and clustering. The maximum input text is 8K tokens and the maximum output vector length is 1536. To run the example: go run titan-text-embedding/main.go "<your input>" # for e.g. go run titan-text-embedding/main.go "cat" go run titan-text-embedding/main.go "dog" go run titan-text-embedding/main.go "trex" This is probably the least exciting output you will see! The truth is, it's hard to figure out anything by looking at a slice of float64s. It is more relevant when combined with other components such as a Vector Database (for storing these embeddings) and use cases like semantic search (to make use of these embeddings). These topics will be covered in future blog posts. For now, just bear with the fact that "it works". Closing Thoughts I hope this proves useful for Go developers as a starting point on how to use Foundation models on Amazon Bedrock to power GenAI applications. Watch out for more articles covering Generative AI topics for Go developers. Until then, happy building!

By Abhishek Gupta

CORE

Simple and Easy-To-Use JSON Parser in C

This article introduces the JSON parser of the open-source C library Melon. I believe many readers have heard of or even used the cJSON. It is a very famous open-source project. In this article, we will compare cJSON and Melon’s JSON component. Let’s take a look together below. Encode Suppose we want to build the following JSON: JSON { "name": "Awesome 4K", "resolutions": [ { "width": 1280, "height": 720 }, { "width": 1920, "height": 1080 }, { "width": 3840, "height": 2160 } ] } So, let’s take a look at the cJSON version at first: JSON #include <stdio.h> #include <cjson/cJSON.h> //NOTE: Returns a heap allocated string, you are required to free it after use. char *create_monitor_with_helpers(void) { const unsigned int resolution_numbers[3][2] = { {1280, 720}, {1920, 1080}, {3840, 2160} }; char *string = NULL; cJSON *resolutions = NULL; size_t index = 0; cJSON *monitor = cJSON_CreateObject(); if (cJSON_AddStringToObject(monitor, "name", "Awesome 4K") == NULL) { goto end; } resolutions = cJSON_AddArrayToObject(monitor, "resolutions"); if (resolutions == NULL) { goto end; } for (index = 0; index < (sizeof(resolution_numbers) / (2 * sizeof(int))); ++index) { cJSON *resolution = cJSON_CreateObject(); if (cJSON_AddNumberToObject(resolution, "width", resolution_numbers[index][0]) == NULL) { goto end; } if (cJSON_AddNumberToObject(resolution, "height", resolution_numbers[index][1]) == NULL) { goto end; } cJSON_AddItemToArray(resolutions, resolution); } string = cJSON_Print(monitor); if (string == NULL) { fprintf(stderr, "Failed to print monitor.\n"); } end: cJSON_Delete(monitor); return string; } int main(void) { char *p; p = create_monitor_with_helpers(); printf("%s\n", p); return 0; } Next, let’s take a look at the Melon’s: JSON #include <stdio.h> #include "mln_json.h" #include "mln_log.h" static mln_string_t *generate(void) { mln_json_t j; mln_string_t *ret; mln_json_init(&j); mln_json_generate(&j, "{s:s,s:[{s:d,s:d},{s:d,s:d},{s:d,s:d}]}", \ "name", "Awesome 4K", "resolutions", "width", 1280, "height", 720, \ "width", 1920, "height", 1080, "width", 3840, "height", 2160); ret = mln_json_encode(&j); mln_json_destroy(&j); return ret; } int main(void) { mln_string_t *p; p = generate(); mln_log(none, "%S\n", p); return 0; } The latter code is very short, and you can intuitively see the JSON format that will be generated. Decode We have the following JSON: JSON { "name": "Awesome 4K", "resolutions": [ { "width": 1280, "height": 720 } ] } Let's take a look at the decoding, cJSON first: JSON #include <stdio.h> #include <cjson/cJSON.h> int supports_full_hd(const char * const monitor) { const cJSON *resolution = NULL; const cJSON *resolutions = NULL; cJSON *monitor_json = cJSON_Parse(monitor); if (monitor_json == NULL) return -1; resolutions = cJSON_GetObjectItemCaseSensitive(monitor_json, "resolutions"); cJSON_ArrayForEach(resolution, resolutions) { cJSON *width = cJSON_GetObjectItemCaseSensitive(resolution, "width"); return width->valuedouble; } cJSON_Delete(monitor_json); return -1; } int main(void) { char p[] = "{\"name\":\"Awesome 4K\",\"resolutions\":[{\"width\":1280,\"height\":720}]}"; int i = supports_full_hd(p); printf("%d\n", i); return 0; } Next is the Melon's: JSON #include <stdio.h> #include "mln_json.h" #include "mln_log.h" static int handler(mln_json_t *j, void *data) { return (int)mln_json_number_data_get(j); } static int parse(mln_string_t *p) { mln_json_t j; mln_string_t exp = mln_string("resolutions.0.width"); mln_json_decode(p, &j); return mln_json_parse(&j, &exp, handler, NULL); } int main(void) { mln_string_t p = mln_string("{\"name\":\"Awesome 4K\",\"resolutions\":[{\"width\":1280,\"height\":720}]}"); int i = parse(&p); mln_log(none, "%d\n", i); return 0; } This time, there is not much difference in the number of lines between the two pieces of code. But in the latter implementation, a red-black tree is used as the storage data structure of the object type. Therefore, when facing objects with many keys, the search efficiency will be very stable. Write at the End Melon's JSON component mainly provides the following four functions to facilitate users to encode and decode JSON: mln_json_decode decodes JSON strings into JSON structure nodes mln_json_parse obtains the corresponding JSON sub-node from the decoded JSON structure based on the given expression mln_json_generate builds a JSON structure based on the given format information mln_json_encode generates a JSON string based on the generated JSON structure Based on these four functions, JSON can be easily encoded and decoded, making it easier for developers to use and code maintenance. Thanks for reading.

By Niklaus Schen

How To Scale Your Python Services

Python is becoming a more and more popular choice among developers for a diverse range of applications. However, as with any language, effectively scaling Python services can pose challenges. This article explains concepts that can be leveraged to better scale your applications. By understanding CPU-bound versus I/O-bound tasks, the implications of the Global Interpreter Lock (GIL), and the mechanics behind thread pools and asyncio, we can better scale Python applications. CPU-Bound vs. I/O-Bound: The Basics CPU-Bound Tasks: These tasks involve heavy calculations, data processing, and transformations, demanding significant CPU power. I/O-Bound Tasks: These tasks typically wait on external resources, such as reading from or writing to databases, files, or network operations. Recognizing if your service is primarily CPU-bound or I/O-bound is the foundation for effective scaling. Concurrency vs. Parallelism: A Simple Analogy Imagine multitasking on a computer: Concurrency: You have multiple applications open. Even if only one is active at a moment, you quickly switch between them, giving the illusion of them running simultaneously. Parallelism: Multiple applications genuinely run at the same time, like running calculations on a spreadsheet while downloading a file. In a single-core CPU scenario, concurrency involves rapidly switching tasks, while parallelism allows multiple tasks to execute simultaneously. Global Interpreter Lock: GIL You might think scaling CPU-bound Python services is as simple as adding more CPU power. However, the Global Interpreter Lock (GIL) in Python's standard implementation complicates this. The GIL is a mutex ensuring only one thread executes Python bytecode at a time, even on multi-core machines. This constraint means that CPU-bound tasks in Python can't fully harness the power of multithreading due to the GIL. Scaling Solutions: I/O-Bound and CPU-Bound ThreadPoolExecutor This class provides an interface for asynchronously executing functions using threads. Though threads in Python are ideal for I/O-bound tasks (since they can release the GIL during I/O operations), they are less effective for CPU-bound tasks due to the GIL. Asyncio Suited for I/O-bound tasks, asyncio offers an event-driven framework for asynchronous I/O operations. It employs a single-threaded model, yielding control back to the event loop for other tasks during I/O waits. Compared to threads, asyncio is leaner and avoids overheads like thread context switches. Here's a practical comparison. We take an example of fetching URL data (I/O bound) and do this without threads, with a thread pool, and using asyncio. Python import requests import timeit from concurrent.futures import ThreadPoolExecutor import asyncio URLS = [ "https://www.example.com", "https://www.python.org", "https://www.openai.com", "https://www.github.com" ] * 50 # Function to fetch URL data def fetch_url_data(url): response = requests.get(url) return response.text # 1. Sequential def main_sequential(): return [fetch_url_data(url) for url in URLS] # 2. ThreadPool def main_threadpool(): with ThreadPoolExecutor(max_workers=4) as executor: return list(executor.map(fetch_url_data, URLS)) # 3. Asyncio with Requests async def main_asyncio(): loop = asyncio.get_event_loop() futures = [loop.run_in_executor(None, fetch_url_data, url) for url in URLS] return await asyncio.gather(*futures) def run_all_methods_and_time(): methods = [ ("Sequential", main_sequential), ("ThreadPool", main_threadpool), ("Asyncio", lambda: asyncio.run(main_asyncio())) ] for name, method in methods: start_time = timeit.default_timer() method() elapsed_time = timeit.default_timer() - start_time print(f"{name} execution time: {elapsed_time:.4f} seconds") if __name__ == "__main__": run_all_methods_and_time() Results Sequential execution time: 37.9845 seconds ThreadPool execution time: 13.5944 seconds Asyncio execution time: 3.4348 seconds The results reveal that asyncio is efficient for I/O-bound tasks due to minimized overhead and the absence of data synchronization requirements, as seen with multithreading. For CPU-bound tasks, consider: Multiprocessing: Processes don't share the GIL, making this approach suitable for CPU-bound tasks. However, ensure that the overhead of spawning processes and inter-process communication doesn't diminish the performance benefits. PyPy: An alternative Python interpreter with a Just-In-Time (JIT) compiler. PyPy can deliver performance improvements, especially for CPU-bound tasks. Here, we have an example of regex matching (CPU bound). We implement it using without any optimization and using multiprocessing. Python import re import timeit from multiprocessing import Pool import random import string # Complex regex pattern for non-repeating characters. PATTERN_REGEX = r"(?:(\w)(?!.*\1)){10}" def find_pattern(s): """Search for the pattern in a given string and return it, or None if not found.""" match = re.search(PATTERN_REGEX, s) return match.group(0) if match else None # Generating a dataset of random strings data = [''.join(random.choices(string.ascii_letters + string.digits, k=1000)) for _ in range(1000)] def concurrent_execution(): with Pool(processes=4) as pool: results = pool.map(find_pattern, data) def sequential_execution(): results = [find_pattern(s) for s in data] if __name__ == "__main__": # Timing both methods concurrent_time = timeit.timeit(concurrent_execution, number=10) sequential_time = timeit.timeit(sequential_execution, number=10) print(f"Concurrent execution time (multiprocessing): {concurrent_time:.4f} seconds") print(f"Sequential execution time: {sequential_time:.4f} seconds") Results Concurrent execution time (multiprocessing): 8.4240 seconds Sequential execution time: 12.8772 secondsClearly, multiprocessing is better than sequential execution. The results will be far more evident with a real-world use case. Conclusion Scaling Python services hinges on recognizing the nature of tasks (CPU-bound or I/O-bound) and choosing the appropriate tools and strategies. For I/O bound services, consider using thread pool executors or asyncio, whereas for CPU-bound services, consider leveraging multiprocessing.

By Mandar Khoje

Gossip Protocol in Social Media Networks: Instagram and Beyond

Gossip protocol is a communication scheme used in distributed systems for efficiently disseminating information among nodes. It is inspired by the way people gossip, where information spreads through a series of casual conversations. This article will discuss the gossip protocol in detail, followed by its potential implementation in social media networks, including Instagram. We will also include code snippets to provide a deeper technical understanding. Gossip Protocol The gossip protocol is based on an epidemic algorithm that uses randomized communication to propagate information among nodes in a network. The nodes exchange information about their state and the state of their neighbors. This process is repeated at regular intervals, ensuring that the nodes eventually become aware of each other's states. The key features of gossip protocol include: Fault-tolerance: The protocol can handle node failures effectively, as it does not rely on a central authority or a single point of failure. Scalability: Gossip protocol can efficiently scale to large networks with minimal overhead. Convergence: The system converges to a consistent state quickly, even in the presence of failures or network delays. Gossip Protocol in Social Media Networks: Instagram Social media networks are distributed systems that need to handle massive amounts of data and user interactions. One of the critical aspects of such networks is efficiently propagating updates and notifications to users. Gossip protocol can be employed to achieve this goal by allowing user nodes to exchange information about their state and the state of their connections. For instance, consider Instagram, a social media platform where users can post photos and follow other users. When a user posts a new photo, it needs to be propagated to all their followers. Using the gossip protocol, the photo can be efficiently disseminated across the network, ensuring that all followers receive the update in a timely manner. Technical Implementation of Gossip Protocol in Social Media Networks To illustrate the implementation of gossip protocol in a social media network, let's consider a simplified example using Python. In this example, we will create a basic network of users who can post updates and follow other users, similar to Instagram. First, let's define a User class to represent a user in the network: Python class User: def __init__(self, user_id): self.user_id = user_id self.followers = set() self.posts = [] def post_photo(self, photo): self.posts.append(photo) def follow(self, user): self.followers.add(user) Next, we'll implement the gossip protocol to propagate updates among users. We will create a GossipNetwork class that manages the user nodes and initiates gossip communication: Python import random class GossipNetwork: def __init__(self): self.users = {} def add_user(self, user_id): self.users[user_id] = User(user_id) def post_photo(self, user_id, photo): self.users[user_id].post_photo(photo) self.gossip(user_id, photo) def gossip(self, user_id, photo): user = self.users[user_id] for follower in user.followers: # Propagate the photo to the follower self.users[follower].posts.append(photo) # Continue gossiping with a random subset of the follower's followers if len(self.users[follower].followers) > 0: next_follower = random.choice(list(self.users[follower].followers)) self.gossip(next_follower, photo) The main method to test the behavior: Python if __name__ == "__main__": # Create a gossip network network = GossipNetwork() # Add users to the network for i in range(1, 6): network.add_user(i) # Establish follower relationships network.users[1].follow(2) network.users[2].follow(3) network.users[3].follow(4) network.users[4].follow(5) # Post a photo by user 1 network.post_photo(1, "photo1") # Print the posts of each user for i in range(1, 6): print(f"User {i}: {network.users[i].posts}") This code creates a simple network of five users with a chain of follower relationships (1 -> 2 -> 3 -> 4 -> 5). When user 1 posts a photo, it will be propagated through the gossip protocol to all users in the chain. The output will show that all users have received the posted photo: Plain Text User 1: ['photo1'] User 2: ['photo1'] User 3: ['photo1'] User 4: ['photo1'] User 5: ['photo1'] In this example, when a user posts a photo, the GossipNetwork.post_photo() method is called. This method initiates gossip communication by propagating the photo to the user's followers and their followers using the GossipNetwork.gossip() method. Conclusion The gossip protocol is an efficient and robust method for disseminating information among nodes in a distributed system. Its implementation in social media networks like Instagram can help propagate updates and notifications to users, ensuring timely delivery and fault tolerance. By understanding the inner workings of the gossip protocol in social media networks, developers can better appreciate its role in maintaining a consistent and reliable distributed platform.

By Arun Pandey

Languages

DZone's Featured Languages Resources

Top Languages Experts

The Latest Languages Topics