Tools

Building streaming applications can be a difficult task — often, just figuring out how to get started can be overwhelming. In this article, we'll cover how and when to use NiFi with Pulsar and Flink and how to get started. Apache NiFi, Apache Pulsar, and Apache Flink are robust, powerful open-source platforms that enable running any size application at any scale. This enables us to take our quickly developed prototype and deploy it as an unbreakable clustered application ready for internet-scale production workloads of any size. Using a drag-and-drop tool can help reduce that difficulty and allow us to get the data flowing. By utilizing Apache NiFi, we can quickly go from ideation to data streaming as live events in our Pulsar topics. Once we have a stream, building applications via SQL becomes a much more straightforward premise. The ability to rapidly prototype, iterate, test, and repeat are critical in modern cloud applications. We are now faced with a familiar scenario where it appears like traditional database-driven or batch applications that most data engineers and programmers use. Before we cover how to get started with NiFi, Pulsar, and Flink, let’s discuss how and why these platforms work for real-time streaming. ChatGPT said: Why Use Apache NiFi With Apache Pulsar and Apache Flink? Architects and developers have many options for building real-time scalable streaming applications, so why should they utilize the combination of Apache NiFi, Apache Pulsar, and Apache Flink? The initial reason I started utilizing this combination of open-source projects is the ease of getting started. I always recommend first starting with the simplest way for anyone exploring solutions to new use cases or problems. The simplest solution to start data flowing from a source or extract is usually Apache NiFi. Apache NiFi is a drag-and-drop tool that works on live data, so I can quickly point to my source of data and start pulling or triggering data from it. Since Apache NiFi supports hundreds of sources, often, the data I want to access is a straightforward drag-and-drop. Once the data starts flowing, I can build an interactive streaming pipeline one step at a time in real time with live data flowing. I can examine the state of that data before building the next step. With the combination of inter-step queues and data provenance, I know the current state of the data and all the previous states with their extensive metadata. In an hour or less, I can usually build the ingest, routing, transforming, and essential data enrichment. The final step of the Apache NiFi portion of our streaming application is to stream the data to Apache Pulsar utilizing the NiFi-Pulsar connector. Next in our application development process is to provide routing and additional enhancements before data is consumed from Pulsar. Within Apache Pulsar, we can utilize Functions written in Java, Python, or Go to enrich, transfer, add schemas, and route our data in real time to other topics. When? Quick Criteria Your Data Source Recommended Platform(s) JSON REST feed Looks like a good fit for NiFi+ Relational tables CDC Looks like a good fit for NiFi+ Complex ETL and Transformation Look at Spark + Pulsar Source requires joins Look at Flink applications Large batch data Look at native applications Mainframe data sources Look at existing applications that can send messages Websocket streams of data Looks like a good fit for NiFi+ Clickstream data Looks like a good fit for NiFi+ Sensor data from devices Just stream directly to Pulsar via MQTT. You may ask when I should use the combination of Apache NiFi/Apache Pulsar/Apache Flink to build my apps, and what if I only need Pulsar? That can be the case many times. Suppose you have an existing application that produces messages or events being sent to Apache Kafka, MQTT, RabbitMQ, REST, WebSockets, JMS, RPC, or RocketMQ. In that case, you can just point that program at a Pulsar cluster or rewrite to us the superior native Pulsar libraries. After an initial prototype with NiFi, if it is too slow, you can deploy your flow on a cluster and resize with Kubernetes, expand out vertically with more RAM and CPU cores or look for solutions with my streaming advisors. The great thing about Apache NiFi is that there are many pre-built solutions, demos, examples, and articles for most use cases spanning from REST to CDC to logs to sensor processing. If you hit a wall at any step, then perhaps Apache NiFi is not right for this data. If it is mainframe data, complex ingest rules require joins, many enrichment steps, and complex ETL or ELT. I suggest looking at custom Java code, Apache Spark, Apache Flink, or another tool. If you don’t have an existing application, but your data requires no immediate changes and comes from a known system, perhaps you can use a native Pulsar source. Check them out at https://hub.streamnative.io/. If you need to do some routing, enrichment, and enhancement, you may want to look at Pulsar Functions which can take your raw data in that newly populated topic event at a time to do that. If you have experience with an existing tool such as Spark and have an environment, that may be a good way for you to bring this data into the Pulsar stream. This is especially true if there are a lot of ETL steps or you are combining it with Spark ML. There are several items you should catalog about your data sources, data types, schemas, formats, requirements, and systems before you finalize infrastructure decisions. A series of questions should be answered. These are a few basic questions. Is this pipeline one that requires Exactly Once semantics? What effects would duplicate data have on your pipeline? What are the scale in events per second, gigabytes per second, and total storage and completion requirements? How many infrastructure resources do you have? What is the sacrifice for speed vs. cost? How much total storage per day? How long do you wish to store your data stream? Does this need to be repeatable? Where will this run? Will it need to run in different locations, countries, availability zones, on-premise, cloud, K8, Edge...? What does your data look like? Prepare data types, schemas, and everything you can about the source and final data. Is your data binary, image, video, audio, documents, unstructured, semi-structured, structured, normalized relational data, etc.? What are the upstream and downstream systems? Do NiFi, Pulsar, Flink, Spark, and other systems have native connectors or drivers for your system? Is this data localized, and does it require translation for formatting or language? What type of enrichment is required? Do you require strict auditing, lineage, provenance, and data quality? Who is using this data and how? What team is involved? Data scientists? Data engineers? Data analysts? Programmers? Citizen streaming developers? Is this batch-oriented? How long will this pipeline live? Is this time series data? Is machine learning or deep learning part of the flow or final usage? References https://dev.to/tspannhw/did-the-user-really-ask-for-exactly-once-fault-tolerance-3fek https://www.datainmotion.dev/2021/01/migrating-from-apache-storm-to-apache.html https://thenewstack.io/pulsar-nifi-better-together-for-messaging-streaming/

Monitoring QuestDB in Kubernetes As any experienced infrastructure operator will tell you, monitoring and observability tools are critical for supporting production cloud services. Real-time analytics and logs help to detect anomalies and aid in debugging, ultimately improving the ability of a team to recover from (and even prevent) incidents. Since container technologies are drastically changing the infrastructure world, new tools are constantly emerging to help solve these problems. Kubernetes and its ecosystem have addressed the need for infrastructure monitoring with a variety of newly emerging solutions. Thanks to the orchestration benefits that Kubernetes provides, these tools are easy to install, maintain, and use. Luckily, QuestDB is built with these concerns in mind. From the presence of core database features to the support for orchestration tooling, QuestDB is easy to deploy on containerized infrastructure. This tutorial will describe how to use today's most popular open-source tooling to monitor your QuestDB instance running in a Kubernetes cluster. Components Our goal is to deploy a QuestDB instance on a Kubernetes cluster while also connecting it to centralized metrics and logging systems. We will be installing the following components in our cluster: A QuestDB database server Prometheus to collect and store QuestDB metrics Loki to store logs from QuestDB Promtail to ship logs to Loki Grafana to build dashboards with data from Prometheus and Loki These components work together as illustrated in the diagram below: Prerequisites To follow this tutorial, we will need the following tools. For our Kubernetes cluster, we will be using kind (Kubernetes In Docker) to test the installation and components in an isolated sandbox, although you are free to use any Kubernetes flavor to follow along. docker or podman kind kubectl jq curl Getting Started Once you've installed kind, you can create a Kubernetes cluster with the following command: Shell kind create cluster This will spin up a single-node Kubernetes cluster inside a Docker container and also modify your current kubeconfig context to point kubectl to the cluster's API server. QuestDB QuestDB Endpoint QuestDB exposes an HTTP metrics endpoint that can be scraped by Prometheus. This endpoint, on port 9003, will return a wide variety of QuestDB-specific metrics, including query, memory usage, and performance statistics. A full list of metrics can be found in the QuestDB docs. Helm Installation QuestDB can be installed using Helm. You can add the official Helm repo to your registry by running the following commands: Shell helm repo add questdb https://helm.questdb.io/ helm repo update This is only compatible with the Helm chart version 0.25.0 and higher. To confirm your QuestDB chart version, run the following command: Shell helm search repo questdb Before installing QuestDB, we need to enable the metrics endpoint. To do this, we can override the QuestDB server configuration in a values.yaml file: Shell <<EOF > questdb-values.yaml --- metrics: enabled: true EOF Once you've added the repo, you can install QuestDB in the default namespace: Shell helm install -f questdb-values.yaml questdb questdb/questdb To test the installation, you can make an HTTP request to the metrics endpoint. First, you need to create a Kubernetes port forward from the QuestDB pod to your localhost: export QUESTDB_POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=questdb,app.kubernetes.io/instance=questdb" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $QUESTDB_POD_NAME 9003:9003 Next, make a request to the metrics endpoint: Shell curl http://localhost:9003/metrics You should see a variety of Prometheus metrics in the response: # TYPE questdb_json_queries_total counter questdb_json_queries_total 0 # TYPE questdb_json_queries_completed_total counter questdb_json_queries_completed_total 0 ... Prometheus Now that we've exposed our metrics HTTP endpoint, we can deploy a Prometheus instance to scrape the endpoint and store historical data for querying. Helm Installation Currently, the recommended way of installing Prometheus is using the official Helm chart. You can add the Prometheus chart to your local registry in the same way that we added the QuestDB registry above: Shell helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update As of this writing, we are using the Prometheus chart version 19.0.1 and app version v2.40.5 Configuration Before installing the chart, we need to configure Prometheus to scrape the QuestDB metrics endpoint. To do this, we will need to add our additional scrape configs to a prom-values.yaml file: Shell <<EOF > prom-values.yaml --- extraScrapeConfigs: | - job_name: questdb metrics_path: /metrics scrape_interval: 15s scrape_timeout: 5s static_configs: - targets: - questdb.default.svc.cluster.local:9003 EOF This config will make Prometheus scrape our QuestDB metrics endpoint every 15 seconds. Note that we are using the internal service URL provided to us by Kubernetes, which is only available to resources inside the cluster. We're now ready to install the Prometheus chart. To do so, you can run the following command: Shell helm install -f prom-values.yaml prometheus prometheus-community/prometheus It may take around a minute for the application to become responsive as it sets itself up inside the cluster. To validate that the server is scraping the QuestDB metrics, we can query the Prometheus server for a metric. First, we need to open up another port forward: Shell export PROM_POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $PROM_POD_NAME 9090 Now we can run a query for available metrics after waiting for a minute or so. We are using jq to filter the output to only the QuestDB metrics: Shell curl -s http://localhost:9090/api/v1/label/__name__/values | jq -r '.data[] | select( . | contains("questdb_"))' You should see a list of QuestDB metrics returned: questdb_commits_total questdb_committed_rows_total ... Loki Metrics are only part of the application support story. We still need a way to aggregate and access application logs for better insight into QuestDB's performance and behavior. While kubectl logs is fine for local development and debugging, we will eventually need a production-ready solution that does not require the use of admin tooling. We will use Grafana's Loki, a scalable open-source solution that has tight Kubernetes integration. Helm Installation Like the other components we worked with, we will also be installing Loki using an official Helm chart, loki-stack. The loki-stack helm chart includes Loki, used as the log database, and Promtail, a log shipper that is used to populate the Loki database. First, let's add the chart to our registry: Shell helm repo add grafana https://grafana.github.io/helm-charts helm repo update Loki and Promtail are both enabled out of the box, so all we have to do is install the Helm chart without even supplying our own values.yaml. Shell helm install loki grafana/loki-stack After around a minute or two, the application should be ready to go. To test that Promtail is shipping QuestDB logs to Loki, we first need to generate a few logs on our QuestDB instance. We can do this by curling the QuestDB HTTP frontend to generate a few INFO-level logs. This is exposed on a different port than the metrics endpoint, so we need to open up another port forward first. Shell export QUESTDB_POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=questdb,app.kubernetes.io/instance=questdb" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $QUESTDB_POD_NAME 9000:9000 Now navigate to http://localhost:9000, which should point to the QuestDB HTTP frontend. Your browser should make a request that causes QuestDB to emit a few INFO-level logs. You can query Loki to check if Promtail picked up and shipped those logs. Like the other components, we need to set up a port forward to access the Loki REST API before running the query. Shell export LOKI_POD=$(kubectl get pods --namespace default -l "name=loki,app=loki" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace default port-forward $LOKI_POD 3100:3100 Now, you can run the following LogQL query against the Loki server to return these logs. By default, Loki will look for logs at most an hour old. We will also be using jq to filter the response data. Shell curl -s -G --data-urlencode 'query={pod="questdb-0"}' http://localhost:3100/loki/api/v1/query_range | jq '.data.result[0].values' You should see a list of logs with timestamps that correspond to the logs from the above sample: [ [ "1670359425100049380", "2022-12-13T20:43:45.099494Z I http-server disconnected [ip=127.0.0.1, fd=23, src=queue]" ], [ "1670359425099842047", "2022-12-13T20:43:45.099278Z I http-server scheduling disconnect [fd=23, reason=12]" ], ... Grafana Now that we have all of our observability components set up, we need an easy way to aggregate our metrics and logs into meaningful and actionable dashboards. We will install and configure Grafana inside your cluster to visualize your metrics and logs in one easy-to-use place. Helm Installation The loki-stack chart makes this very easy for us to do. We just need to enable Grafana by customizing the chart's values.yaml and upgrading it. Shell <<EOF > loki-values.yaml --- grafana: enabled: true EOF With this setting enabled, not only are we installing Grafana, but we are also registering Loki as a data source in Grafana to save us the extra work. Now we can upgrade our Loki stack to include Grafana: Shell helm upgrade -f loki-values.yaml loki grafana/loki-stack To get the admin password for Grafana, you can run the following command: Shell kubectl get secret --namespace default loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo And to access the Grafana front-end, you can use a port forward: Shell kubectl port-forward --namespace default service/loki-grafana 3000:80 Configuration First, navigate to http://localhost:3000 in your browser. You can log in using the username admin and the password that you obtained in the previous step. Once you've logged in, use the sidebar to navigate to the "data sources" tab: Here, you can see that the Loki data source is already registered for us: We still need to add our Prometheus data source. Luckily, Grafana makes this easy for us. Click "Add Data Source" in the upper right and select "Prometheus". From here, the only thing you need to do is enter the internal cluster URL of your Prometheus server's Service: http://prometheus-server.default.svc.cluster.local. Scroll down to the bottom, click "Save & test", and wait for the green checkmark popup in the right corner. Now you're ready to create dashboards with QuestDB metrics and logs! Conclusion I have provided a step-by-step tutorial to install and deploy QuestDB with a monitoring infrastructure in a Kubernetes cluster. While there may be additional considerations to make if you want to improve the reliability of the monitoring components, you can get very far with a setup just like this one. Here are a few ideas: Add alerting to a number of targets using Alertmanager Build interactive dashboards that combine metrics and logs using Grafana variables Configure Loki to use alternative deployment modes to improve reliability and scalability Leverage Thanos to incorporate high availability into your Prometheus deployment If you like this content, we'd love to know your thoughts! Feel free to share your feedback or just come and say hello in the QuestDB Community Slack.

In this post, you will learn how to deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster. The following topics have been covered: How to use the franz-go Go Kafka client to connect to MSK Serverless using IAM authentication Write a Go Lambda function to process data in MSK topic. Create the infrastructure: VPC, subnets, MSK cluster, Cloud9 etc. Configure Lambda and Cloud9 to access MSK using IAM roles and fine-grained permissions. MSK Serverless is a cluster type for Amazon MSK that makes it possible for you to run Apache Kafka without having to manage and scale cluster capacity. It automatically provisions and scales capacity while managing the partitions in your topic, so you can stream data without thinking about right-sizing or scaling clusters. Consider using a serverless cluster if your applications need on-demand streaming capacity that scales up and down automatically.- MSK Serverless Developer Guide Prerequisites You will need an AWS account to install AWS CLI, as well as a recent version of Go (1.18 or above). Clone this GitHub repository and change it to the right directory: git clone https://github.com/abhirockzz/lambda-msk-serverless-trigger-golang cd lambda-msk-serverless-trigger-golang Infrastructure Setup AWS CloudFormation is a service that helps you model and set up your AWS resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and CloudFormation takes care of provisioning and configuring those resources for you. You don't need to individually create and configure AWS resources and figure out what's dependent on what; CloudFormation handles that.- AWS CloudFormation User Guide Create VPC and Other Resources Use a CloudFormation template for this. aws cloudformation create-stack --stack-name msk-vpc-stack --template-body file://template.yaml Wait for the stack creation to complete before proceeding to other steps. Create MSK Serverless Cluster Use AWS Console to create the cluster. Configure the VPC and private subnets created in the previous step. Create an AWS Cloud9 Instance Make sure it is in the same VPC as the MSK Serverless cluster and choose the public subnet that you created earlier. Configure MSK Cluster Security Group After the Cloud9 instance is created, edit the MSK cluster security group to allow access from the Cloud9 instance. Configure Cloud9 To Send Data to MSK Serverless Cluster The code that we run from Cloud9 is going to produce data to the MSK Serverless cluster. So we need to ensure that it has the right privileges. For this, we need to create an IAM role and attach the required permissions policy. aws iam create-role --role-name Cloud9MSKRole --assume-role-policy-document file://ec2-trust-policy.json Before creating the policy, update the msk-producer-policy.json file to reflect the required details including MSK cluster ARN etc. aws iam put-role-policy --role-name Cloud9MSKRole --policy-name MSKProducerPolicy --policy-document file://msk-producer-policy.json Attach the IAM role to the Cloud9 EC2 instance: Send Data to MSK Serverless Using Producer Application Log into the Cloud9 instance and run the producer application (it is a Docker image) from a terminal. export MSK_BROKER=<enter the MSK Serverless endpoint> export MSK_TOPIC=test-topic docker run -p 8080:8080 -e MSK_BROKER=$MSK_BROKER -e MSK_TOPIC=$MSK_TOPIC public.ecr.aws/l0r2y6t0/msk-producer-app The application exposes a REST API endpoint using which you can send data to MSK. curl -i -X POST -d 'test event 1' http://localhost:8080 This will create the specified topic (since it was missing, to begin with) and also send the data to MSK. Now that the cluster and producer applications are ready, we can move on to the consumer. Instead of creating a traditional consumer, we will deploy a Lambda function that will be automatically invoked in response to data being sent to the topic in MSK. Configure and Deploy the Lambda Function Create Lambda Execution IAM Role and Attach the Policy A Lambda function's execution role is an AWS Identity and Access Management (IAM) role that grants the function permission to access AWS services and resources. When you invoke your function, Lambda automatically provides your function with temporary credentials by assuming this role. You don't have to call sts:AssumeRole in your function code. aws iam create-role --role-name LambdaMSKRole --assume-role-policy-document file://lambda-trust-policy.json aws iam attach-role-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaMSKExecutionRole --role-name LambdaMSKRole Before creating the policy, update the msk-consumer-policy.json file to reflect the required details including MSK cluster ARN etc. aws iam put-role-policy --role-name LambdaMSKRole --policy-name MSKConsumerPolicy --policy-document file://msk-consumer-policy.json Build and Deploy the Go Function and Create a Zip File Build and zip the function code: GOOS=linux go build -o app zip func.zip app Deploy to Lambda: export LAMBDA_ROLE_ARN=<enter the ARN of the LambdaMSKRole created above e.g. arn:aws:iam::<your AWS account ID>:role/LambdaMSKRole> aws lambda create-function \ --function-name msk-consumer-function \ --runtime go1.x \ --zip-file fileb://func.zip \ --handler app \ --role $LAMBDA_ROLE_ARN Lambda VPC Configuration Make sure you choose the same VPC and private subnets as the MSK cluster. Also, select the same security group ID as MSK (for convenience). If you select a different one, make sure to update the MSK security group to add an inbound rule (for port 9098), just like you did for the Cloud9 instance in an earlier step. Configure the MSK Trigger for the Function When Amazon MSK is used as an event source, Lambda internally polls for new messages from the event source and then synchronously invokes the target Lambda function. Lambda reads the messages in batches and provides these to your function as an event payload. The maximum batch size is configurable (the default is 100 messages). Lambda reads the messages sequentially for each partition. After Lambda processes each batch, it commits the offsets of the messages in that batch. If your function returns an error for any of the messages in a batch, Lambda retries the whole batch of messages until processing succeeds or the messages expire. Lambda sends the batch of messages in the event parameter when it invokes your function. The event payload contains an array of messages. Each array item contains details of the Amazon MSK topic and partition identifier, together with a timestamp and a base64-encoded message. Make sure to choose the right MSK Serverless cluster and enter the correct topic name. Verify the Integration Go back to the Cloud9 terminal and send more data using the producer application. I used a handy JSON utility called jo (sudo yum install jo). APP_URL=http://localhost:8080 for i in {1..5}; do jo email=user${i}@foo.com name=user${i} | curl -i -X POST -d @- $APP_URL; done In the Lambda function logs, you should see the messages that you sent. Conclusion You were able to set up, configure and deploy a Go Lambda function and trigger it in response to events sent to a topic in an MSK Serverless cluster!

AWS Identity and Access Management (IAM) is a service that enables you to manage users and user permissions for your AWS account. With IAM, you can create and manage users, groups, and policies that control access to Amazon EC2 instances, Amazon S3 buckets, and other AWS resources. This article will discuss the basics of AWS IAM: what it is, how it works, and how you can use it to secure your AWS account. What Is IAM Used For? IAM is used to manage the security of user access to AWS resources. It is basically responsible for managing user life cycles, meaning creating accounts, assigning roles, granting access, deleting accounts, enforcing policy, and more. With IAM solutions in place, organizations can enable secure access and authentication of user accounts while minimizing the risk of unauthorized access. You can manage users and groups, assign permissions, and control user access to your AWS resources. For example, you could create a group of users with permission to view Amazon S3 buckets but not modify them or create a user that only has permission to manage EC2 instances. How Does IAM Work? AWS IAM provides access control through the use of policies. Policies are documents that define who has access to what resources and what actions they can take on those resources. For example, you could create a policy that allows only certain users to view S3 buckets or modify EC2 instances. Once you've created your policies, you assign them to users or groups of users. Then, when an AWS user attempts to access a resource, IAM evaluates the user's permissions against the policy assigned to them and either grants or denies access accordingly. AWS IAM Components AWS IAM consists of four core components: users, groups, roles, and policies. Users Users are individual AWS accounts that can be granted access to your AWS resources. You can assign users specific permissions with policies or assign them to groups so they inherit the group's permissions. This means you can give different levels of access to certain services and control what types of actions each user is able to perform. Groups Groups are collections of users that share the same set of permissions. When you assign a policy to a group, all members of the group will receive those same permissions. AWS IAM groups provide a secure and consistent way for teams with varying needs and roles to access cloud resources without needing multiple administrative logins. Policies Policies define what actions a user or service may take on AWS resources. They are written using JSON and contain one or more statements that control who has access, what actions they may take, and which resources they can access. Policies are assigned to users or groups and govern how they interact with AWS resources, such as Amazon S3 buckets and EC2 instances. Below you can find an example of JSON policy syntax from the IAM documentation: JSON { "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::example_bucket" } } Roles Roles are similar to groups. They also have associated policies, but roles are not tied to a particular user or group. They can be used to grant limited access to applications and users, allowing for greater security and control over resources. For example, an IAM Role can be assigned to an IAM user, and this role will determine what part of the AWS environment they have access to, such as EC2 instances or S3 buckets. Each IAM Role also includes a set of permissions rules which further limit what user activities can be performed within that role's scope. Using AWS IAM The AWS IAM console is the main interface for managing users, groups, and policies. From here, you can create new users and groups, assign policies to them, manage existing user permissions, and view access logs. You can also use the AWS CLI or APIs to manage your IAM resources from the command line or programmatically. This allows you to integrate IAM into automated processes, such as setting up EC2 instances or deploying applications. The console provides a graphical user interface for managing IAM components, while the CLI is used for more complex tasks like creating custom policies. Features of the Identity Access Management AWS IAM provides a number of features to help you manage your users and resources. Here are some of the key features: Multi-factor authentication (MFA): MFA can be used to increase security by requiring users to provide additional forms of identification, such as FIDO security keys, TOTP hardware tokens, or time-based one-time passwords generated from a virtual authenticator app. Access control lists (ACLs): ACLs can be used to restrict access to specific resources or actions on those resources. For example, you can create an ACL that only allows certain users to view S3 buckets but not modify them. Identity federation: Identity federation enables users from other systems, such as Active Directory, to log in with their existing credentials. This can be used to simplify user management and reduce the burden of maintaining separate accounts for each system. Identity and access auditing: IAM provides audit logs that track user activities such as login attempts, policy changes, and resource accesses. These logs can be used to monitor user activity and detect potential security issues. AWS IAM is an essential part of any AWS account. It provides a secure way to manage users and resources and control who has access to what resources. With IAM, you can create policies that define user permissions, assign them to users or groups, and use MFA and ACLs for additional security. The audit logging features allow you to monitor user activity and detect potential issues. In addition, AWS IAM is an important tool for ensuring your AWS account remains secure and compliant with industry standards. Conclusion This article has provided an overview of the basics of AWS IAM, what it is, how it works, and how you can use it to secure your AWS account. To learn more about IAM, including creating users and groups, assigning permissions with policies, and managing user access logs, be sure to check out the official Amazon documentation on Identity Access Management.

In this sixth installment of the series covering my journey into the world of cloud-native observability, I'm going to start diving into an open-source project called Perses. If you missed any of the previous articles, head on back to the introduction for a quick update. After laying out the groundwork for this series in the initial article, I spent some time in the second article sharing who the observability players are. I also discussed the teams that these players are on in this world of cloud-native o11y. For the third article, I looked at the ongoing discussion around monitoring pillars versus phases. In the fourth article, I talked about keeping your options open with open-source standards. In my last installment, the fifth article in this series, I talked about bringing monolithic applications into the cloud native o11y world. Being a developer from my early days in IT, it's been very interesting to explore the complexities of cloud-native o11y. Monitoring applications goes way beyond just writing and deploying code, especially in the cloud-native world. One thing remains the same: maintaining your organization's architecture always requires both a vigilant outlook and an understanding of available open standards. In this sixth article, I'm going to provide you with an introduction to an up-and-coming open-source metrics dashboard project I'm getting involved in. Not only will this article provide an introduction to the project, but I'm going to get you started hands-on with a workshop I'm developing to get started with dashboards and visualization. This article is my start at getting practical hands-on experience in the cloud-native o11y world. I've chosen to start with the rather new, up-and-coming open-source project Perses. Not only am I exploring this project, but as I learn I am sharing this knowledge in a free online workshop that you can follow as it's developed here: Now let's explore the origins of this new project. The Origins of Perses Perses is the first project under the CoreDash community umbrella which is part of the Linux Foundation. It's a centralized effort to define a standard for visualization and dashboards. Perses is licensed under the Apache License 2.0, which is a big difference from the recent changes to what used to be the default dashboard project before they opted to change to Affero General Public License (AGPL) v3. This change means users that apply any modification have to share them back into the project, which is a bit more restrictive than most users want. Its main goal is becoming an exploration into finding an open-source standard for visualization and dashboards for metrics monitoring. Its first code commit was made in January 2022, and since then has been quite active. There are clear project goals: Become an open standard dashboard visualization tool. Provide embeddable charts and dashboards in any user interface. Target Kubernetes (k8s) native mode. Provide complete static validation for CI/CD pipelines. Architecture supporting future plugins Time will tell if these goals can be met, but you can check out a more in-depth introduction in this free online workshop lab 1: Next, we can look at installation options for this project Installing Perses There are two options for installing Perses on your local machine. For the first option, you can build the source code project and run it from there, but there are a few software dependencies that you need to meet first to do that. Second, if the bar is too high to build the project from its source, you can install and run Perses from a container image. I've put together a simple supporting project that you can use called Perses Easy Install project. This project contains a simple installation script that allows you to choose either a container install using Podman, or to build the project from its source code. Both methods include sanity checks for the dependencies on your machine before allowing you to install the project. Install in a Container (Podman) This is an installation using the provided Perses container image. You will run this container on a virtual machine provided by Podman. Prerequisites: Podman 4.x+ with your Podman machine started Download and unzip the demo (see project README for links to this download). Run 'init.sh' with the correct argument: $ podman machine init $ ./init.sh podman 3. The Perses container image is now running and pre-loaded with demo examples, connect in your browser: http://localhost:8080 For an installation from the source, the following process is needed. Install on Your Local Machine This is an installation from the source code of the Perses project. You will test, build, and deploy the Perses server locally on your machine. Prerequisites: Go version 1.18+, NodeJS version 16+, npm version 8+ Download and unzip the demo as linked in step 1 above (see project README for links to this download). Run 'init.sh' with the correct argument: $ ./init.sh source 3. Perses is now running, connect to the Perses dashboards in your browser: http://localhost:8080 For step-by-step instructions on how to install Perses using a container image or from the source code in the project itself, see this free online workshop lab 2: More To Come Next up, I plan to continue working through the Perses project with more workshop materials to share. Stay tuned for more insights into a real, practical experience as my cloud native o11y journey continues.

Building Real-Time Weather Dashboards With Apache NiFi, Apache Pulsar, Apache Pinot, and Apache SuperSet It is so easy to build Pulsar to Pinot applications for real-time analytics. I added another source of data for weather feeds for the U.S. I am looking at adding transit and aircraft data feeds next. The sky is the limit with Pulsar + Pinot. I will probably pull more sources from my big list from the Let's Monitor talk. Apache NiFi acquires our weather feed for the United States from NOAA. This is really easy to do, and I have it well-documented at the the source referenced. Reference: https://github.com/tspannhw/SmartWeather Reference: Weather How-To The first thing we will need to do for infrastructure—put this in your DevOps box—is to create topics. We can let the producer of the first message automatically do this if you are on your laptop. Most production clusters want this pre-defined and run by an ops person. So here you go; we will take a look at the list of existing topics and then build it. Since Apache Pulsar is multi-tenant, you can create and specify a custom tenant and namespace for the weather application if you desire. Each organization will decide how to set up and build their hierarchy of tenants and namespaces that matches their application architecture landscape. Weather Pulsar Topic bin/pulsar-admin topics list public/default bin/pulsar-admin topics create persistent://public/default/weather bin/pulsar-admin topics create persistent://public/default/aircraftweather2 As part of the general application, after the data is sent from Apache NiFi to Apache Pulsar via the NiFi Connector, I have a Java Pulsar Function that creates a new schema for it. I developed this schema so it joins well with ADSB Aircraft data. Weather Function in Pulsar to Produce Our Feed Reference: https://github.com/tspannhw/pulsar-weather-function bin/pulsar-admin functions stop --name Weather --namespace default --tenant public bin/pulsar-admin functions delete --name Weather --namespace default --tenant public bin/pulsar-admin functions create --auto-ack true --jar /Users/tspann/Documents/code/pulsar-weather-function/target/weather-1.0.jar --classname "dev.pulsarfunction.weather.WeatherFunction" --dead-letter-topic "persistent://public/default/aircraftweatherdead" --inputs "persistent://public/default/weather" --log-topic "persistent://public/default/aircraftweatherlog" --name Weather --namespace default --tenant public --max-message-retries 5 Once the data is flowing, it is easy to check it and receive the data real-time with the Pulsar client command line consumer as seen below. You can see we are getting data with extra properties and keys. The data matches the schema and is in clean JSON. You can easily verify the schema from the command line if you wish as well. Any commands you run with the Pulsar CLI can also be done via REST, the Pulsar Manager, StreamNative Console, snctl, or Java Pulsar Administration API. Consume Weather Topic from Pulsar bin/pulsar-client consume "persistent://public/default/aircraftweather2" -s test1 -n 0 ----- got message ----- key:[9a88cbf5-92df-4546-bda5-a57dba7e453f], properties:[language=Java], content:{"location":"Greenwood, Greenwood County Airport, SC","station_id":"KGRD","latitude":34.24722,"longitude":-82.15472,"observation_time":"Last Updated on Dec 8 2022, 8:56 am EST","observation_time_rfc822":"Thu, 08 Dec 2022 08:56:00 -0500","weather":"Fog","temperature_string":"61.0 F (16.1 C)","temp_f":61.0,"temp_c":16.1,"relative_humidity":100,"wind_string":"Calm","wind_dir":"North","wind_degrees":0,"wind_mph":0.0,"wind_kt":0,"pressure_string":"1023.5 mb","pressure_mb":1023.5,"pressure_in":30.24,"dewpoint_string":"61.0 F (16.1 C)","dewpoint_f":61.0,"dewpoint_c":16.1,"heat_index_f":0,"heat_index_c":0,"visibility_mi":0.25,"icon_url_base":"https://forecast.weather.gov/images/wtf/small/","two_day_history_url":"https://www.weather.gov/data/obhistory/KGRD.html","icon_url_name":"fg.png","ob_url":"https://www.weather.gov/data/METAR/KGRD.1.txt","uuid":"d429a3e3-d12d-4297-9192-81a2985d8725","ts":1670520773418} Pulsar JSON Schema { "type" : "record", "name" : "Weather", "namespace" : "dev.pulsarfunction.weather", "fields" : [ { "name" : "dewpoint_c", "type" : "double" }, { "name" : "dewpoint_f", "type" : "double" }, { "name" : "dewpoint_string", "type" : [ "null", "string" ], "default" : null }, { "name" : "heat_index_c", "type" : "int" }, { "name" : "heat_index_f", "type" : "int" }, { "name" : "heat_index_string", "type" : [ "null", "string" ], "default" : null }, { "name" : "icon_url_base", "type" : [ "null", "string" ], "default" : null }, { "name" : "icon_url_name", "type" : [ "null", "string" ], "default" : null }, { "name" : "latitude", "type" : "double" }, { "name" : "location", "type" : [ "null", "string" ], "default" : null }, { "name" : "longitude", "type" : "double" }, { "name" : "ob_url", "type" : [ "null", "string" ], "default" : null }, { "name" : "observation_time", "type" : [ "null", "string" ], "default" : null }, { "name" : "observation_time_rfc822", "type" : [ "null", "string" ], "default" : null }, { "name" : "pressure_in", "type" : "double" }, { "name" : "pressure_mb", "type" : "double" }, { "name" : "pressure_string", "type" : [ "null", "string" ], "default" : null }, { "name" : "relative_humidity", "type" : "int" }, { "name" : "station_id", "type" : [ "null", "string" ], "default" : null }, { "name" : "temp_c", "type" : "double" }, { "name" : "temp_f", "type" : "double" }, { "name" : "temperature_string", "type" : [ "null", "string" ], "default" : null }, { "name" : "ts", "type" : "long" }, { "name" : "two_day_history_url", "type" : [ "null", "string" ], "default" : null }, { "name" : "uuid", "type" : [ "null", "string" ], "default" : null }, { "name" : "visibility_mi", "type" : "double" }, { "name" : "weather", "type" : [ "null", "string" ], "default" : null }, { "name" : "wind_degrees", "type" : "int" }, { "name" : "wind_dir", "type" : [ "null", "string" ], "default" : null }, { "name" : "wind_kt", "type" : "int" }, { "name" : "wind_mph", "type" : "double" }, { "name" : "wind_string", "type" : [ "null", "string" ], "default" : null } ] } Build Pinot Schema docker exec -it pinot-controller bin/pinot-admin.sh JsonToPinotSchema \ -timeColumnName ts \ -metrics "pressure_in,temp_c,temp_f,wind_mph,relative_humidity,pressure_mb"\ -dimensions "station_id,location,latitude,longitude" \ -pinotSchemaName=weather \ -jsonFile=/data/weather.json \ -outputDir=/config In order to add a table to Apache Pinot, we need a schema. Instead of hand-building a schema, we can use the JsonToPinotSchema tool to convert an example JSON record into a schema. In Docker, this is easy to execute, as seen above. Now that we have a generated schema, we can quickly build a table JSON file, which is very straightforward. Once complete, we can load the schema via the AddSchema tool. For the easiest way to load the table, I utilize the REST API and curl. Choose the DevOps mechanism that meets your enterprise standards. Load Pinot Schema and Table docker exec -it pinot-controller bin/pinot-admin.sh AddSchema \ -schemaFile /config/weatherschema.json \ -exec curl -X POST "http://localhost:9000/tables" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"tableName\": \"weather\", \"tableType\": \"REALTIME\", \"segmentsConfig\": { \"timeColumnName\": \"ts\", \"schemaName\": \"weather\", \"replication\": \"1\", \"replicasPerPartition\": \"1\" }, \"ingestionConfig\": { \"batchIngestionConfig\": { \"segmentIngestionType\": \"APPEND\", \"segmentIngestionFrequency\": \"DAILY\" } }, \"tableIndexConfig\": { \"loadMode\": \"MMAP\", \"streamConfigs\": { \"streamType\": \"pulsar\", \"stream.pulsar.topic.name\": \"persistent://public/default/aircraftweather2\", \"stream.pulsar.bootstrap.servers\": \"pulsar://Timothys-MBP:6650\", \"stream.pulsar.consumer.type\": \"lowlevel\", \"stream.pulsar.fetch.timeout.millis\": \"10000\", \"stream.pulsar.consumer.prop.auto.offset.reset\": \"largest\", \"stream.pulsar.consumer.factory.class.name\": \"org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory\", \"stream.pulsar.decoder.class.name\": \"org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder\", \"realtime.segment.flush.threshold.rows\": \"0\", \"realtime.segment.flush.threshold.time\": \"1h\", \"realtime.segment.flush.threshold.segment.size\": \"5M\" } }, \"tenants\": {}, \"metadata\": {}" Pinot Table Format JSON { "tableName": "weather", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "weather", "replication": "1", "replicasPerPartition": "1" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" } }, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "pulsar", "stream.pulsar.topic.name": "persistent://public/default/aircraftweather2", "stream.pulsar.bootstrap.servers": "pulsar://SERVERNAME:6650", "stream.pulsar.consumer.type": "lowlevel", "stream.pulsar.fetch.timeout.millis": "10000", "stream.pulsar.consumer.prop.auto.offset.reset": "largest", "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory", "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "1h", "realtime.segment.flush.threshold.segment.size": "5M" } }, "tenants": {}, "metadata": {} } The most important pieces of this connection file is: "tableName" will be the name of your new table. "tableType" will be "REALTIME". "timeColumnName" is the field that has timestamp / UNIX time. "schemaName" is the name of the schema you specified in the schema file. "streamType" which is "pulsar". "stream.pulsar.topic.name" which is your topic. "stream.pulsar.bootstrap.servers" which connects to your Pulsar server Pinot Queries Against Real-Time Table To ensure that our data is working, first run a few test queries in the Query Console within Apache Pinot. select * from weather order by ts desc limit 102; select dewpoint_string,location,latitude,longitude, temperature_string, weather, wind_string, observation_time, ts from weather order by ts desc limit 102; Once our table and schema are loaded, we are connected to Pulsar and will start consuming messages from the listed Pulsar topic. List tables in Pinot Weather table Weather Query Weather Query in the Console To build a dashboard to the dash streamed to our real-time table in Apache Pinot, we will use Apache Superset. Let's connect from Superset to Pinot, it's easy. Pinot to Superset Connection In order to connect Superset to Pinot, you need to add the following URL depending on your server name or Docker IP.pinot+http://192.168.1.157:8099/query?server=192.168.1.157:9000/ pinot+http://SERVERNAME:8099/query?server=http://SERVERNAME:9000/ See: https://docs.pinot.apache.org/integrations/superset Superset Analytics Add a Pinot dataset for weather to Superset Add a dataset Add a chart Weather dashboards in Superset Example Data - data/weather.json {"location":"Union County Airport - Troy Shelton Field, SC","station_id":"K35A","latitude":34.68695,"longitude":-81.64117,"observation_time":"Last Updated on Dec 7 2022, 3:35 pm EST","observation_time_rfc822":"Wed, 07 Dec 2022 15:35:00 -0500","weather":"Overcast","temperature_string":"64.0 F (17.7 C)","temp_f":64.0,"temp_c":17.7,"relative_humidity":98,"wind_string":"Southwest at 4.6 MPH (4 KT)","wind_dir":"Southwest","wind_degrees":220,"wind_mph":4.6,"wind_kt":4,"pressure_mb":0.0,"pressure_in":30.26,"dewpoint_string":"63.1 F (17.3 C)","dewpoint_f":63.1,"dewpoint_c":17.3,"heat_index_f":0,"heat_index_c":0,"visibility_mi":10.0,"icon_url_base":"https://forecast.weather.gov/images/wtf/small/","two_day_history_url":"https://www.weather.gov/data/obhistory/K35A.html","icon_url_name":"ovc.png","ob_url":"https://www.weather.gov/data/METAR/K35A.1.txt","uuid":"5d6ac217-9c3d-4228-87d4-778cbf8561a2","ts":1670508009894} We have built a full live dashboard system, but we may have some other real-time analytics use cases so we can connect to Apache Flink. With Flink, we can run real-time SQL against Pulsar topics as each event arrives in the topic. This lets use run continuous queries, joins, inserts, updates, and advanced SQL applications. We often use this for fraud detection and trigger alerts as things happen. Start Flink in Docker See https://github.com/streamnative/flink-example/blob/main/docs/sql-example.md ./bin/start-cluster.sh ./bin/sql-client.sh Airport Weather Flink SQL Table CREATE CATALOG pulsar WITH ( 'type' = 'pulsar-catalog', 'catalog-service-url' = 'pulsar://Timothys-MBP:6650', 'catalog-admin-url' = 'http://Timothys-MBP:8080' ); USE CATALOG pulsar; show databases; use `public/default`; SHOW TABLES; CREATE TABLE airportweather3 ( `dewpoint_c` DOUBLE, `dewpoint_f` DOUBLE, `dewpoint_string` STRING, `heat_index_c` INT, `heat_index_f` INT, `heat_index_string` STRING, `icon_url_base` STRING, `icon_url_name` STRING, `latitude` DOUBLE, `location` STRING, `longitude` DOUBLE, `ob_url` STRING, `observation_time` STRING, `observation_time_rfc822` STRING, `pressure_in` DOUBLE, `pressure_mb` DOUBLE, `pressure_string` STRING, `relative_humidity` INT, `station_id` STRING, `temp_c` DOUBLE, `temp_f` DOUBLE, `temperature_string` STRING, `ts` DOUBLE, `two_day_history_url` STRING, `visibility_mi` DOUBLE, `weather` STRING, `wind_degrees` INT, `wind_dir` STRING, `wind_kt` INT, `wind_mph` DOUBLE, `wind_string` STRING ) WITH ( 'connector' = 'pulsar', 'topics' = 'persistent://public/default/aircraftweather2', 'format' = 'json', 'admin-url' = 'http://Timothys-MBP:8080', 'service-url' = 'pulsar://Timothys-MBP:6650' ) desc aircraftweather2; +-------------------------+--------+-------+-----+--------+-----------+ | name | type | null | key | extras | watermark | +-------------------------+--------+-------+-----+--------+-----------+ | dewpoint_c | DOUBLE | FALSE | | | | | dewpoint_f | DOUBLE | FALSE | | | | | dewpoint_string | STRING | TRUE | | | | | heat_index_c | INT | FALSE | | | | | heat_index_f | INT | FALSE | | | | | heat_index_string | STRING | TRUE | | | | | icon_url_base | STRING | TRUE | | | | | icon_url_name | STRING | TRUE | | | | | latitude | DOUBLE | FALSE | | | | | location | STRING | TRUE | | | | | longitude | DOUBLE | FALSE | | | | | ob_url | STRING | TRUE | | | | | observation_time | STRING | TRUE | | | | | observation_time_rfc822 | STRING | TRUE | | | | | pressure_in | DOUBLE | FALSE | | | | | pressure_mb | DOUBLE | FALSE | | | | | pressure_string | STRING | TRUE | | | | | relative_humidity | INT | FALSE | | | | | station_id | STRING | TRUE | | | | | temp_c | DOUBLE | FALSE | | | | | temp_f | DOUBLE | FALSE | | | | | temperature_string | STRING | TRUE | | | | | ts | BIGINT | FALSE | | | | | two_day_history_url | STRING | TRUE | | | | | uuid | STRING | TRUE | | | | | visibility_mi | DOUBLE | FALSE | | | | | weather | STRING | TRUE | | | | | wind_degrees | INT | FALSE | | | | | wind_dir | STRING | TRUE | | | | | wind_kt | INT | FALSE | | | | | wind_mph | DOUBLE | FALSE | | | | | wind_string | STRING | TRUE | | | | +-------------------------+--------+-------+-----+--------+-----------+ 32 rows in set Flink SQL Row Flink SQL Results Source Code for our Weather Application: (Pinot-Pulsar Repo) References https://github.com/streamnative/pulsar-flink-patterns https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/

The safety of these applications is crucial to prevent attackers from compromising the computer on which developers are working, as they could use this access to obtain sensitive information, alter source code, and further pivot into the company's internal network. This time, my team and I dive into a new vulnerability I identified in one of the most popular IDEs: Visual Studio Code. It allowed attackers to craft malicious links that, once interacted with, would trick the IDE into executing unintended commands on the victim's computer. By reporting the issue to Microsoft, who quickly patched it, our researchers helped to secure the developer ecosystem. Impact The vulnerability can be used to target developers that have the Visual Studio Code IDE installed. Upon clicking on a malicious link crafted by an attacker, victims are prompted to clone a Git repository in Visual Studio Code. This operation is genuine and part of the workflow of most users. For instance, this is how GitLab allows easier cloning of projects: If the developer accepts this operation, attackers can execute arbitrary commands on the victim's computer. Interestingly, Workspace Trust, a feature to harden the IDEs and reduce the risk of unintended commands being executed, is not strictly enforced here. If the last Visual Studio Code window with focus is trusted by the current workspace, this security feature will not have the expected effect. I disclosed this finding to Microsoft through their Researcher Portal, and the patch was released as part of Visual Studio Code 1.67.1 and higher. Microsoft published limited information about this bug as part of their security bulletin and assigned it CVE-2022-30129. In the sections below, I'll first describe how URL handlers are designed in Visual Studio Code and then review the implementation of the one reserved for Git actions to identify an argument injection bug. Further sections will describe how it could be exploited to gain the ability to execute arbitrary commands, as well as the patch implemented by Microsoft. Visual Studio Code URL Handlers Visual Studio Code is most commonly used as a stand-alone desktop application, thanks to Electron. This choice still allows some level of integration with the user's operating system, for instance, by allowing applications to register custom URL protocol handlers. In the case of Visual Studio Code, vscode:// is registered, and vscode-insiders:// for the nightly builds (also called Insiders build). This feature is named Deep Links. The IDE allows internal and external extensions to listen to such events and handle them by registering sub-handlers. The main listener will handle such OS-level events and redirect them to the right extension. They have to implement a simple interface with a method named handleUri() and announce it to the IDE with window.registerUriHandler(): src/vscode-dts/vscode.d.ts TypeScript export interface UriHandler { handleUri(uri: Uri): ProviderResult<void>; } Finding an Argument Injection in the Git Module With this design in mind, it is now possible to start looking for URL handlers in the core of Visual Studio Code. At that time, three were available: extensions/github-authentication and extensions/microsoft-authentication to authenticate with third-party services and obtain the resulting access tokens, and extensions/git to allow users to clone remote repositories as shown above in GitLab. With my prior experience reviewing the code of developer tools, I know that external invocations of version control tools are often riddled with argument injection bugs—you can head to the Related Posts section for a few examples. Let's look at the extensions/git's implementation of handlerUri first! extensions/git/src/protocolHandler.ts TypeScript export class GitProtocolHandler implements UriHandler { // [...] handleUri(uri: Uri): void { switch (uri.path) { case '/clone': this.clone(uri); } } private clone(uri: Uri): void { const data = querystring.parse(uri.query); // [...] commands.executeCommand('git.clone', data.url); } The git.clone command is implemented in extensions/git/src/commands.ts; it is also possible to invoke it manually: extensions/git/src/commands.ts TypeScript @command('git.clone') async clone(url?: string, parentPath?: string): Promise<void> { await this.cloneRepository(url, parentPath); } Let's continue to dig deeper into the code to identify where the external Git binary is invoked: extensions/git/src/commands.ts TypeScript async cloneRepository(url?: string, parentPath?: string, options: { recursive?: boolean } = {}): Promise<void> { // [...] try { // [...] const repositoryPath = await window.withProgress( opts, (progress, token) => this.git.clone( url!, { parentPath: parentPath!, progress, recursive: options.recursive }, token) ); extensions/git/src/git.ts TypeScript async clone(url: string, options: ICloneOptions, cancellationToken?: CancellationToken): Promise<string> { let baseFolderName = decodeURI(url).replace(/[\/]+$/, '').replace(/^.*[\/\\]/, '').replace(/\.git$/, '') || 'repository'; let folderName = baseFolderName; let folderPath = path.join(options.parentPath, folderName); // [...] try { let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; if (options.recursive) { command.push('--recursive'); } await this.exec(options.parentPath, command, { cancellationToken, env: { 'GIT_HTTP_USER_AGENT': this.userAgent }, onSpawn, }); As promised, there is an argument injection bug in this code: the URL to clone the Git repository is fully controlled and concatenated into the external command line. If this URL starts with dashes, Git will understand it as an option instead of a positional argument. Exploiting an Argument Injection on a Git Clone Operation Argument injection vulnerabilities are very interesting because they are all different and often imply some subtleties; this one is not an exception. This section describes one way to exploit it; other ways exist and are left as an exercise to the reader. Git used to implement git-remote-ext to "bridge smart transport to an external command," but this feature is now disabled by default. As a reminder, we have two injection points: url: the URL of the remote Git repository to clone; folderPath: the destination folder computed from the URL of the Git repository. This is very important in this case as our injected option takes the place of a positional argument: without the second injection point, Git wouldn't have anything to clone from, and the injection wouldn't be exploitable. Finally, if there is any space in the payload, it will be URL-encoded before its interpolation in the command line; it will be easier to try to craft one without spaces: extensions/git/src/git.ts TypeScript let command = ['clone', url.includes(' ') ? encodeURI(url) : url, folderPath, '--progress']; My team and I came up with the following payload: vscode://: the custom scheme registered by Visual Studio Code to the operating system; vscode.git/clone?url=: required to trigger the git.clone command in Visual Studio Code; -u$({open,-a,calculator}): we inject the option -u, equivalent to --upload-pack, to override the command that will be used to communicate with the remote end; :x: this part is required to trick Git into using the transport layer, recognize it as PROTO_LOCAL and invoke the aforementioned upload-pack. Patch Microsoft addressed this issue by improving its validation on the URL of the Git repository to clone as part of the commit c5da533. The URL is parsed using Uri, an internal URI parser, to validate the scheme against a pre-established allow list. The rationale behind this change is that the argument injection bug can only happen if the prefix of the data is fully controlled, which won't be possible if the scheme part of the URL has to be part of this list. diff --- a/extensions/git/src/protocolHandler.ts +++ b/extensions/git/src/protocolHandler.ts @@ -7,6 +7,8 @@ import { UriHandler, Uri, window, Disposable, commands } from 'vscode'; import { dispose } from './util'; import * as querystring from 'querystring'; +const schemes = new Set(['file', 'git', 'http', 'https', 'ssh']); + export class GitProtocolHandler implements UriHandler { private disposables: Disposable[] = []; @@ -26,9 +28,27 @@ export class GitProtocolHandler implements UriHandler { if (!data.url) { console.warn('Failed to open URI:', uri); + return; + } + + if (Array.isArray(data.url) && data.url.length === 0) { + console.warn('Failed to open URI:', uri); + return; + } + + let cloneUri: Uri; + try { + cloneUri = Uri.parse(Array.isArray(data.url) ? data.url[0] : data.url, true); + if (!schemes.has(cloneUri.scheme.toLowerCase())) { + throw new Error('Unsupported scheme.'); + } + } + catch (ex) { + console.warn('Invalid URI:', uri); + return; } - commands.executeCommand('git.clone', data.url); + commands.executeCommand('git.clone', cloneUri.toString(true)); } dispose(): void { This finding was not eligible for a reward from the Microsoft Bug Bounty Program, as only the core is part of the scope; built-in extensions are de facto excluded even if they are enabled by default. This submission still yielded us 40 points for the Microsoft Researcher Recognition Program and got us on the MSRC 2022 Q2 Leaderboard. It is also interesting to note that the Visual Studio Code team started publishing information about security issues on GitHub on top of the usual security bulletin and release notes: the label security is now added to issues, and GitHub security advisories are published. Timeline Date Action 2022-04-05 This issue is reported to Microsoft on their Researcher Portal. 2022-05-06 Microsoft confirms the issue and starts working on a patch. 2022-05-10 The patch is part of the release 1.67.1. Summary In this publication, I demonstrated how a vulnerability in one of the Visual Studio Code URL handlers could lead to the execution of arbitrary commands on the victim's host. The exploitation technique I demonstrated can also be applied to any other argument injection on a git clone invocation. My team and I urge all developers to upgrade their IDE to the latest version and to remain careful when opening foreign links.

A couple of weeks before, in AWS re:invent, Amazon made a lot of innovative announcements, and one of the announcements was the AWS Application Composer service, which allows a user to drag and drop elements to the Canvas and quickly design and deploy serverless applications. Introduction Application Composer service is in the preview phase as this is being written. It allows you to drag and drop a list of resources to a canvas, make connections between them and provide the required configuration. It allows you to design a workflow on the front end, and in the background, it generates the necessary code and template using the Serverless Architecture Model (SAM). SAM CLI is the tool you can use to quickly deploy this template to the AWS environment. Serverless Architecture Model (SAM) is an open-source framework for creating serverless applications in YAML format. You can think of SAM as a shorthand representation of CloudFormation template; SAM syntax allows you to define APIs, databases, functions, and other AWS resources in just a few lines without putting all details. So, when you create an application composer project, basically, you are creating a SAM template. Application Composer allows you to import any existing SAM template or a CloudFormation template. Implementation In this article, to demonstrate the Application Composer service, we are going to create 2 API gateway endpoints; one to get a list of employees and another to create an employee. These endpoints will point to 2 different AWS Lambda’s CreateEmployee and ListEmployees backed by an employee DynamoDb table. To follow along with this video, you need to have your environment setup and have Java11and Gradle installed on your machine. You also need your AWS Serverless Application Model (SAM) CLI installed on your system to build and deploy the generated template. You can download the SAM CLI from here Continue with the following video tutorial for step-by-step instructions for Application Composer Service. Demo Download You can download the source code for this video from GitHub. Conclusion With the introduction of the Application Composer, Amazon is putting a step forward to encourage developers to utilize their infrastructure as a code tools. This product completely relies and builds on top of CloudFormation, and the AWS team is trying to simplify that with SAM Model. Cloud formation also provides a designer for generating the template behind the science, but Application Composer is way more advanced than that and taking cloud formation to next level. Personally, I liked and enjoyed trying out this service; please let me know what you think about the product in the comments.

Modern-day application stores a lot of information and data to understand customer expectations and deliver personalized solutions. All these pieces of information are stored in databases. Now, developers can leverage data from databases using various methods. What confuses them most is the choice of service or method they need to pick in order to manage their database. They can pick either relational or non-relational methods of managing databases. In terms of tools and services, there exists a plethora of services in the market that developers can pick. All these options are enough to make a developer trip over! In this article, I have tried to address whether one should pick Amazon RDS or Dynamo DB. But before we can proceed to that, let’s have a quick look at the difference between relational and non-relational databases. Understanding Relational vs. Non-Relational Databases There are two different approaches to constructing a database — relational and non-relational databases. A relational database leverages SQL, a.k.a structured query language, to interact with the database. Such databases always have some sort of relational and tabular data with underlying rules to maintain consistency and integrity. SQL databases can be scaled vertically by increasing the hardware capability of the server, like CPU, RAM, storage, etc. A non-relational database is a class of database management systems that don’t rely on SQL. Instead, they adapt to dynamic schemas and unstructured data. If your data sets keep on changing frequently, a non-relational database is the best way to deal with it. Unlike SQL databases, non-relational databases can be scaled horizontally. This makes scalability a lot easier. What Is Amazon DynamoDB? Amazon DynamoDB is a fully managed NoSQL database service from AWS. It is designed to automatically scale tables to adjust for capacity and maintain high performance with zero administration. No more provisioning, patching, or managing servers for scalability and availability No more installing, operating, or maintaining database software Built-in fault tolerance, security, backup and restore, automated multi-Region replication, and more. AWS does most of the heavy lifting to support DynamoDB. However, you need to be mindful when modeling data in DynamoDB. What Is Amazon RDS? Amazon RDS(Relational Database Service) is a service offered by AWS to operate and scale SQL-based databases on AWS. It helps with relational database management tasks like — migration, backup, recovery, and patching. Amazon RDS is also responsible for the deployment and maintenance of regional databases. There are six SQL-based database engine options provided by AWS: Amazon Aurora MySQL MariaDB PostgreSQL Oracle Microsoft SQL Server Performance Comparison: DynamoDB vs. Amazon RDS What’s good is learning a new skill if it’s not in demand or performance oriented. DynamoDB is well known in terms of performance, as it can cope with more than 10 trillion requests within a single day. At its peak, it is easily capable of handling 20 million requests per second. Apart from the high-performance SSD, the DynamoDB Accelerator (DAX) also contributes to minimizing latency. You can reduce your reading speeds by using DAX from milliseconds to microseconds! Irrespective of the table size, if you use proper indexing and partitioning, DynamoDB is a high-performing database. Amazon RDS is the high-performance choice for developing OLTP(Online Transactions Processes) applications. It is also considered a cost-effective solution for general-purpose use. Amazon RDS delivers 3 IOPS per provisioned GB for a general-purpose solution that can be scaled up to 3000 IOPS. A performance-oriented RDS instance can deliver up to 40,000 IOPS per database instance. The IOPS rate is maintained throughout the lifetime of the database instance. When To Use Dynamo DB? Amazon DynamoDB is ideal for low-scale operations due to its simplicity. But it also shines at operating on an ultra-high scale, as demanded by Amazon.com. It powers many other powerful applications, including Snapchat, Zoom, Dropbox, and more. You should consider using DynamoDB for: Key-value or simple queries When high-performance reads and writes are required Auto-sharding Low latency applications High durability When no tuning is required When there are no size or throughput limits When you have scalability issues with other traditional database systems Online transaction processing (OLTP) workloads Mission-critical applications that must be highly available all the time without manual intervention When Not To Use DynamoDB Amazon DynamoDB may not be suitable for: Multi-item/row or cross-table transactions Complex queries and joins Real-time Analytics on historical data Services that need ad hoc query access Online analytical processing (OLAP) or data warehouse implementations Binary large object (BLOB) storage. However, blobs can be stored in Amazon S3 and its object pointers in a DynamoDB table. When To Use Amazon RDS? As discussed above, Amazon RDS is based on some sort of relational or structured data. For enterprise applications that rely on relational databases due to the nature of data storage, we need to use Amazon RDS. It is useful for Building traditional applications CRM and e-commerce solutions When you want an automatic scaling solution Good for OLAP (Online analytical processes) applications When Not To Use Amazon RDS? Amazon RDS may not be suitable when: You need to restore an individual database. This is because the process is utterly complex and eliminates the biggest benefit of using a PaaS service. If you are considering patching to the latest edition. This is because although Amazon uses hot patching, it doesn’t always have the current CU available for you. If you care about licensing, consider sticking with Azure services. As AWS doesn’t own the source code, you don’t have the option of running the developer edition for non-production workloads. Million Dollar Question After the in-depth comparison between Amazon RDS and DynamoDB, it all narrows down to the million-dollar question — “which one should you learn?” Both of them have developer-oriented pros and cons. As far as learning is concerned, learning Amazon RDS makes more sense than DynamoDB. This is because, in most cases, relational databases are capable enough of solving the problem. But, if you ask me, don’t make a choice when you don’t have to! Learning both of them will give you a better edge in the future. Being a developer doesn't mean we solve problems with the knowledge/skills that we have, but we pick the best solution to deal with the problem at hand! If you have any queries regarding the same, feel free to reach out to me in the comments below.

Companies leveraging IBM i to run their core systems of record often require integrating these applications with external CRM, eCommerce, and other enterprise applications. Modern integration architectures focus on surfacing the data and business logic via reusable and standards-based APIs or event streams. There are several approaches and tools available to implement APIs for IBM i based applications. One option is to implement and host the APIs directly within IBM i application platform. IBM Integrated Web Services (IWS), Krengeltech’s RPG-XML, Profound API, and several other tools fall into this category and offer products to develop and operate the integration assets within IBM i environment. This option offers several benefits, the most obvious being the simplified architecture and no need for a separate dedicated middleware layer. As with everything in life, it comes with a number of downsides and considerations. As companies evolve and mature their architecture and integration competence, these points become more prominent. Most customers eventually shift most of their integration workloads to dedicated integration platforms such as Mulesoft Anypoint, Confluent Cloud, or AWS or Azure integration stacks that offer high performance, advanced security policies, governance, and process automation out of the box. In this article, we will discuss the main considerations for serving and consuming APIs directly from IBM i. We will use Mulesoft Anypoint as an example of a dedicated integration platform throughout the article. However, the same points would apply to most other integration stacks. Security Transport: HTTP-based protocols typically represent a higher risk of exploitation. For example, IBM published an HTTP server vulnerability as recently as this summer. A quick google of "IBM I" cve will provide quite a list of issues. If attackers can gain access to the infrastructure where the API is running, this means they break directly into critical systems of record. When serving the APIs via MuleSoft, the APIs operate on a different infrastructure and in a different security zone. The interactions between the and IBM i are typically limited to raw socket connections on specific ports only for Database, Data Queue, and Program Call operations. Therefore, even if an attacker gains access to MuleSoft runtimes and raw port traffic, they will have very limited opportunity to gain access to the underlying IBM i systems. Essentially the integration platform represents a “demilitarized zone” that offers more fine-tuned options to limit the surface of potential attack and set up perimeter-, role- and device-based security policies. Authorization and Authentication: At a minimum, any integration tool must always communicate via HTTPS (TLS / encrypted protocol) and authenticate all requests with OAuth2 or a similar token-based provider. In most cases, when setting up dedicated secure connections with regulated external providers, companies may require more complex authorization schemas such as mutual TLS, request signing, and other advanced security features. Most IBM i tools include some of these features, but they are specific to the tool in question and have to be managed separately from companywide authorization and authentication policies. MuleSoft supports external identity management, role-based access, and authorization services such as Okta, as well as advanced features such as mutual TLS, out of the box with little to no custom development. API gateway and policies are intended to stop the attack before it reaches the critical back-end system, preventing impacts on daily operations. Out-of-the-box MuleSoft includes Rate Limiting, JSON/XML threat protection, and several other policies that are critical for secure API operations. Most IBM i based tools do not include external API gateways or policies, meaning the traffic (and potential attack) reaches the server first then the policies are at best evaluated directly on the system, making it more vulnerable to Denial of Service attacks when the system becomes overwhelmed by a high volume of dummy requests. Secrets Management: Any mature integration or API tool must support externalizing and encrypting credentials and other sensitive properties. Mulesoft provides Security Vault and native encryption services to ensure no sensitive data is stored or transmitted in clear text. MuleSoft also can be configured to work with external secret management services, offering an extra layer of protection for securing access credentials for critical endpoint systems. Quality of Service Alerting and Monitoring: MuleSoft can be natively connected with alerting, monitoring, log aggregation services, and analytic platforms such as Splunk. When an issue happens in the API implementation, an alert can be configured in Splunk to notify relevant IT and business teams via a variety of channels (email, SMS, Teams, Webex, Slack) to address the issue. On the other hand, when the API is hosted directly on IBM i servers, they lack native configuration bases integrations with logging, monitoring, or alerting platforms like Splunk. This means companies must develop it manually or miss the important alerts, notifications, and insights. Retries and Time-outs: Most integration components communicate with external services over the network. It is natural for distributed systems to have occasional breaks in connection or for the target services not to be available. However, most flexible integration designs account for API consumers or providers occasionally being unavailable and include the ability to auto-recover per retry policy and back off models, as well as save the unprocessed message into Dead Letter Queue or similar hold area for future review and recovery process. The MuleSoft platform supports these resilient integration patterns. Caching: For slow-changing data, the API or integration flow can be configured to cache the response and return it right away based on previously obtained results. DevOps, CI/CD, and Test Automation Most of the IBM i tools favor visual (“low code / no code”) API development, which is great and offers non-integration developers and analysts a quick option to launch the APIs. The downside is that the process is manual, relies on specific screen actions, and does not offer the same level of source code management and automation practices as most other modern development platforms. MuleSoft provides native support for Git repositories (GitHub, bitbucket, GitLab, Azure Develops, etc.). The de-facto standard for agile teams, multiple developers can work on several features for the same API in parallel. MuleSoft applications can be automatically built using a standard build manager (Maven plugin) that will automatically execute all unit tests, verify the dependencies, and assemble the deployable artifacts. Many DevOps providers offer to build and deployment pipelines that can be easily integrated with MuleSoft and automatically triggered based on the repository events to have a fully automated flow from feature code push to deployment and smoke tests. This approach greatly reduces the risks of unintended or buggy code making it to production and increases the development team's agility and throughput. Test coverage is an important metric for mature integration teams. MuleSoft munit framework provides a low code API test automation tool where tests can be developed addressing both internal MuleSoft API functionality and optional end-to-end functional tests. IBM i System Impact When the integration work is offloaded to a scaled-out platform such as MuleSoft, it protects the IBM i system from performance degradation and provides more options for tuning the API throughput. When the API/integration tool is running directly on the IBM i it typically consumes more system resources. Furthermore, Mulesoft workloads run in isolated containers, meaning one API performance or usage has minimal or no impact on another API or the endpoint system. If running multiple APIs and integration assets on the IBM i system, if one component starts consuming significant amounts of system resources, this can impact the performance of other integration components or the underlying system. East of Use, Training, and Standards MuleSoft is one of the leading integrations platforms with plenty of specialists and resources available for companies standardizing on this stack, but there is a wide variety of other middleware options such as Kafka, AWS, and Azure. Whereas running on APIs hosted directly on IBM i servers might require specialized training to develop and operate the APIs and increases operational risk in case a key developer familiar with the specialized tool leaves the team. Conclusions In this article, we reviewed several key considerations for any mature integration practice, focusing on security, quality of service, and performance impact. Introducing a dedicated integration platform such as MuleSoft may appear to be an overkill for some organizations, and surfacing the business data and logic as APIs directly from IBM i seems to be a simpler and easier approach. I hope this article helps evaluate these points and come up with a more mature target state integration architecture.