Amlan Patnaik

Director of Software Development at Top IT Consulting Firm

San Francisco, US

Joined Feb 2023

About

Amlan J Patnaik stands as a distinguished senior engineering leader, currently holding the esteemed position of technology executive with one of the top IT consulting firm in the world. With a remarkable track record spanning over 17 years, Amlan shines as a beacon of expertise in spearheading and executing large-scale strategic technology initiatives in the areas of software and data engineering, artificial intelligence, machine learning, web application development and DevOps engineering.

Stats

Reputation:	688
Pageviews:	114.1K
Articles:	23
Comments:	0

Articles

Articles

Pillars of a Strong and Resilient Data Architecture

This article explores advanced data architecture practices for embracing a more holistic viewpoint of data architecture.

May 29, 2024

· 1,129 Views · 1 Like

Azure Stream Analytics Upsert Operation Using Cosmos DB

Upsert operations allow you to update existing records or insert new ones based on specific criteria within your streaming pipelines.

March 15, 2024

· 4,146 Views · 1 Like

Blueprint for Migrating On-Premise Data Pipelines To Azure Cloud

This article aims to provide a detailed and comprehensive outline for migrating an on-premise data pipeline built on Cloudera to Azure.

July 13, 2023

· 5,198 Views · 2 Likes

Medallion Architecture: Efficient Batch and Stream Processing Data Pipelines With Azure Databricks and Delta Lake

In this article, learn about Medallion Architecture and how to create processing pipelines with Azure Databricks and Delta Lake.

July 13, 2023

· 7,324 Views · 3 Likes

Architecting and Building LLM-Powered Generative AI Applications

This research article explores the process of architecting and building an LLM (language model-based learning) powered application.

July 6, 2023

· 8,964 Views · 3 Likes

Leveraging Infrastructure as Code for Data Engineering Projects: A Comprehensive Guide

Infrastructure as Code (IaC) revolutionizes data engineering projects by automating infrastructure provisioning, deployment, and management through code.

July 3, 2023

· 3,792 Views · 3 Likes

Simplify Data Processing With Azure Data Factory REST API and HDInsight Spark

In this blog post, we will explore how to leverage Azure Data Factory and HDInsight Spark to create a robust data processing pipeline.

June 19, 2023

· 3,318 Views · 2 Likes

Metadata and Config-Driven Python Framework for Big Data Processing Using Spark

Introducing the Metadata and Config-Driven Python Framework for Data Processing with Spark that offers a streamlined and flexible approach to processing big data.

June 16, 2023

· 6,499 Views · 4 Likes

Leveraging FastAPI for Building Secure and High-Performance Banking APIs

Explore the importance of FastAPI for developing banking APIs and how it can empower financial institutions to deliver efficient and secure services to their customers.

June 5, 2023

· 3,317 Views · 3 Likes

Developing Metadata-Driven Data Engineering Pipelines Using Apache Spark and Python Dictionary

This article will discuss metadata-driven programming techniques to make our PySpark code more flexible, maintainable, and reusable.

May 10, 2023

· 4,419 Views · 4 Likes

Python Dictionary: A Powerful Tool for Data Engineering

This article discusses the Python dictionary use cases in data engineering as a powerful tool and data structure to perform tasks efficiently and accurately.

May 9, 2023

· 3,404 Views · 2 Likes

Frequently Faced Challenges in Implementing Spark Code in Data Engineering Pipelines

This article discusses some of the common challenges faced by data engineers in Pyspark applications and the possible solutions to overcome these challenges.

April 25, 2023

· 5,096 Views · 2 Likes

Lambda Architecture: A Powerful Approach to Data Engineering

In this article, we will explore the Lambda Architecture in detail, understanding its key concepts, benefits, and challenges.

April 21, 2023

· 4,744 Views · 6 Likes

Demystifying Data Fabric Architecture: A Comprehensive Overview

This article will provide a comprehensive overview of the data fabric architecture, its key components, and how it works.

April 18, 2023

· 5,345 Views · 2 Likes

Python Stack Data Structure: A Versatile Tool for Real-time Applications

In this article, we will explore the Python stack data structure, its implementation, and real-time use cases.

April 17, 2023

· 3,027 Views · 2 Likes

Building an Optimized Data Pipeline on Azure Using Spark, Data Factory, Databricks, and Synapse Analytics

This article will explore how Apache Spark, Azure Data Factory, Databricks, and Synapse Analytics can be used together to create an optimized data pipeline in the cloud.

April 11, 2023

· 5,458 Views · 3 Likes

Data Mesh Architecture: A Paradigm Shift in Data Engineering

This article will discuss a new architectural paradigm called Data Mesh which enables more efficient and effective data engineering.

April 7, 2023

· 9,309 Views · 2 Likes

The Kappa Architecture: A Cutting-Edge Approach for Data Engineering

In this article, we will explore the Kappa Architecture and its key features that make it a cutting-edge approach for data engineering.

April 6, 2023

· 5,862 Views · 2 Likes

PySpark Data Pipeline To Cleanse, Transform, Partition, and Load Data Into Redshift Database Table

In this article, we will discuss how to create an optimized data pipeline using PySpark and load the data into a Redshift database table.

March 16, 2023

· 5,625 Views · 2 Likes

Factors for Determining Optimized File Format for Spark Applications

This article will discuss how to select the optimized file format for Apache Spark applications.

March 14, 2023

· 2,281 Views · 2 Likes

Important Data Structures and Algorithms for Data Engineers

Explore important data structures and algorithms that data engineers should know, including their uses and advantages.

March 8, 2023

· 9,772 Views · 3 Likes

Write Optimized Spark Code for Big Data Applications

While PySpark makes writing applications easy, tuning them is often challenging. In this article, we will explore some tips for tuning PySpark applications.

March 7, 2023

· 2,855 Views · 2 Likes

Leveraging Data Locality to Optimize Spark Applications

Efficient ways to optimize a PySpark application using the concept of data locality.

March 5, 2023

· 3,239 Views · 2 Likes