DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Curious about the future of data-driven systems? Join our Data Engineering roundtable and learn how to build scalable data platforms.

Data Engineering: The industry has come a long way from organizing unstructured data to adopting today's modern data pipelines. See how.

Threat Detection: Learn core practices for managing security risks and vulnerabilities in your organization — don't regret those threats!

Managing API integrations: Assess your use case and needs — plus learn patterns for the design, build, and maintenance of your integrations.

Avatar

Amlan Patnaik

Director of Software Development at Top IT Consulting Firm

San Francisco, US

Joined Feb 2023

About

Amlan J Patnaik stands as a distinguished senior engineering leader, currently holding the esteemed position of technology executive with one of the top IT consulting firm in the world. With a remarkable track record spanning over 17 years, Amlan shines as a beacon of expertise in spearheading and executing large-scale strategic technology initiatives in the areas of software and data engineering, artificial intelligence, machine learning, web application development and DevOps engineering.

Stats

Reputation: 688
Pageviews: 114.1K
Articles: 23
Comments: 0
  • Articles

Articles

article thumbnail
Pillars of a Strong and Resilient Data Architecture
This article explores advanced data architecture practices for embracing a more holistic viewpoint of data architecture.
May 29, 2024
· 1,129 Views · 1 Like
article thumbnail
Azure Stream Analytics Upsert Operation Using Cosmos DB
Upsert operations allow you to update existing records or insert new ones based on specific criteria within your streaming pipelines.
March 15, 2024
· 4,146 Views · 1 Like
article thumbnail
Blueprint for Migrating On-Premise Data Pipelines To Azure Cloud
This article aims to provide a detailed and comprehensive outline for migrating an on-premise data pipeline built on Cloudera to Azure.
July 13, 2023
· 5,198 Views · 2 Likes
article thumbnail
Medallion Architecture: Efficient Batch and Stream Processing Data Pipelines With Azure Databricks and Delta Lake
In this article, learn about Medallion Architecture and how to create processing pipelines with Azure Databricks and Delta Lake.
July 13, 2023
· 7,324 Views · 3 Likes
article thumbnail
Architecting and Building LLM-Powered Generative AI Applications
This research article explores the process of architecting and building an LLM (language model-based learning) powered application.
July 6, 2023
· 8,964 Views · 3 Likes
article thumbnail
Leveraging Infrastructure as Code for Data Engineering Projects: A Comprehensive Guide
Infrastructure as Code (IaC) revolutionizes data engineering projects by automating infrastructure provisioning, deployment, and management through code.
July 3, 2023
· 3,792 Views · 3 Likes
article thumbnail
Simplify Data Processing With Azure Data Factory REST API and HDInsight Spark
In this blog post, we will explore how to leverage Azure Data Factory and HDInsight Spark to create a robust data processing pipeline.
June 19, 2023
· 3,318 Views · 2 Likes
article thumbnail
Metadata and Config-Driven Python Framework for Big Data Processing Using Spark
Introducing the Metadata and Config-Driven Python Framework for Data Processing with Spark that offers a streamlined and flexible approach to processing big data.
June 16, 2023
· 6,499 Views · 4 Likes
article thumbnail
Leveraging FastAPI for Building Secure and High-Performance Banking APIs
Explore the importance of FastAPI for developing banking APIs and how it can empower financial institutions to deliver efficient and secure services to their customers.
June 5, 2023
· 3,317 Views · 3 Likes
article thumbnail
Developing Metadata-Driven Data Engineering Pipelines Using Apache Spark and Python Dictionary
This article will discuss metadata-driven programming techniques to make our PySpark code more flexible, maintainable, and reusable.
May 10, 2023
· 4,419 Views · 4 Likes
article thumbnail
Python Dictionary: A Powerful Tool for Data Engineering
This article discusses the Python dictionary use cases in data engineering as a powerful tool and data structure to perform tasks efficiently and accurately.
May 9, 2023
· 3,404 Views · 2 Likes
article thumbnail
Frequently Faced Challenges in Implementing Spark Code in Data Engineering Pipelines
This article discusses some of the common challenges faced by data engineers in Pyspark applications and the possible solutions to overcome these challenges.
April 25, 2023
· 5,096 Views · 2 Likes
article thumbnail
Lambda Architecture: A Powerful Approach to Data Engineering
In this article, we will explore the Lambda Architecture in detail, understanding its key concepts, benefits, and challenges.
April 21, 2023
· 4,744 Views · 6 Likes
article thumbnail
Demystifying Data Fabric Architecture: A Comprehensive Overview
This article will provide a comprehensive overview of the data fabric architecture, its key components, and how it works.
April 18, 2023
· 5,345 Views · 2 Likes
article thumbnail
Python Stack Data Structure: A Versatile Tool for Real-time Applications
In this article, we will explore the Python stack data structure, its implementation, and real-time use cases.
April 17, 2023
· 3,027 Views · 2 Likes
article thumbnail
Building an Optimized Data Pipeline on Azure Using Spark, Data Factory, Databricks, and Synapse Analytics
This article will explore how Apache Spark, Azure Data Factory, Databricks, and Synapse Analytics can be used together to create an optimized data pipeline in the cloud.
April 11, 2023
· 5,458 Views · 3 Likes
article thumbnail
Data Mesh Architecture: A Paradigm Shift in Data Engineering
This article will discuss a new architectural paradigm called Data Mesh which enables more efficient and effective data engineering.
April 7, 2023
· 9,309 Views · 2 Likes
article thumbnail
The Kappa Architecture: A Cutting-Edge Approach for Data Engineering
In this article, we will explore the Kappa Architecture and its key features that make it a cutting-edge approach for data engineering.
April 6, 2023
· 5,862 Views · 2 Likes
article thumbnail
PySpark Data Pipeline To Cleanse, Transform, Partition, and Load Data Into Redshift Database Table
In this article, we will discuss how to create an optimized data pipeline using PySpark and load the data into a Redshift database table.
March 16, 2023
· 5,625 Views · 2 Likes
article thumbnail
Factors for Determining Optimized File Format for Spark Applications
This article will discuss how to select the optimized file format for Apache Spark applications.
March 14, 2023
· 2,281 Views · 2 Likes
article thumbnail
Important Data Structures and Algorithms for Data Engineers
Explore important data structures and algorithms that data engineers should know, including their uses and advantages.
March 8, 2023
· 9,772 Views · 3 Likes
article thumbnail
Write Optimized Spark Code for Big Data Applications
While PySpark makes writing applications easy, tuning them is often challenging. In this article, we will explore some tips for tuning PySpark applications.
March 7, 2023
· 2,855 Views · 2 Likes
article thumbnail
Leveraging Data Locality to Optimize Spark Applications
Efficient ways to optimize a PySpark application using the concept of data locality.
March 5, 2023
· 3,239 Views · 2 Likes

User has been successfully modified

Failed to modify user

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: