Data Engineering Resources

Delta, Hudi, and Iceberg: The Data Lakehouse Trifecta

Get a detailed overview of Delta Lake, Apache Hudi, and Apache Iceberg as we discuss their data storage, processing capabilities, and deployment options.

April 26, 2023

by Andrey Gusarov

· 11,653 Views · 4 Likes

Processing Large Files With Repeating Elements and Header/Trailer Fields

How to use IBM App Connect Enterprise File Nodes and DFDL Parser to process very large files with repeating elements and header/trailer fields.

April 26, 2023

by Dave Crighton

· 3,040 Views · 1 Like

The Equivalence Rationale of Neural Networks and Decision Trees: Towards Improving the Explainability and Transparency of Neural Networks

Let's unravel a new approach to improving the explainability and transparency of neural networks using an equivalent decision tree to represent them.

April 25, 2023

by Priya Kumari

· 2,890 Views · 2 Likes

Streamlining Master Data Management With Snowflake and SnapLogic

Building a Master Data Management (MDM) solution simplifies data integration, improves data quality and governance, and enhances master data management and analytics.

April 25, 2023

by Rama Krishna Panguluri

· 4,122 Views · 3 Likes

Azure Data Box

Azure Data Box is a product offered by Microsoft Azure that helps organizations transfer large amounts of data securely and efficiently to and from Azure.

April 25, 2023

by Shripad Barve

· 4,018 Views · 2 Likes

Connection Pool High Availability With CockroachDB and PgCat

This article covers pooler high availability working with PgCat to provide a highly available configuration where clients can see multiple connections.

April 25, 2023

by Artem Ervits

CORE

· 3,761 Views · 1 Like

Frequently Faced Challenges in Implementing Spark Code in Data Engineering Pipelines

This article discusses some of the common challenges faced by data engineers in Pyspark applications and the possible solutions to overcome these challenges.

April 25, 2023

by Amlan Patnaik

· 7,063 Views · 2 Likes

Building a RESTful API With Java Spring Boot

Build powerful RESTful APIs using Java Spring Boot. Use our quick and easy tutorial for retrieving credit scores and setting up a basic REST API using Postman.

April 25, 2023

by Dylan Frankcom

· 5,325 Views · 1 Like

A Critical Detail About Kafka Partitioners

Understanding the role of a partitioner in the Kafka producer is important to understand to build effective Kafka applications.

April 25, 2023

by Bill Bejeck

· 4,638 Views · 2 Likes

Potential Benefits of Using Generative AI in the Development and Operations of IT Systems

Many organizations are experimenting with how generative AI can be used to develop and operate IT systems.

April 25, 2023

by Divakar R

· 5,687 Views · 1 Like

How To Use Cypress Intercept for Handling Network Requests

Deep diving into using Cypress intercept for handling network requests, let’s first understand the nuances of network testing while performing Cypress testing.

April 25, 2023

by Kailash Pathak

CORE

· 3,555 Views · 2 Likes

How To Install CMAK, Apache Kafka, Java 18, and Java 19 [Video Tutorials]

This article has video tutorials on installing CMAK, Apache, Kafka, Java 18, Java 19, and JDK on Windows, Amazon EC2, and Linux.

April 25, 2023

by Ram N

· 10,562 Views · 2 Likes

AWS ECS vs. AWS Lambda: Top 5 Main Differences

AWS ECS and AWS Lambda are suited to specific purposes, making selecting the right one an essential part of Cloud application deployment.

April 24, 2023

by Rodolfo Ortega

· 5,865 Views · 3 Likes

How To Place Text and Shapes on an Image in Java

This article provides two API solutions that can be used to programmatically add text and shapes to an image with ready-to-run Java code examples.

April 24, 2023

by Brian O'Neill

CORE

· 5,368 Views · 2 Likes

Integration Architecture Guiding Principles, A Reference

The Integration Architecture guiding principles are guidelines to increase the consistency and quality of technology decision-making for the integration solutions.

April 24, 2023

by Susmit Dey

CORE

· 27,375 Views · 19 Likes

Get an In-Depth Look at the Best Ways To Store Application Parameters in AWS

Learn the top techniques for storing application parameters in AWS. Check out this comprehensive guide and optimize your cloud infrastructure today.

April 24, 2023

by Satrajit Basu

CORE

· 5,520 Views · 1 Like

Best Practices and Hacks: Data Loading in Snowflake From Stages

This second article of the series will help you to understand the best practices for file-based data loading in Snowflake.

April 24, 2023

by Kedhar Natekar

· 4,631 Views · 3 Likes

API Gateway Pattern: Features and the AWS Implementation

This article will provide a deep-dive study of API Gateways, starting with the problems that are addressed by API Gateways in enterprise computing

April 24, 2023

by Rodrigo Dinis

· 4,245 Views · 1 Like

10 Best Practices for Using Kubernetes Network Policies

In this article, we will explore ten best practices for using Kubernetes Network Policies to enhance the security and reliability of your applications.

April 24, 2023

by Emmanuel Moila

· 8,198 Views · 3 Likes

Make E2e Testing Easier With the Right Tools

E2e testing is notably slower and more expensive than lower-level tests. In this article, we will cover tools and approaches to make e2e testing easier.

April 24, 2023

by Mirza Sisic

· 3,097 Views · 1 Like

The Latest Data Engineering Topics