Data Engineering Resources

Many things can break, but when the logging breaks, then we are completely lost. In this article, understand how to log efficiently in distributed systems.

September 9, 2024

by Adam Furmanek

CORE

· 4,548 Views · 3 Likes

Data Storage Formats for Big Data Analytics: Performance and Cost Implications of Parquet, Avro, and ORC

This article compares the performance and cost efficiency of three storage formats Parquet, Avro, and ORC on Google Cloud Platform.

September 9, 2024

by Rahul Sarabu

· 8,578 Views · 4 Likes

Mainframe Modernization: A Comprehensive Technical Blueprint

This post explores a practical hybrid approach to mainframe workload migration to the cloud using both IBM Cloud and AWS examples.

September 6, 2024

by Manjit Chakraborty

· 6,931 Views · 1 Like

Comparison of Apache Astro and Airflow

Compare Astro and Apache Airflow, explaining their architecture, features, scalability, usability, community support, and integration capabilities.

September 6, 2024

by Krishnamurty Raju Mudunuru

· 11,568 Views · 2 Likes

Principles of Modern Data Infrastructure

Explore principles of modern data infrastructure such as scalability, high availability, speed, security, maintainability, efficiency, and developer experience.

September 6, 2024

by Joe Zhou

· 5,596 Views · 1 Like

DRY Your Apache APISIX Config

DRY is an important principle in software development. This post will show you how to apply it to Apache APISIX configuration.

September 6, 2024

by Nicolas Fränkel

· 4,496 Views · 4 Likes

The AI Security Gap: Protecting Systems in the Age of Generative AI

As GenAI adoption surges, organizations face new security risks. Learn to protect your AI systems from prompt hacking and other emerging threats.

September 5, 2024

by Tom Smith

CORE

· 5,626 Views · 1 Like

Keeping Two Multi-Master Databases Aligned With a Vector Clock

In this article, learn about an experience in keeping two different databases aligned with two different technologies by using an application-level solution.

September 5, 2024

by Claudio Guidi

CORE

· 7,079 Views · 4 Likes

Setting Up a Data Warehouse for Starlight: A Comprehensive Guide

Learn architectural considerations, essential tools, and technologies, and see sample code snippets to illustrate key steps of a data warehouse setup.

September 5, 2024

by Harsh Daiya

CORE

· 4,317 Views · 1 Like

Supercharging IaC With AI for Next-Gen Infrastructure Efficiency

This article explores some important areas where AI is reshaping IaC operations and discusses what the future may hold.

September 4, 2024

by Omry Hay

· 7,345 Views · 2 Likes

My Journey With AWS CDK and Java: What You Need To Know

Learn how Java is a very viable option when working with the AWS CDK, especially for workload teams already familiar with the language and its ecosystem.

September 4, 2024

by Jeroen Reijn

CORE

· 8,667 Views · 5 Likes

How to Conduct Effective Data Security Audits for Big Data Systems

Learn key strategies for conducting thorough data security audits in big data systems to safeguard sensitive information.

September 4, 2024

by Devin Partida

· 5,606 Views · 1 Like

MLOps: How to Build a Toolkit to Boost AI Project Performance

AI projects could end up among the 90% that fail due to common implementation pitfalls. Here, learn how to change the game with the right MLOps tools.

September 3, 2024

by Alexander Simonov

· 7,375 Views · 4 Likes

How We Optimized Read Performance: Readahead, Prefetch, and Cache

In this in-depth analysis, take a deep dive into how JuiceFS enhances read performance using readahead, prefetch, and cache.

September 3, 2024

by Edric Mo

· 5,295 Views · 2 Likes

Automating Cucumber Data Table to Java Object Mapping in Your Cucumber Tests

Learn how to use the library cucumber-datatable-to-bean-mapping to simplify Data Table usage on your Cucumber tests.

September 3, 2024

by Thomas Deblock

· 8,990 Views · 4 Likes

Configuring a Connection Pool

A connection pooler is a software component that manages database connections. In this post, learn what a connection pooler is and how to configure it.

September 3, 2024

by Adam Furmanek

CORE

· 3,514 Views · 4 Likes

Open Standards for Data Lineage: OpenLineage for Batch and Streaming

Explores trends and efforts to provide an open standard with OpenLineage, and how data governance solutions help fulfill enterprise-wide data governance needs.

September 3, 2024

by Kai Wähner

CORE

· 6,564 Views · 1 Like

Bridging the Gap: How Developers and Security Professionals Can Collaborate for Better Cybersecurity

Experts at Black Hat 2024 reveal how developers and security pros can collaborate better: from shifting left to embracing AI and prioritizing user experience.

September 3, 2024

by Tom Smith

CORE

· 5,341 Views · 1 Like

Confusion Matrix vs. ROC Curve: When to Use Which for Model Evaluation

The Confusion Matrix and the ROC Curve evaluate model performance in machine learning and data science. Compare and learn when to use each in model evaluation.

September 3, 2024

by Fizza Jatniwala

· 7,172 Views · 1 Like

Workload Protection in the Cloud: Why It Matters More Than Ever

Protecting cloud workloads is crucial as threats evolve and environments become more complex. Learn about effective Cloud Workload Protection (CWP) here.

September 3, 2024

by Josephine Eskaline Joyce

CORE

· 4,935 Views · 4 Likes

The Latest Data Engineering Topics