Iceberg Compaction and Fine-Grained Access Control: Performance Challenges and Solutions

Modern data lakes increasingly rely on Apache Iceberg for managing large analytical datasets, while organizations simultaneously demand fine-grained access control (FGAC) to secure sensitive data. However, combining these technologies can create unexpected performance bottlenecks that significantly impact query execution times. This article explores the technical challenges that arise when implementing FGAC on Iceberg tables and provides practical guidance for choosing the right processing engine for your use case.

Understanding Iceberg Compaction

Apache Iceberg is an open table format designed for huge analytical datasets. One of its core features is compaction — the process of combining smaller data files into larger, more efficient ones to optimize query performance and reduce metadata overhead.

Iceberg compaction works through several mechanisms:

• File consolidation: Merging small files into larger ones to reduce the number of files that need to be processed
• Metadata management: Maintaining detailed manifest files that track every data file, change, and snapshot
• Snapshot management: Enabling time travel capabilities through comprehensive version tracking
• Partition optimization: Organizing data files within partitions for optimal query performance

The compaction process generates extensive metadata in the form of manifest files. These files contain critical information about data file locations, schemas, partition information, and file-level statistics. While this metadata enables powerful features like time travel and efficient query planning, it also creates potential performance challenges when combined with security layers.

Fine-Grained Access Control Requirements

Fine-grained access control (FGAC) provides security at the row and column level, going beyond traditional table-level permissions. FGAC systems evaluate security policies against user requests and filter data accordingly, ensuring users only access data they're authorized to see.

Key FGAC capabilities include:

• Row-level security: Filtering rows based on user attributes and data content
• Column-level security: Restricting access to specific columns or masking sensitive data
• Cell-level security: Combining row and column restrictions for granular control
• Dynamic policy evaluation: Real-time policy enforcement during query execution

When FGAC is applied to Iceberg tables, the security system must evaluate policies against the extensive metadata structure, creating additional processing overhead during query planning and execution.

FGAC Behavior in Different Processing Engines

Glue 5.0: Native FGAC Support

The evolution from Glue 4.0 to Glue 5.0 marked a significant milestone in data lake security capabilities. While Glue 4.0 provided robust ETL functionality, it lacked native support for fine-grained access control on data catalog tables, forcing organizations to implement complex workarounds or rely on external security layers. Glue 5.0 fundamentally changed this landscape by introducing built-in FGAC support, enabling secure data processing with proper permission enforcement at both row and column levels.

This advancement represents more than just a feature addition — it's an architectural shift that allows data teams to implement comprehensive security policies without sacrificing performance or operational simplicity. The integration is seamless, requiring minimal configuration changes while providing enterprise-grade security capabilities.

How FGAC Works in Glue 5.0

The technical implementation of FGAC in Glue 5.0 leverages Spark's resource profile capabilities in an innovative way. Rather than applying security as an external layer, Glue 5.0 uses Spark resource profiles to create two distinct execution contexts:

• User profile: Executes user-supplied code with appropriate permissions
• System profile: Enforces Lake Formation policies and security constraints

This dual-profile architecture creates a secure execution environment where security policies are evaluated and enforced at the Spark engine level, ensuring that data access restrictions are applied consistently across all operations. The system profile acts as a security gateway, intercepting data access requests and applying the appropriate filters before data reaches the user profile.

Standard Table Implementation

For organizations working with traditional catalog tables, the implementation is straightforward and requires minimal code changes:

from pyspark.sql import SparkSession 
spark = SparkSession.builder.getOrCreate()
# Read from catalog table
df = spark.sql("SELECT * FROM datalake_link.events") 
df.show()

Required Job Parameters

• Key: --enable-lakeformation-fine-grained-access
• Value: true
• Glue version: 5.0 (Supports Spark 3.5, Scala 2, Python 3)

The simplicity of this approach masks the sophisticated security processing happening behind the scenes. When the job executes, Glue 5.0 automatically applies the appropriate security filters based on the user's permissions, ensuring that only authorized data is accessible.

Iceberg Table Implementation

Working with Iceberg tables introduces additional complexity due to the format's metadata-heavy architecture. However, Glue 5.0 handles this complexity gracefully with proper configuration:

from pyspark.sql import SparkSession 
spark = SparkSession.builder.getOrCreate()
# Configuration for Iceberg with FGAC
catalog_name = "spark_catalog"  # Don't change
region = "us-east-1"  # Provide the Glue Job's region
account_id = "123456789012"  # Provide the Glue Job's account
warehouse_path = "s3://your-data-bucket/events"  # Source S3 bucket with LF rules
spark = SparkSession.builder \
    .config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkSessionCatalog") \
    .config(f"spark.sql.catalog.{catalog_name}.warehouse", f"{warehouse_path}") \
    .config(f"spark.sql.catalog.{catalog_name}.catalog-impl", "org.apache.iceberg.catalog.GlueCatalog") \
    .config(f"spark.sql.catalog.{catalog_name}.io-impl", "org.apache.iceberg.io.S3FileIO") \
    .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config(f"spark.sql.catalog.{catalog_name}.client.region", f"{region}") \
    .config(f"spark.sql.catalog.{catalog_name}.account-id", f"{account_id}") \
    .getOrCreate()
df = spark.sql("SELECT * FROM datalake_link.events") 
df.show()

Additional Job Parameters for Iceberg

• Key: --enable-lakeformation-fine-grained-access
• Value: true
• Key: --datalake-formats
• Value: iceberg

The key difference in Iceberg implementation lies in the extensive Spark configuration required to properly integrate with the Iceberg catalog and file system. These configurations ensure that Glue 5.0 can effectively communicate with Iceberg's metadata layer while maintaining security policy enforcement.

Redshift: Performance Challenges With Iceberg

Redshift's approach to querying Iceberg tables reveals fundamental architectural differences that create significant performance challenges when combined with FGAC. Unlike modern distributed processing engines that were designed with metadata-heavy formats in mind, Redshift's query execution model struggles with Iceberg's manifest file architecture.

The core issue stems from Redshift's query planning process, which requires comprehensive metadata analysis before query execution can begin. For traditional data formats, this planning phase is typically brief and efficient. However, Iceberg tables with their extensive manifest file structures create a bottleneck that can render queries practically unusable.

When organizations attempt to leverage Redshift's powerful analytical capabilities with Iceberg tables, they often encounter a frustrating reality: queries that should complete in seconds or minutes instead take tens of minutes or fail to complete entirely. This performance degradation becomes even more pronounced when FGAC policies are applied, as the security evaluation adds another layer of complexity to an already strained process.

Setup Configuration

The initial setup for Redshift with Iceberg tables appears straightforward, following standard external schema creation patterns:

-- Create external schema

CREATE EXTERNAL SCHEMA telemetry_lake_link
FROM DATA CATALOG
DATABASE 'datalake_link'
IAM_ROLE 'arn:aws:iam::account:role/RedshiftLakeFormationRole'
REGION 'us-east-1';

-- Test query
SELECT namespace 
FROM telemetry_lake_link.events 
WHERE event_timestamp > current_date - interval '7' day;

However, this seemingly simple configuration masks the complex performance challenges that emerge during actual query execution.

Performance Issues

The reality of Redshift's interaction with Iceberg tables becomes apparent during query execution. The fundamental issue lies in Redshift's approach to metadata processing, which was optimized for traditional data warehouse scenarios rather than metadata-heavy formats like Iceberg.

During the query planning phase, Redshift must fetch and process every manifest file associated with the Iceberg table. This creates a cascading performance problem:

• Initial run: Approximately 20 minutes after fetching 375 manifest files
• Second run: Improved to 12 minutes, suggesting some temporary caching
• Third run: Degraded to 35+ minutes without completion, indicating cache ineffectiveness
• No persistent caching mechanism: Each query session may need to repeat the entire manifest processing cycle

This performance pattern reveals a critical architectural mismatch between Redshift's query execution model and Iceberg's metadata structure.

Monitoring Query Behavior

Understanding Redshift's behavior with Iceberg tables requires deep visibility into the query execution process. The following system queries provide crucial insights:

-- Monitor query history
SELECT *
FROM sys_query_history 
ORDER BY start_time DESC 
LIMIT 100;

-- Track S3 client operations
SELECT *
FROM stl_s3client 
WHERE pid = 1073947796
AND bucket = 'your-data-bucket'
ORDER BY recordtime DESC;

-- Count S3 operations
SELECT count(1) 
FROM stl_s3client 
WHERE pid = 1073947796
AND bucket = 'your-data-bucket';

These monitoring queries reveal the extent of S3 operations required for manifest file processing, often showing hundreds of individual file operations for a single query attempt.

Planning Stage Analysis

The query planning stage becomes the primary bottleneck, with extensive S3 operations as Redshift attempts to fetch manifest files sequentially. This process can consume hours without producing any query results, effectively making Redshift unsuitable for Iceberg tables that undergo frequent compaction cycles. The lack of intelligent manifest file caching compounds this issue, forcing each query to repeat the expensive metadata processing operation.

Athena: Balanced Performance for Ad-Hoc Queries

Athena occupies a middle ground in the Iceberg FGAC landscape, offering better performance characteristics than Redshift while maintaining the flexibility needed for analytical workloads. Unlike Redshift's problematic approach to manifest file processing, Athena was designed with cloud-native data formats in mind, providing more intelligent handling of metadata-heavy table formats.

The service represents a pragmatic choice for organizations that need to balance performance, cost, and operational complexity. While it may not offer the raw computational power of dedicated data warehouses or the advanced ETL capabilities of modern Spark-based solutions, Athena provides a reliable platform for interactive data analysis and exploration.

Strengths

Athena's architecture provides several advantages when working with Iceberg tables and FGAC:

• Intelligent manifest file handling: Unlike Redshift's sequential processing approach, Athena employs more efficient metadata processing strategies
• Ad-hoc analysis capabilities: Excellent for exploratory data analysis and one-time investigations
• Pay-per-query pricing model: Cost-effective for intermittent usage patterns
• Minimal operational overhead: Quick setup with minimal configuration requirements
• Cloud-native optimization: Designed specifically for cloud storage and modern data formats

Limitations

Despite its advantages, Athena has specific constraints that limit its applicability in certain scenarios:

• Query timeout restrictions: The default 30-minute timeout can be problematic for complex analytical queries
• Higher per-TB costs: At $5/TB, costs can escalate quickly for large-scale data processing
• Limited scalability: Not designed for high-throughput production workloads
• ETL limitations: Not suitable for continuous or complex ETL operations

Use Cases

Athena excels in specific scenarios where its strengths align with organizational needs:

• Exploratory data analysis: Interactive investigation of data patterns and relationships
• One-time data investigations: Ad-hoc queries for specific business questions
• Small to medium-sized analytical queries: Regular reporting and analysis within reasonable data volumes
• Development and testing scenarios: Prototyping and validation of data processing logic

The key to successful Athena implementation lies in understanding its operational boundaries and designing workloads that leverage its strengths while avoiding its limitations.

The combination of Iceberg compaction and fine-grained access control presents both opportunities and challenges for modern data architectures. While traditional processing engines may struggle with the manifest file overhead inherent in Iceberg tables, modern Spark-based solutions have evolved to handle these challenges effectively.

Disclaimer: The opinions expressed in this article are solely those of the author and do not represent the opinions or positions of any organization or employer.