Overview

Relevant source files

• src/iceberg/CMakeLists.txt
• src/iceberg/expression/meson.build
• src/iceberg/meson.build
• src/iceberg/table.cc
• src/iceberg/table.h
• src/iceberg/test/CMakeLists.txt
• src/iceberg/test/meson.build
• src/iceberg/type_fwd.h

Purpose and Scope

This document provides a high-level overview of the Apache Iceberg C++ library, its architecture, and its core components. It introduces the fundamental concepts, modular design, and key subsystems that make up the implementation.

For detailed information about specific subsystems:

• Building and integrating the library: see Getting Started
• Core type system and error handling: see Core Concepts
• Table metadata structure: see Table Metadata
• Expression system for filtering: see Expressions and Filtering
• Catalog implementations: see Catalog System
• Data format integration: see File Format Support

Sources: src/iceberg/type_fwd.h1-195 src/iceberg/CMakeLists.txt1-227

What is Apache Iceberg C++?

Apache Iceberg C++ is a native C++ implementation of the Apache Iceberg table format. Apache Iceberg is an open table format for large analytic datasets that provides:

• Schema evolution - Add, rename, or drop columns without rewriting data
• Partition evolution - Change partitioning layout as data and queries evolve
• Hidden partitioning - Partition pruning without requiring users to specify partition filters
• Time travel and rollback - Query historical versions and revert to previous states
• ACID transactions - Reliable concurrent writes with snapshot isolation
• Scalable metadata - Metadata structures that scale to billions of files

This C++ library enables native applications to read, write, and manage Iceberg tables without depending on Java Virtual Machine-based implementations.

Sources: src/iceberg/table_metadata.h src/iceberg/snapshot.h

Library Architecture

The library follows a layered architecture with clear separation of concerns:

Sources: src/iceberg/CMakeLists.txt18-92 src/iceberg/type_fwd.h27-194

Core vs Bundle Architecture

The library uses a two-tier modular design to balance functionality and minimal dependencies:

Component	Library	Purpose	Dependencies
Core	libiceberg	Fundamental Iceberg operations, metadata management, schema evolution, expression system	nanoarrow, nlohmann_json, CRoaring, spdlog, zlib
Bundle	libiceberg_bundle	Data format support for Arrow, Parquet, and Avro	Core + Apache Arrow + Apache Parquet + Apache Avro

Core Library (libiceberg)

Always built and provides:

• Type system and schema definitions (src/iceberg/type.h src/iceberg/schema.h)
• Table metadata management (src/iceberg/table_metadata.h)
• Expression and predicate system (src/iceberg/expression/)
• In-memory catalog (src/iceberg/catalog/memory/in_memory_catalog.cc)
• JSON serialization (src/iceberg/json_internal.cc)
• Result-based error handling (src/iceberg/result.h)

Bundle Library (libiceberg_bundle)

Optional, enabled via ICEBERG_BUILD_BUNDLE flag:

• Arrow filesystem integration (src/iceberg/arrow/arrow_fs_file_io.cc)
• Parquet reader/writer (src/iceberg/parquet/parquet_reader.cc src/iceberg/parquet/parquet_writer.cc)
• Avro reader/writer (src/iceberg/avro/avro_reader.cc src/iceberg/avro/avro_writer.cc)
• Schema conversion utilities (src/iceberg/avro/avro_schema_util.cc src/iceberg/parquet/parquet_schema_util.cc)

Sources: src/iceberg/CMakeLists.txt99-220 src/iceberg/meson.build137-161

Key Components and Code Entities

Type System and Schema

The type system forms the foundation of Iceberg's data model:

• TypeId enum - Identifies primitive and nested types (src/iceberg/type_fwd.h35-53)
• Type class hierarchy - Base class with derived types like StructType, ListType, MapType, primitives (src/iceberg/type.h)
• Schema class - Inherits from StructType, manages field IDs and schema evolution (src/iceberg/schema.h)
• SchemaField - Individual fields with id, name, type, required/optional flag (src/iceberg/schema_field.h)

Table and Metadata

Tables are the primary interface for interacting with Iceberg datasets:

• Table class - Main table interface (src/iceberg/table.h37-154)
  • Table::Make() - Factory method for creating tables
  • Table::schema(), Table::spec(), Table::sort_order() - Access current metadata
  • Table::NewScan() - Create table scan builders
  • Table::NewTransaction() - Begin multi-operation transactions
• StagedTable - Table created but not yet committed (src/iceberg/table.h157-172)
• StaticTable - Read-only table without catalog (src/iceberg/table.h176-192)
• TableMetadata - Immutable metadata structure containing schemas, partition specs, sort orders, snapshots (src/iceberg/table_metadata.h)
• TableMetadataBuilder - Fluent API for building metadata updates

Catalog Interface

Catalogs manage namespace and table lifecycles:

• Catalog abstract interface - Namespace and table operations (src/iceberg/catalog.h)
• InMemoryCatalog - Thread-safe in-memory implementation (src/iceberg/catalog/memory/in_memory_catalog.cc22)
• RestCatalog - HTTP-based catalog client (src/iceberg/catalog/rest/)

Expression System

Type-safe expression trees for filtering and optimization:

• Expression base class - Root of expression hierarchy (src/iceberg/expression/expression.h)
• Predicate classes - UnaryPredicate, LiteralPredicate, SetPredicate (src/iceberg/expression/predicate.h)
• Term classes - UnboundTerm, BoundTerm, BoundReference, BoundTransform (src/iceberg/expression/term.h)
• Transform enum and classes - identity, bucket, truncate, temporal transforms (src/iceberg/transform.h)
• Expressions factory - Static methods to build expressions (src/iceberg/expression/expressions.h)

Sources: src/iceberg/type_fwd.h83-194 src/iceberg/table.h37-154 src/iceberg/CMakeLists.txt20-92

Table Operation Flow

The following diagram shows how a typical table operation flows through the system components:

Sources: src/iceberg/table.cc39-83 src/iceberg/catalog/memory/in_memory_catalog.cc

Core Dependencies

The following table lists the core dependencies and their roles:

Dependency	Purpose	Usage
nanoarrow	Apache Arrow C interface	Lightweight Arrow data structure handling
nlohmann_json	JSON parsing and serialization	Metadata file format, REST API communication
CRoaring	Compressed bitmap library	Efficient data filtering and metrics
spdlog	Logging framework	Diagnostic logging throughout the library
zlib	Compression	GZIP compression for metadata files

Bundle-only dependencies:

Dependency	Purpose	Build Flag
Apache Arrow	Columnar data format and I/O	ICEBERG_BUILD_BUNDLE
Apache Parquet	Columnar file format	ICEBERG_BUILD_BUNDLE
Apache Avro	Row-oriented data format	ICEBERG_BUILD_BUNDLE

Sources: src/iceberg/CMakeLists.txt99-124 src/iceberg/meson.build123-135

Build Configuration

The library supports both CMake and Meson build systems with identical functionality:

Build Flags

Flag	Default	Description
ICEBERG_BUILD_BUNDLE	OFF	Build data format support (Arrow, Parquet, Avro)
ICEBERG_BUILD_REST	OFF	Build REST catalog client
ICEBERG_BUILD_TESTS	OFF	Build test suite
ICEBERG_BUILD_BENCHMARKS	OFF	Build performance benchmarks

Library Outputs

• Static libraries: libiceberg_static.a, libiceberg_bundle_static.a
• Shared libraries: libiceberg_shared.so, libiceberg_bundle_shared.so (or .dll/.dylib)

Sources: src/iceberg/CMakeLists.txt1-227 src/iceberg/meson.build1-216 src/iceberg/test/CMakeLists.txt31-53

Error Handling Pattern

The library uses a Result<T> pattern for error handling, avoiding exceptions for better performance and explicit error handling:

Key macros for error propagation:

• ICEBERG_ASSIGN_OR_RAISE(lhs, rhs) - Assigns result or returns early on error
• ICEBERG_RETURN_IF_ERROR(expr) - Returns early if expression is an error status

Sources: src/iceberg/result.h src/iceberg/util/macros.h

File Organization

The source code is organized into logical subsystems:

src/iceberg/
├── catalog/                 # Catalog implementations
│   ├── memory/             # In-memory catalog
│   └── rest/               # REST catalog (optional)
├── expression/             # Expression and predicate system
├── manifest/               # Manifest file handling
├── row/                    # Row-based data structures
├── update/                 # Table update operations
├── util/                   # Utility functions
├── arrow/                  # Arrow integration (bundle only)
├── avro/                   # Avro integration (bundle only)
└── parquet/                # Parquet integration (bundle only)

Sources: src/iceberg/CMakeLists.txt20-150 src/iceberg/meson.build42-115

Next Steps

To learn more about specific subsystems:

• Getting Started - Building, installing, and integrating the library
• Core Concepts - Deep dive into architecture, error handling, and type system
• Table Metadata - Understanding table structure and evolution
• Expressions and Filtering - Using the expression system for query optimization
• Catalog System - Working with different catalog implementations
• File Format Support - Reading and writing Arrow, Parquet, and Avro data