[1fanwang]

Which issue does this PR close?

Fixes #2419

Supersedes #2438.

What changes are included in this PR?

Writing a Puffin file with footer compression enabled fails when the writer closes:

FeatureUnsupported => LZ4 compression is not supported currently

Writers can now finish these files, and readers can open them. The encoding follows the Puffin spec.

Footer compression stays opt-in because the Java reader cannot decode these footers yet.

Are these changes tested?

Testing Done

Rust 1.97.0-nightly on macOS, using the local filesystem and cached dependencies.

The same standalone program writes real Puffin files, closes them, and reads back their metadata and blobs through the public API. It starts with an empty file with footer compression enabled.

#	Scenario	Command	Result
1	Without the fix	cargo run with the main-snapshot manifest below	Closing the compressed footer fails; exit 1
2	With the fix	cargo run with the PR manifest below	Eight writer/read-back cases pass; exit 0

Save the manifests and shared program below at their labelled paths, then run from the checkout root. The unchanged source for the first run is 000e791.

mkdir -p ../iceberg-main-snapshot
git archive 000e791d5229d9d136cda2b41fa0d7563fa58e1e | tar -x -C ../iceberg-main-snapshot

Without the fix:

env CARGO_BUILD_JOBS=2 CARGO_INCREMENTAL=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_TARGET_DIR=target cargo run --quiet --offline --manifest-path evidence/puffin-footer-repro-main/Cargo.toml -- evidence/puffin-footer-repro/main-files
printf 'exit_code=%s\n' "$?"

Error: FeatureUnsupported => LZ4 compression is not supported currently

exit_code=1

With the fix:

env CARGO_BUILD_JOBS=2 CARGO_INCREMENTAL=0 CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_TARGET_DIR=target cargo run --quiet --offline --manifest-path evidence/puffin-footer-repro/Cargo.toml -- evidence/puffin-footer-repro/fixed-files
printf 'exit_code=%s\n' "$?"

PASS case=footer-lz4-blob-empty compress_footer=true blobs=0 bytes=55
PASS case=footer-lz4-blob-none compress_footer=true blobs=1 bytes=9942
PASS case=footer-lz4-blob-lz4 compress_footer=true blobs=1 bytes=348
PASS case=footer-lz4-blob-zstd compress_footer=true blobs=1 bytes=300
PASS case=footer-none-blob-empty compress_footer=false blobs=0 bytes=32
PASS case=footer-none-blob-none compress_footer=false blobs=1 bytes=9920
PASS case=footer-none-blob-lz4 compress_footer=false blobs=1 bytes=326
PASS case=footer-none-blob-zstd compress_footer=false blobs=1 bytes=278
PASS 8 real-file writer/read-back cases
exit_code=0

The passing cases cover empty files and uncompressed, LZ4, and Zstd blobs, each with footer compression enabled and disabled. Metadata and blob contents are compared after read-back.

Reproducer source: evidence/puffin-footer-repro/Cargo.toml

[package]
name = "puffin-footer-repro"
version = "0.0.0"
edition = "2024"
publish = false

[workspace]

[dependencies]
iceberg = { path = "../../crates/iceberg" }
tokio = { version = "=1.53.1", features = ["macros", "rt-multi-thread"] }

Reproducer source: evidence/puffin-footer-repro-main/Cargo.toml

[package]
name = "puffin-footer-repro-main"
version = "0.0.0"
edition = "2024"
publish = false

[workspace]

[[bin]]
name = "puffin-footer-repro-main"
path = "../puffin-footer-repro/src/main.rs"

[dependencies]
iceberg = { path = "../../../iceberg-main-snapshot/crates/iceberg" }
tokio = { version = "=1.53.1", features = ["macros", "rt-multi-thread"] }

Reproducer source: evidence/puffin-footer-repro/src/main.rs

use std::collections::HashMap;
use std::error::Error;
use std::fs;
use std::path::PathBuf;

use iceberg::io::FileIO;
use iceberg::puffin::{Blob, CompressionCodec, PuffinReader, PuffinWriter};

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    let directory = PathBuf::from(
        std::env::args_os()
            .nth(1)
            .ok_or("usage: puffin-footer-repro <output-directory>")?,
    );
    fs::create_dir_all(&directory)?;
    let file_io = FileIO::new_with_fs();
    let blob = Blob::builder()
        .r#type("org.example.puffin-round-trip".to_owned())
        .fields(vec![1])
        .snapshot_id(7)
        .sequence_number(1)
        .data(
            "Puffin footer compression round-trip.\n"
                .repeat(256)
                .into_bytes(),
        )
        .properties(HashMap::new())
        .build();
    let mut cases = 0;

    for compress_footer in [true, false] {
        for codec in [
            None,
            Some(CompressionCodec::None),
            Some(CompressionCodec::Lz4),
            Some(CompressionCodec::zstd_default()),
        ] {
            let footer = if compress_footer { "lz4" } else { "none" };
            let blob_codec = codec.map_or("empty", |value| value.name());
            let name = format!("footer-{footer}-blob-{blob_codec}");
            let path = directory.join(format!("{name}.puffin"));
            let properties = if codec.is_some() {
                HashMap::from([("case".to_owned(), name.clone())])
            } else {
                HashMap::new()
            };
            let output = file_io.new_output(path.to_str().ok_or("non-UTF-8 output path")?)?;
            let mut writer =
                PuffinWriter::new(&output, properties.clone(), compress_footer).await?;
            if let Some(codec) = codec {
                writer.add(blob.clone(), codec).await?;
            }
            writer.close().await?;

            let reader = PuffinReader::new(output.to_input_file()).await?;
            let metadata = reader.file_metadata().await?;
            assert_eq!(metadata.properties(), &properties);
            assert_eq!(metadata.blobs().len(), usize::from(codec.is_some()));
            for blob_metadata in metadata.blobs() {
                assert_eq!(Some(blob_metadata.compression_codec()), codec);
                assert_eq!(reader.blob(blob_metadata).await?, blob);
            }
            println!(
                "PASS case={name} compress_footer={compress_footer} blobs={} bytes={}",
                metadata.blobs().len(),
                fs::metadata(&path)?.len()
            );
            cases += 1;
        }
    }

    println!("PASS {cases} real-file writer/read-back cases");
    Ok(())
}

AI Disclosure

AI disclosure: GitHub Copilot CLI wrote the implementation and tests; GitHub Copilot CLI (GPT-6.1 Sol) handled this update. No Java runtime interoperability run was performed.

[Copilot]

Pull request overview

Implements end-to-end LZ4 support needed for Puffin footer compression/decompression so PuffinWriter can successfully close when compress_footer=true, and FileMetadata reading can open Puffin files with an LZ4-compressed footer per the Puffin spec.

Changes:

• Implement LZ4 frame compression/decompression via lz4_flex in CompressionCodec.
• Update Puffin writer/reader tests to validate LZ4-compressed blobs and LZ4-compressed footers round-trip.
• Add lz4_flex as a direct workspace dependency (and in the iceberg crate) to make the codec implementation explicit.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/iceberg/src/puffin/writer.rs	Updates tests to expect LZ4 compression to work and adds coverage for compressed footers.
crates/iceberg/src/puffin/metadata.rs	Updates footer-metadata test to ensure LZ4-compressed footer payloads decode successfully.
crates/iceberg/src/compression.rs	Implements LZ4 frame compress/decompress and expands codec tests (including frame magic assertion).
crates/iceberg/Cargo.toml	Adds lz4_flex as a direct dependency for the iceberg crate.
Cargo.toml	Adds lz4_flex to workspace dependencies to share the version across crates.
Cargo.lock	Records lz4_flex as a direct dependency of iceberg.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JeonDaehong]

Checked the frames against the reference liblz4 (1.9.4, via python-lz4) both ways: frames from liblz4 (incl. multi-block and block-checksum variants) decode here, and frames written here decode with liblz4 with the content size set. LGTM. A few small things:

• Parquet uses lz4_flex 0.14.0, so lz4_flex = "0.14" would share its copy (0.13 comes from another dep). It builds and the tests pass with 0.14 unchanged.
• log = "0.4.28" in the workspace Cargo.toml isn't used anywhere.
• Java can't read LZ4 footers yet (PuffinFormat compress/decompress are still TODO), so it may be worth a note on compress_footer that such files aren't readable by iceberg-java today.

[1fanwang]

A few small things:

@JeonDaehong done in 927c8aa. All three suggestions are addressed.

[JeonDaehong]

Thanks @1fanwang, all three look good: lz4_flex 0.14 shared with parquet, no unused log dependency, and the compress_footer doc now warns about iceberg-java.