INTERNAL_ICEBERG Connections

These connections map to an Iceberg table managed by the Lakekeeper catalog. The reference format is catalog.schema.table. Transformers connected via an input or output of this type will be provided configuration for both the Iceberg REST catalog and MinIO (since Iceberg tables are backed by object storage).

When submitting a pipeline, the ref must be a three-part dotted string:

"ICEBERG_INPUT": {
  "conn_type": "INTERNAL_ICEBERG",
  "ref": "my_catalog.my_schema.my_table"
}

Each part of the reference (catalog, schema, table) must be non-empty. For example, my_catalog.my_schema.my_table is valid, but my_catalog..my_table is not.

Environment Provided

Transformers with at least one INTERNAL_ICEBERG connection will receive the following environment variables:

  • ICEBERG_CATALOG_REST_HOST - The Iceberg REST catalog endpoint (e.g. Lakekeeper)

  • ICEBERG_CATALOG_REST_CATALOG_PATH - The catalog path on the REST endpoint

In addition, the MinIO environment variables (MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY) are also provided, since Iceberg tables are backed by object storage.