INTERNAL_ICEBERG Connections
These connections map to an Iceberg table managed by the Lakekeeper catalog. The reference format is catalog.schema.table. Transformers connected via an input or output of this type will be provided configuration for both the Iceberg REST catalog and S3-compatible object storage.
When submitting a pipeline, the ref must be a three-part dotted string:
"ICEBERG_INPUT": {
"conn_type": "INTERNAL_ICEBERG",
"ref": "my_catalog.my_schema.my_table"
}
Each part of the reference (catalog, schema, table) must be non-empty. For example, my_catalog.my_schema.my_table is valid, but my_catalog..my_table is not.
Environment Provided
Transformers with at least one INTERNAL_ICEBERG connection will receive the following environment variables:
-
ICEBERG_CATALOG_REST_HOST- The Iceberg REST catalog endpoint (e.g. Lakekeeper) -
ICEBERG_CATALOG_REST_CATALOG_PATH- The catalog path on the REST endpoint
In addition, the object-storage environment variables (S3_ENDPOINT, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY) are also provided, since Iceberg tables are backed by object storage.