Geospatial Pipelines

SDL Data Pipelines can publish geospatial data into services used by Maps and GeoServer. Use these workflows to convert raster files to Cloud Optimized GeoTIFFs (COGs), copy map files into the GeoServer filesystem, or publish GeoJSON features as GeoServer layers.

Common Workflows

Workflow Pipeline shape Result

Raster to COG

s3 input, raster to cog, s3 output

Source rasters are converted with GDAL and written as COG files in object storage.

Raster to COG for GeoServer

s3 input, raster to cog, geoserver output

Raster files are copied into the GeoServer filesystem so GeoServer can serve them.

GeoJSON to GeoServer

Kafka GeoJSON input, postgis output

GeoJSON features are written to PostGIS and published as GeoServer WMS/WFS layers.

Raster to COG

The raster to cog transformer converts raster files to Cloud Optimized GeoTIFFs using GDAL gdal_translate. It reads source object metadata from a streamed S3 input and writes the COG directly to object storage.

Use this workflow when raw imagery, elevation, or other raster products need to be optimized for range requests and map serving.

Setting Purpose

INPUT_CONNECTOR=s3

Reads raster files from object storage.

INPUT_FORMAT=raster

Selects raster input handling.

OUTPUT_FORMAT=cog

Selects COG conversion.

S3_TRANSFER_MODE=streamed

Passes object metadata through the pipeline instead of loading full raster bytes into memory.

S3_IN_LOCATION

Source bucket or prefix, for example s3://imagery/raw/.

S3_OUT_LOCATION

Destination bucket or prefix, for example s3://imagery/cogs/.

S3_OUT_OVERWRITE_POLICY

Destination conflict behavior: fail, overwrite, or if_source_newer. Default is if_source_newer.
Streamed S3 mode requires INPUT_CONNECTOR=s3. For raster to cog, the output connector can be s3, logger, or geoserver.

COG Compression Options

COG_COMPRESS maps to GDAL’s COG COMPRESS creation option. The transformer default is LZW, which is a safe lossless default for mixed raster data.

Value Use when Notes

LZW

You need a broadly compatible lossless default.

Good general-purpose choice. Often larger than DEFLATE or ZSTD.

DEFLATE

You need lossless compression with broad compatibility.

Often smaller than LZW, but may be slower to write.

ZSTD

You need lossless compression with a strong speed and size tradeoff.

Requires GDAL/libtiff support in the runtime image.

JPEG

You are converting visual imagery where lossy compression is acceptable.

Not appropriate for rasters where exact cell values must be preserved.

WEBP

You are optimizing visual imagery and the target GDAL build supports WebP.

Lossy or lossless behavior depends on GDAL options and runtime support.

LERC

You are storing numeric rasters with controlled error tolerance.

Useful for elevation or scientific rasters.

LERC_DEFLATE

You need LERC encoding plus DEFLATE compression.

Useful when numeric precision and smaller files both matter.

LERC_ZSTD

You need LERC encoding plus ZSTD compression.

Requires GDAL/libtiff support in the runtime image.

LZMA

You need lossless compression and can accept slower writes.

Compatibility and performance vary by GDAL build.

NONE

You are debugging or need uncompressed output.

Produces large files and is rarely best for production.

Additional GDAL creation options can be supplied with COG_EXTRA_CREATION_OPTIONS as comma-separated NAME=VALUE pairs. Examples include LEVEL=9, QUALITY=85, PREDICTOR=YES, or MAX_Z_ERROR=0.001.

Setting Purpose Default

COG_COMPRESS

GDAL COG compression method.

LZW

COG_BLOCKSIZE

GDAL COG block size. Must be a positive integer divisible by 16.

512

COG_BIGTIFF

BigTIFF handling: YES, NO, IF_NEEDED, or IF_SAFER.

IF_SAFER

COG_OVERVIEWS

Overview handling: AUTO, IGNORE_EXISTING, FORCE_USE_EXISTING, or NONE.

AUTO

COG_RESAMPLING

Optional resampling method for image data.

None

COG_OVERVIEW_RESAMPLING

Optional resampling method for overviews.

None

COG_PREDICTOR

Optional predictor setting for compatible compression methods.

None

COG_EXTRA_CREATION_OPTIONS

Extra comma-separated GDAL COG creation options.

None

GeoServer File Sync

The geoserver output connector copies streamed S3 objects into a GeoServer filesystem path. This is intended for file-backed GeoServer layers where GeoServer and the transformer pod share storage.

Setting Purpose

OUTPUT_CONNECTOR=geoserver

Selects the GeoServer filesystem writer.

S3_TRANSFER_MODE=streamed

Required. The writer expects streamed S3 source metadata.

S3_IN_LOCATION

Source S3 bucket or prefix used by the writer.

GEOSERVER_FILESYSTEM_PATH

Absolute destination root in the GeoServer filesystem.

GEOSERVER_OVERWRITE_POLICY

Destination conflict behavior: fail or overwrite. Default is fail.

The writer preserves the source object key under GEOSERVER_FILESYSTEM_PATH. For example, s3://imagery/cogs/site-a/image.tif copied with GEOSERVER_FILESYSTEM_PATH=/gis_data_files becomes /gis_data_files/site-a/image.tif when the input prefix is imagery/cogs.

GeoJSON to GeoServer

The GeoJSON to GeoServer workflow writes GeoJSON features to PostGIS, then publishes the table or view as a GeoServer layer. Use it for live tracks, feature collections, and other vector data that Maps should discover as WMS/WFS layers.

Setting Purpose

OUTPUT_CONNECTOR=postgis

Writes GeoJSON features to the PostGIS database backing GeoServer.

POSTGRES_TABLE_NAME

Layer/view name. The history table uses the configured table name followed by _history.

GEOSERVER_URL

Enables GeoServer REST integration when set.

GEOSERVER_USERNAME

GeoServer REST username. Default is admin.

GEOSERVER_PASSWORD

GeoServer REST password. Required when GEOSERVER_URL is set.

GEOSERVER_WORKSPACE

GeoServer workspace. Default is rdp.

GEOSERVER_DATASTORE

GeoServer datastore. Default is rdp-pg.

GEOSERVER_ACCESS_GROUPS

Comma-separated access groups to add as GeoServer access:* keywords.

GEOSERVER_MVT_ENABLED

Enables GeoWebCache Mapbox Vector Tile configuration, bbox refresh, and tile cache truncation.

When GeoServer REST integration is enabled, layer publishing is non-fatal. If GeoServer calls fail, the PostGIS write still succeeds and layer publishing is retried on later writes.

Related Information