Compass Pipeline

This guide walks operators through onboarding Compass video feeds through SDL. The workflow discovers live Compass feeds, selects one feed as the pipeline trigger, and starts a Compass-to-Kafka pipeline for downstream processing.

Prerequisites

Before starting, make sure you have:

  • Access to SDL

  • Permission to connect data sources and create pipelines

  • Access to Compass

  • The Compass host URL for the target environment

  • A Compass API token with permission to discover video feeds

Step 1: Start data onboarding

  1. In SDL, open Catalog.

  2. Select Connect Data Source.

    Catalog page with Connect Data Source button

Step 2: Select Compass Video Feeds

  1. On the Add Data page, keep Sources selected.

  2. Search for compass.

  3. Under Platform Sources, select Compass Video Feeds.

    Compass Video Feeds source card

Use Compass Archived Feeds only when onboarding archived recordings for a known sensor and time range.

Step 3: Configure the Compass source

Enter the connection information for the Compass source. The token can be generated from Compass if you do not already have one.

Compass source configuration form

 
 

Field Required What to enter Example

Connection Name

Yes

A readable name for this Compass connection.

compass-test

Connection Groups

Yes

The group that should own the connection.

Public

Classification

Yes

The classification marking for datasets created from the source.

U

Compass API Token

Yes

Compass API token issued for this workflow. Treat this value as sensitive.

<COMPASS_API_TOKEN>

Compass Host

Yes

Base host URL for the Compass environment.

http://compass.test.url

Skip TLS Verification

Yes

Set to true for this workflow when directed by the environment owner.

true

Enable JSONL Sidecar

Yes

Set to true so JSONL sidecar metadata is emitted for discovered feeds.

true

Enable Kafka KLV Publish

Yes

Set to true so KLV data is published to Kafka for downstream processing.

true

Step 4: Generate a Compass token

If you need a new Compass token, generate one from Compass. In Chrome, the generated token is copied to the clipboard after you select Generate.

  1. Log in to Compass.

  2. Open Settings.

  3. From the Access menu, select API.

    Compass Access API menu

  4. On the API Token screen, choose the token expiration, address, and role required for SDL access.

  5. Select Generate.

  6. Paste the copied token into the Compass API Token field in SDL.

    Compass API Token generation screen

If no token is copied, use Chrome and try again. If token generation still fails, ask a Compass administrator to confirm your account can generate API tokens and that the selected role and address are valid for video feed discovery.

Step 5: Save the source and select a discovered feed

  1. Return to the SDL Compass source configuration.

  2. Confirm the Compass host, token, and boolean settings.

  3. Select Save Source.

  4. Wait for the Discovered Datasets list to populate.

  5. Select the live feed that should trigger the pipeline.

  6. Select Design Pipeline.

    Discovered Compass video feed datasets

If no feeds appear, select Check Again. If the list remains empty, verify the Compass host, API token permissions, Skip TLS Verification, Enable JSONL Sidecar, and Enable Kafka KLV Publish settings.

Step 6: Design the pipeline

Choose how the Compass feed should be processed.

  • Select Compass to Kafka to use the preconfigured pipeline template.

  • Select Build Custom Flow if the mission requires a custom source-anchored flow.

The preconfigured Compass to Kafka template places Compass data on Kafka and includes downstream stages for KLV metadata and GeoJSON/GeoServer processing.

  1. Select the desired starting point.

  2. Review the flow preview.

  3. Select Configure Outputs.

    Compass to Kafka pipeline design screen

Step 7: Configure outputs

Review each output exposed by the selected template. Update dataset names, storage names, topic names, or sink-specific settings as needed.

  1. Review the KLV Metadata Stream output.

  2. Review the selected trigger dataset.

  3. Review the KLV JSON to GeoJSON → GeoJSON to GeoServer output.

  4. Configure the optional GeoJSON to GeoServer sink target if required.

  5. Select Review & Start.

    Compass pipeline output configuration

Step 8: Review and start

Review the output summary and launch summary before starting the pipeline. Confirm the selected template, trigger dataset, output datasets, and sink settings are correct.

  1. Confirm all required outputs show Ready.

  2. Confirm the launch summary references the expected Compass video feed.

  3. Select Start Pipeline.

    Compass pipeline review and start screen

Step 9: View the running pipeline

After the pipeline starts, SDL opens the pipeline view. Confirm the pipeline is running and that the expected stages are connected.

The preconfigured Compass pipeline includes:

  • Compass Video Ingest

  • KLV Metadata Stream

  • KLV JSON to GeoJSON

  • KLV JSON to GeoJSON → GeoJSON to GeoServer

  • GeoJSON to GeoServer

    Running Compass to Kafka pipeline

Validate Compass data

After the pipeline starts, verify:

Check Expected result

Source

The Compass source is configured and feed discovery succeeds.

Trigger dataset

The selected Compass feed appears as the trigger dataset.

Pipeline status

The Compass pipeline is running.

Outputs

Output datasets or streams are created for KLV metadata and downstream processing.

Logs

Logs do not show authentication, feed discovery, TLS, or KLV publishing errors.

Troubleshooting

Symptom What to check

Compass token does not copy

Generate the token in Chrome. If it still does not copy, ask a Compass administrator to verify your token-generation permissions.

No discovered feeds appear

Verify the Compass host URL, API token, token role, and feed discovery permissions.

TLS or certificate errors

Confirm Skip TLS Verification is set to true for this workflow when directed by the environment owner.

Metadata is missing downstream

Confirm Enable JSONL Sidecar and Enable Kafka KLV Publish are set to true.

Pipeline starts but no messages appear

Confirm the selected Compass feed is live or has available KLV metadata.