Simple SDM Pipeline

This example walks through a basic SDL streaming pipeline that generates synthetic Cursor-on-Target (CoT) XML messages, converts them to SDM entities, and publishes the entities to the SDM service.

Overview

In this example, you will:

  • Create a pipeline from scratch.

  • Add the built-in CoT XML Generator.

  • Store generated CoT XML messages in a streaming dataset.

  • Transform CoT XML into SDM protobuf messages.

  • Publish the transformed entities to SDM.

  • Confirm the published entities from the SDM page.

The completed pipeline follows this flow:

CoT XML Generator → CoT XML dataset → CoT XML to SDM transformer → SDM Entities dataset → Publish to SDM

Simple pipeline overview placeholder

Prerequisites

Before starting, ensure you have:

  • An active SDL account with permission to create and run pipelines.

  • Access to the CoT XML Generator, CoT XML to SDM, and Publish to SDM pipeline components.

  • Access to the SDM page to verify published entities.

Step 1: Create a Pipeline

  1. Open Catalog.

  2. Select Pipelines.

  3. Click Create.

  4. Choose Start From Scratch.

  5. Enter a name such as Simple CoT to SDM Pipeline.

  6. Add an optional description and tags.

  7. Click Continue.

Step 2: Add the CoT XML Generator

The CoT XML Generator creates synthetic CoT XML messages so you can test the pipeline without connecting an external TAK server or live tactical feed.

  1. From the pipeline canvas, open the add menu.

  2. Select the Sources tab.

  3. Search for CoT XML Generator.

  4. Add CoT XML Generator to the canvas.

CoT XML Generator placeholder

Step 3: Create a CoT XML Dataset

Create a streaming dataset to hold the generated CoT XML messages.

  1. Open the Datasets tab.

  2. Add Create New Dataset to the canvas.

  3. Select the dataset node.

  4. Configure the dataset:

    Field Value

    Display Name

    CoT XML

    Description

    Synthetic Cursor-on-Target XML messages

    Resource Type

    Streaming

    Resource Name

    example-cot-xml

  5. Click Done.

  6. Connect the CoT XML Generator output to the CoT XML dataset.

  7. Click Save Changes.

A storage warning can appear before the pipeline runs for the first time. The backing Kafka topic is created when the pipeline starts.

Step 4: Convert CoT XML to SDM

Add the transformer that converts each CoT XML message into an SDM entity message.

  1. Open the Transformers tab.

  2. Search for CoT XML to SDM.

  3. Add the CoT XML to SDM transformer to the canvas.

  4. Add another streaming dataset for transformed SDM entities.

  5. Configure the dataset:

    Field Value

    Display Name

    SDM Entities

    Description

    SDM entities converted from synthetic CoT XML

    Resource Type

    Streaming

    Resource Name

    example-sdm-entities

  6. Connect CoT XML to CoT XML to SDM.

  7. Connect CoT XML to SDM to SDM Entities.

  8. Click Save Changes.

CoT XML to SDM transformer placeholder

Step 5: Publish to SDM

Add the sink that publishes the transformed SDM entities to the SDM service.

  1. Open the Sinks tab.

  2. Search for Publish to SDM.

  3. Add Publish to SDM to the canvas.

  4. Connect the SDM Entities dataset to Publish to SDM.

  5. Open the sink configuration.

  6. Leave the default SDM endpoint settings unless your environment requires custom values.

  7. Click Done.

  8. Click Save Changes.

Publish to SDM placeholder

Step 6: Run the Pipeline

  1. Confirm each node has no required configuration warnings.

  2. Click the start toggle in the pipeline header.

  3. Wait for the pipeline stages to start.

  4. Select each dataset and use Data Preview to confirm records are flowing.

The CoT XML dataset should show generated XML messages. The SDM Entities dataset should show transformed SDM entity messages.

Step 7: Verify Published Entities

Use the SDM page to confirm that the sink published entities successfully.

  1. Open SDM from the main navigation.

  2. Click Search with the default empty filter.

  3. Confirm matching entity records appear.

  4. Optionally filter by an entity ID from the pipeline output.

  5. Click View in Maps to open Maps with the current SDM filter.

For more SDM search details, see SDM Page.

Verify SDM entities placeholder

Expected Results

Stage Expected result

CoT XML Generator

Synthetic CoT XML messages are produced continuously.

CoT XML dataset

Generated CoT XML messages are visible in data preview.

CoT XML to SDM

CoT XML messages are transformed into SDM entity messages.

SDM Entities dataset

Transformed SDM entity messages are visible in data preview.

Publish to SDM

SDM entities are published to the SDM service.

SDM page

Published entities appear in SDM search results and can be opened in Maps.

Key Takeaways

This example introduces the basic pipeline workflow:

  • generate test data,

  • persist stream data between stages,

  • transform tactical messages into SDM,

  • publish SDM entities to the platform service,

  • verify the result through the operator UI.

After completing this example, replace the generator with a real CoT source, such as a TAK Server connection, to use the same pattern with live operational data.