Documentation

Quick Start

Get up and running with Sorcha in under 5 minutes. This guide will walk you through creating your first secure data flow blueprint.

Prerequisites

• .NET 10 SDK or later
• Visual Studio 2024 or VS Code with C# extension
• Basic understanding of C# and async programming

Installation

Using .NET CLI

Install the Sorcha blueprint execution engine via NuGet:

dotnet new console -n MyDataFlow
cd MyDataFlow
dotnet add package Sorcha.Core
dotnet add package Sorcha.Cryptography
dotnet add package Sorcha.Aspire

Using Visual Studio

1. Create a new .NET Console Application
2. Right-click project → Manage NuGet Packages
3. Search for "Sorcha.Core" and install
4. Install additional packages as needed (Sorcha.Cryptography, Sorcha.Aspire)

Verify Installation

dotnet list package | grep Sorcha

You should see the Sorcha packages listed with their versions.

Your First Blueprint

Let's create a simple data flow that reads data from a source, validates it, and writes it to a destination with cryptographic signing.

Step 1: Create the Blueprint

using Sorcha.Core;
using Sorcha.Cryptography;

var blueprint = Blueprint.Create("HelloSorcha")
    // Define your data source
    .AddSource<string>(config => config
        .FromConsole())

    // Transform the data
    .Transform<string, string>(data =>
        data.ToUpper())

    // Sign the data
    .Sign(signature => signature
        .WithAlgorithm(CryptoAlgorithm.ED25519)
        .UsingKey("my-signing-key"))

    // Output to destination
    .ToDestination<string>(config => config
        .ToConsole())

    .Build();

Step 2: Execute the Blueprint

// Execute the blueprint
await blueprint.ExecuteAsync();

Console.WriteLine("Blueprint executed successfully!");

Step 3: Run Your Application

dotnet run

Enter some text, and you'll see it transformed to uppercase and cryptographically signed before being output.

Understanding Blueprints

Blueprints are the core concept in Sorcha. They define the entire data flow pipeline from source to destination, including all transformations, validations, and security measures.

Blueprint Lifecycle

1. Definition: Create blueprint using fluent API
2. Validation: Sorcha validates the blueprint structure
3. Compilation: Blueprint is compiled into executable code
4. Execution: Data flows through the defined pipeline
5. Monitoring: Track execution with built-in observability

Blueprint Components

• Sources: Where data enters the pipeline
• Transforms: Operations that modify data
• Validators: Ensure data meets requirements
• Routers: Direct data to different destinations
• Destinations: Where data exits the pipeline
• Security: Cryptographic operations and access control

Data Sources

Sorcha supports multiple data source types for receiving data into your pipeline.

Built-in Sources

.AddSource<MyData>(config => config
    .FromQueue("my-queue")
    .WithBatchSize(100)
    .WithPolling(TimeSpan.FromSeconds(5)))

.AddSource<CustomerData>(config => config
    .FromDatabase("CustomerDB")
    .WithQuery("SELECT * FROM Customers WHERE UpdatedAt > @lastRun")
    .WithPolling(TimeSpan.FromMinutes(15)))

.AddSource<ApiResponse>(config => config
    .FromHttpEndpoint("https://api.example.com/data")
    .WithAuthentication(auth => auth.BearerToken(token))
    .WithPolling(TimeSpan.FromMinutes(10)))

.AddSource<FileData>(config => config
    .FromFileSystem("/data/incoming")
    .WithPattern("*.json")
    .WatchForChanges())

Cryptographic Security

Sorcha provides built-in cryptographic capabilities to secure your data flows.

Digital Signatures

Sign data to ensure authenticity and detect tampering:

.Sign(signature => signature
    .WithAlgorithm(CryptoAlgorithm.ED25519)  // or NISTP256, RSA4096
    .UsingKey("my-signing-key")
    .IncludeTimestamp()
    .IncludeNonce())

Key Management

Generate and manage cryptographic keys:

using Sorcha.Cryptography;

// Generate a new key
var keyManager = new KeyManager();
var key = await keyManager.GenerateKeyAsync(
    algorithm: CryptoAlgorithm.ED25519,
    keyId: "my-signing-key");

// Store securely
await keyManager.StoreKeyAsync(key,
    storageType: KeyStorageType.AzureKeyVault);

Signature Verification

Verify signatures before processing data:

.Validate<SignedData>(validation => validation
    .VerifySignature()
    .WithPublicKey("partner-public-key")
    .RejectIfInvalid())

Supported Algorithms

Deployment

Sorcha blueprints can be deployed in various environments, from local development to cloud-native production scenarios.

Local Development

dotnet run --project MyDataFlow.csproj

Docker Deployment

# Build container
docker build -t my-dataflow:latest .

# Run container
docker run -d \
  --name my-dataflow \
  -e SORCHA_CONFIG=/config/blueprint.json \
  -v ./config:/config \
  my-dataflow:latest

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sorcha-blueprint
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sorcha-blueprint
  template:
    metadata:
      labels:
        app: sorcha-blueprint
    spec:
      containers:
      - name: blueprint-executor
        image: my-dataflow:latest
        env:
        - name: ASPNETCORE_ENVIRONMENT
          value: Production
        - name: SORCHA_BLUEPRINT_PATH
          value: /config/blueprint.json
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "512Mi"
            cpu: "1000m"

.NET Aspire Deployment

Deploy with .NET Aspire for cloud-native orchestration:

var builder = DistributedApplication.CreateBuilder(args);

var blueprint = builder.AddProject<Projects.MyDataFlow>("dataflow")
    .WithReplicas(3)
    .WithEnvironment("SORCHA_MODE", "production");

builder.Build().Run();

API Reference

Complete API documentation is available in the GitHub repository.

Core Namespaces

• Sorcha.Core - Blueprint engine and execution
• Sorcha.Cryptography - Cryptographic operations
• Sorcha.Aspire - Cloud-native integration
• Sorcha.Schema - Schema management
• Sorcha.Gateway - YARP-based API gateway

Common Methods

Next Steps