Distributed Execution Models: A Comparative Analysis

Modern software systems increasingly depend on distributed execution to handle scale, reliability, and fault tolerance. Yet the choice of execution model is often made implicitly - inherited from existing infrastructure or selected based on familiarity rather than analysis. This paper examines five execution architectures: queue-based, event stream, work pool, orchestrated, and direct execution. For each, we evaluate reliability, latency, fault tolerance, operational complexity, and appropriate use cases. We provide a decision framework for selecting an execution model based on system requirements rather than convention.

1. Introduction

Execution is the least visible part of most software systems and the most consequential when it fails. A delayed payment, a dropped message, a stalled workflow - these failures trace back to execution architecture decisions made early in a system’s design.

Most teams adopt an execution model based on what their cloud provider offers, what their framework defaults to, or what they used at their previous company. The decision is rarely revisited. This paper argues that execution model selection should be a deliberate architectural choice, evaluated against specific system requirements.

2. The Five Models

block-beta
    columns 3
    block:Q["Queue-Based"]
        columns 1
        Q1["RabbitMQ, SQS"]
        Q2["Async background jobs"]
    end
    block:E["Event Stream"]
        columns 1
        E1["Kafka, Pulsar"]
        E2["Ordered, replayable"]
    end
    block:W["Work Pool"]
        columns 1
        W1["gRPC pools"]
        W2["Stateless, predictable"]
    end
    block:O["Orchestrated"]
        columns 1
        O1["Temporal, Step Functions"]
        O2["Multi-step workflows"]
    end
    block:D["Direct"]
        columns 1
        D1["In-process"]
        D2["Zero overhead"]
    end

2.1 Queue-Based Execution

A producer places work into a queue. One or more consumers pull work from the queue and process it. The queue persists until work is acknowledged as complete.

Mechanism: FIFO or priority queues. Message brokers (RabbitMQ, SQS, Redis). At-least-once or exactly-once delivery semantics.

Strengths: Simple. Well-understood. Natural load balancing across consumers. Work survives consumer failure.

Weaknesses: No execution order guarantees without additional coordination. Exactly-once semantics are difficult and expensive. Queue depth can hide systemic slowdowns.

Best for: Asynchronous background jobs. Email delivery. Image processing. Work where ordering does not matter.

Example: Foundry uses queue-based execution for user-triggered background tasks. A user deploys a configuration change. The request enters a queue. Workers apply changes. Failures retry automatically.

2.2 Event Stream Execution

Events are published to an ordered, append-only log. Consumers read events sequentially and maintain their position. Multiple consumers can process the same event for different purposes.

Mechanism: Kafka, Pulsar, NATS JetStream. Append-only logs. Consumer groups with offset tracking.

Strengths: Ordered execution. Replayable. Multiple independent consumers. High throughput. Decouples producers and consumers completely.

Weaknesses: Operational complexity. Requires ZooKeeper or equivalent for coordination. Consumer lag must be monitored. Ordering is partition-level, not global.

Best for: Event-driven architectures. Audit logs. Systems where multiple services need the same event. High-throughput data pipelines.

Example: Relay uses event stream execution for signaling events. Connection requests, ICE candidate exchanges, and disconnect events flow through ordered streams. Multiple services consume the same events for logging, analytics, and billing.

2.3 Work Pool Execution

A pool of workers listens for work. Work is dispatched to the first available worker. Workers are stateless and interchangeable. The pool can grow or shrink based on load.

Mechanism: Worker pools behind a dispatcher. gRPC connection pools. HTTP connection pools. Load balancers distributing across identical instances.

Strengths: Predictable latency under known load. Simple scaling (add more workers). No message broker dependency. Good for CPU-bound work.

Weaknesses: Work is lost if the dispatcher fails without persistence. Workers are stateless, so stateful work requires external storage. Pool sizing is guesswork without load testing.

Best for: API request handling. Real-time computation. Work with predictable duration. Systems where latency matters more than durability.

Example: Covenant uses work pool execution for signature processing. Signature requests arrive at an API endpoint. A pool of workers processes them. Latency is predictable. Work is idempotent - if a worker fails, the client retries.

2.4 Orchestrated Execution

A central orchestrator coordinates execution across multiple services. It tracks state, handles retries, manages timeouts, and compensates when steps fail.

Mechanism: Temporal, Cadence, AWS Step Functions, Camunda. Directed acyclic graphs of tasks. State persisted externally.

Strengths: Handles complex workflows. Built-in retry and timeout logic. Visibility into execution state. Compensation for partial failures. Long-running workflows supported natively.

Weaknesses: Central orchestrator is a single point of failure (mitigated by persistence). Operational complexity. Learning curve. Overkill for simple work.

Best for: Multi-step business processes. Order fulfillment. Approval workflows. Processes spanning multiple services with compensation requirements.

Example: Dominion uses orchestrated execution for infrastructure provisioning. A single request triggers multiple steps: validate configuration, provision compute, configure networking, deploy software, verify health. Each step can fail. The orchestrator retries or compensates. The workflow can run for minutes or hours.

2.5 Direct Execution

The caller executes work directly in the same process or thread. No queuing. No dispatching. No coordination. The simplest possible model.

Mechanism: Synchronous function calls. In-process execution. Thread-local work.

Strengths: Zero operational overhead. Zero infrastructure dependencies. Lowest possible latency. Simplest to debug.

Weaknesses: No fault tolerance. If the process dies, work is lost. No retry. No load balancing. Caller is blocked until work completes. Cannot scale beyond one process.

Best for: Work that must happen immediately. Work where failure is acceptable. Prototypes. Single-user systems. CLI tools.

Example: Focus, the development environment, uses direct execution for file operations. Save a file. It writes to disk. No queue. No stream. Direct. The trade-off is accepted because the user is present and can retry manually.

3. Decision Framework

flowchart TD
    R{"What matters most?"}
    R -->|Survive failure| QE["Queue or Event Stream"]
    R -->|Must be in order| EO["Event Stream or Orchestrated"]
    R -->|Latency under 50ms| DW["Direct or Work Pool"]
    R -->|Multi-step + compensation| OR["Orchestrated"]
    R -->|High throughput| ES["Event Stream"]
    R -->|Simple, no coordination| DI["Direct"]
    R -->|Occasional background| QU["Queue"]

4. Anti-Patterns

block-beta
    columns 2
    block:QORDER["Queues When Order Matters"]
        columns 1
        QA1["Race conditions"]
        QA2["Incorrect state"]
        QA3["Use event stream instead"]
    end
    block:SIMPLE["Event Streams for Simple Work"]
        columns 1
        QA4["Operational waste"]
        QA5["Kafka for 50 jobs/day"]
        QA6["Queue is sufficient"]
    end
    block:STATELESS["Orchestrators for Stateless Work"]
        columns 1
        QA7["Unnecessary latency"]
        QA8["Added complexity"]
        QA9["Direct is correct"]
    end
    block:DURABLE["Direct Execution for Durable Work"]
        columns 1
        QA10["Data loss on crash"]
        QA11["No fault tolerance"]
        QA12["Use queue or stream"]
    end

4.1 Queues When Order Matters

Using a queue for ordered work leads to race conditions and incorrect state. If order matters, use an event stream or orchestrator.

4.2 Event Streams for Simple Work

Deploying Kafka for a system that processes 50 background jobs per day is operational waste. A queue is simpler and sufficient.

4.3 Orchestrators for Stateless Work

Wrapping a stateless API call in an orchestration workflow adds latency, cost, and complexity with no benefit. Direct or work pool execution is correct.

4.4 Direct Execution for Durable Work

Processing a payment synchronously in-process guarantees data loss on crash. Payments belong in queues or streams.

5. The Foundry Execution Model

Foundry, the execution infrastructure developed by CODECX (a TELOSIS brand), implements a hybrid model. Users define work. Foundry routes it to the appropriate execution model based on the work’s characteristics.

• Background work with durability requirements goes to queues.
• Real-time signaling events go to event streams.
• API request handling uses work pools.
• Long-running provisioning workflows use orchestration.
• CLI commands use direct execution.

The routing logic is transparent to the user. They declare what they want done. Foundry decides how to execute it.

6. Conclusion

There is no single correct execution model. There are only models that fit the work and models that do not. The mistake most systems make is not choosing the wrong model - it is never making a deliberate choice at all.

The right approach is to evaluate each type of work against the decision framework above, select the simplest model that satisfies the requirements, and document the choice. When requirements change, the model can change. But the model should always be chosen, never inherited.

References

1. Kleppmann, M. Designing Data-Intensive Applications. O’Reilly, 2017.
2. Kreps, J. The Log: What every software engineer should know about real-time data’s unifying abstraction. LinkedIn Engineering, 2013.
3. CODECX Engineering. Why We Chose WebRTC Over WebSockets for Relay. CODECX Journal, 2026.
4. TELOSIS Research. Self-Hosting Is Not a Feature - It Is Infrastructure. TELOSIS-RP-2026-001, 2026.

TELOSIS Research. (2026). Distributed Execution Models: A Comparative Analysis. TELOSIS-RP-2026-002.

TELOSIS Research. "Distributed Execution Models: A Comparative Analysis." TELOSIS-RP-2026-002, 2026.

@techreport{telosis_rp_2026_002,
  author = {TELOSIS Research},
  title = {Distributed Execution Models: A Comparative Analysis},
  number = {TELOSIS-RP-2026-002},
  year = {2026}
}