Long-running computational tasks, ranging from batch data processing to multi-day simulations, present operational challenges that differ fundamentally from request-response workloads. They cannot be served by standard autoscaling groups, they do not fit neatly into container orchestration patterns designed for short-lived processes, and they demand different approaches to fault tolerance, resource allocation, and cost management. This paper examines three distributed execution models that have proven effective for long-running workloads: work queue architectures, actor-based processing, and stateful stream processing. For each model, we document the operational characteristics, the failure modes, and the conditions under which the model outperforms alternatives. We find that the choice of execution model should be driven by task characteristics, namely statefulness, idempotency, and duration, rather than by organizational familiarity or available tooling.
The majority of distributed systems literature and tooling focuses on request-response workloads: small tasks that arrive, are processed, and return a result within seconds or minutes. This focus is appropriate for web services, API gateways, and real-time data processing, where latency and throughput are the primary operational metrics.
A different class of workload exists alongside these: long-running tasks that take minutes, hours, or days to complete. These include batch data transformations, machine learning training runs, scientific simulations, large-scale code migrations, and multi-step data pipelines. These workloads have different operational requirements, different failure modes, and different cost profiles, yet they are frequently forced into execution models designed for short-lived processes.
This paper examines three distributed execution models that have proven effective for long-running workloads, drawing on operational experience from production systems at three organizations. For each model, we document the operational characteristics that matter most: how the model handles failures, how it manages resource allocation, how it provides visibility into task progress, and how it scales.
The work queue model decomposes a large task into independent units of work, each of which can be processed by any available worker. The work units are placed in a durable queue, and workers pull work from the queue, process it, and acknowledge completion.
Work queues provide natural load balancing: workers that finish quickly pull more work, while workers that encounter difficulty slow down without affecting other workers. This self-balancing behavior is particularly valuable for long-running tasks with variable per-unit processing times.
The durability of the queue provides fault tolerance: if a worker fails mid-processing, the unacknowledged work unit is returned to the queue and can be retried by another worker. This requires that work units be idempotent, meaning that processing the same unit twice produces the same result as processing it once.
The primary failure mode in work queue systems is poison pills: work units that cause workers to crash or hang indefinitely. A robust work queue implementation requires a dead letter mechanism that moves repeatedly failing work units out of the active queue and into a monitoring queue where the failure can be investigated.
The secondary failure mode is queue saturation: when the rate of work production exceeds the rate of work consumption, the queue grows without bound. Effective work queue systems require backpressure mechanisms that either throttle work production or provision additional workers automatically.
Work queues are most effective when the task can be decomposed into independent units, when each unit is idempotent, and when the units have similar but not identical processing times. They are less effective when units have dependencies on each other, when processing requires maintaining state across units, or when units vary dramatically in processing time, causing stragglers that delay overall completion.
The actor model assigns each task or sub-task to a named actor, a self-contained processing entity that maintains its own state and communicates with other actors exclusively through message passing. Unlike work queue workers, actors are persistent: they maintain state across multiple messages and can be restarted on any available node, with their state reconstructed from message history.
Actor-based systems excel at tasks that require maintaining state across multiple processing steps. An actor representing a multi-stage data transformation, for example, can process each stage sequentially, maintaining the intermediate state in memory or in a lightweight persistent store, without the overhead of serializing and deserializing state between steps.
The actor model also provides natural task isolation: because each actor has its own state and processes messages sequentially, there are no concurrent access conflicts within an actor. Concurrency is achieved through parallelism across actors, not within them.
The primary failure mode is actor state loss. If an actor’s state is not durable and the actor’s host node fails, the actor’s progress is lost. The operational cost of durable actor state is significant: every state transition must be persisted before the actor can process the next message, creating a serialization bottleneck that limits throughput.
The secondary failure mode is message ordering. The actor model assumes that messages between any two actors arrive in order. In distributed systems, guaranteeing message ordering requires careful design of the messaging infrastructure, and violations of this assumption can cause subtle data corruption that is difficult to detect and reproduce.
Actor-based processing is most effective when tasks require persistent state, when the number of concurrent tasks is large but the state per task is small, and when task isolation is important for correctness. It is less effective for purely stateless tasks, where the overhead of actor infrastructure provides no benefit over simpler work queue models.
Stateful stream processing treats long-running tasks as continuous data flows, where data is processed incrementally as it arrives, state is maintained across the processing pipeline, and results are emitted continuously rather than at the end of a batch.
Stream processing provides the lowest latency for tasks that produce results incrementally: rather than waiting for a batch to complete, consumers can process results as soon as they are emitted. This is valuable for tasks where partial results are useful, such as real-time analytics dashboards or progressive data enrichment pipelines.
The state management model in stream processing is typically windowed: state is maintained for a defined time window or count of events, and is discarded after the window closes. This provides a natural mechanism for managing memory consumption in long-running tasks that would otherwise accumulate unbounded state.
The primary failure mode is state inconsistency during failover. When a stream processing node fails, in-flight state must be reconstructed from durable storage. The time required for state reconstruction determines the recovery time objective, and for stateful processing with large state stores, this recovery time can be significant.
The secondary failure mode is backpressure propagation. In a multi-stage stream processing pipeline, a slow downstream stage can cause backpressure that propagates upstream, slowing or halting the entire pipeline. Managing backpressure requires careful capacity planning and monitoring across all stages of the pipeline.
Stateful stream processing is most effective when tasks produce results incrementally, when the data source is continuous or high-volume, and when low latency between input and output is important. It is less effective for tasks that are naturally batch-oriented, where the overhead of stream processing infrastructure provides no latency benefit.
The choice between these three models should be driven by task characteristics, not by organizational familiarity or available tooling. We propose three primary selection criteria:
Statefulness: Does the task require maintaining state across processing steps? If no, use a work queue. If yes, the choice depends on the state lifecycle.
Idempotency: Can each processing step be safely retried without side effects? If yes, a work queue is the simplest effective model. If no, actor-based or stream processing models provide the state isolation required for correctness.
Duration and latency requirements: Does the task need to produce results incrementally, or can it produce all results at completion? If incremental results are valuable, stream processing is appropriate. If not, the choice between work queue and actor models depends on the statefulness and idempotency criteria above.
These criteria are necessary but not sufficient. Operational considerations, including team expertise, existing infrastructure, and monitoring capabilities, will influence the practical choice. The goal is to ensure that the operational considerations are secondary to the task characteristics, not primary.
Long-running tasks are a distinct workload category that requires deliberate selection of execution model. The three models examined in this paper, work queues, actor-based processing, and stateful stream processing, each have specific operational profiles that make them more or less suitable for different task characteristics. The most common failure mode is choosing an execution model based on available tooling rather than task requirements, resulting in unnecessary complexity, poor fault tolerance, or both.
The selection criteria we propose, statefulness, idempotency, and duration, provide a structured framework for matching task characteristics to execution models. Organizations that apply these criteria early in the design process avoid the costly rework that results from discovering, late in development, that the chosen execution model does not match the task’s fundamental requirements.
TELOSIS Research. (2026). Distributed Execution Models for Long-Running Tasks. TELOSIS-RP-2026-003.
© 2026 TELOSIS Research. All rights reserved.
This paper is published by TELOSIS Research. Reproduction in whole or in part without written permission is prohibited.