Toil Adds CWL Loops and Container Resource Accounting

Toil is a workflow engine for batch pipelines, with strong CWL and WDL support. This week’s activity is worth reading because it adds CWL loop execution, injects container resource accounting, and tightens restart coverage for WDL jobs that die on purpose.

The CWL loop commit is the largest change in this window. Most of it lands in src/toil/cwl/cwltoil.py, with coverage added in src/toil/test/cwl/cwlTest.py. The practical result is support for the cwltool:Loop extension inside Toil’s CWL runner when extension support is enabled.

The new path models each loop as Toil work, not as a local interpreter trick. CWLLoop evaluates loopWhen, schedules the embedded tool as a child job, then adds the next loop iteration as a follow on. That matters for operators because loop bodies still participate in the normal Toil graph. They can use the same job store, file store, logging, and retry machinery as other workflow steps.

The implementation also adds CWLLoopAccumulate for output collection. A loop can return the last completed iteration with last_iteration, or collect one array per output with all_iterations. If loopWhen is false before the first body run, Toil now has defined output behavior: null for last output mode and empty arrays for accumulated output mode.

The useful part of a loop runner is not just running the same command again. It is rebinding inputs correctly after each iteration. This change handles LoopInput records with output sources, defaults, valueFrom, linkMerge, and pickValue. That is the difference between a toy counter loop and a loop that can carry state through a real CWL workflow.

There is also a clear safety guard. src/toil/options/cwl.py adds --cwl-loop-iteration-limit, defaulting to 1000. If loopWhen never becomes false, Toil raises a workflow error instead of letting the workflow burn through an unbounded number of tasks. For long scientific or data preparation workflows, that limit is the knob to review before moving an existing loop extension into production.

The tests are useful as documentation. They cover one input, two inputs, zero body runs, accumulated outputs, default values, multi source loop inputs, nested loops, and loop use under scatter through an embedded workflow. They also assert that direct combinations of loop with scatter or when on the same step are rejected. That is a healthy boundary. It keeps the extension expressive without pretending every composition is valid.

The resource accounting commit addresses a common blind spot in workflow stats. When Docker runs work through a daemon, the actual workload may not be visible as a child process of the Toil worker. Host side process accounting can under report CPU and memory for container tasks.

The new shared code in src/toil/lib/interpreter.py injects a POSIX sh monitor into the container command. It samples cgroup CPU and memory files, writes messages under .toil_runtime, and then hands those samples back to ResourceMonitor. The test fixture in src/toil/test/cwl/waste_cpu_memory.cwl is blunt on purpose: it allocates about 1 GiB and burns about 30 seconds of CPU to make the accounting visible.

CWL gets this through a Docker specific command line job wrapper. WDL also uses the shared injection path from src/toil/wdl/wdltoil.py when it is running through the Docker Swarm container path. The same commit touches src/toil/worker.py to pass through interrupted MiniWDL exit codes when it can parse them from the exception message.

This is still accounting, not a billing grade profiler. It samples once per second and depends on cgroup files being available in the container. That is fine for the target problem. Operators mainly need resource reports that are close enough to explain why a task saturated memory or why cluster usage looks larger than Toil previously reported.

The WDL restart test commit is only one test, but it covers a failure mode worth keeping. The new case in src/toil/test/wdl/wdltoil_test.py creates the input file before the first run, then forces worker failure with --badWorker=1.0 and --badWorkerFailInterval=0.01.

That setup removes ambiguity. The workflow does not fail because an input file is missing. It fails because the worker is killed. The restart path then reruns without --badWorker, using the same job store and --restart, and checks both parsed output lines and exported output files.

The export assertion is the important part. A restarted workflow can look successful if it returns JSON, while still failing to materialize the files an operator expects at the end. This test keeps that final handoff in view.

If you use CWL extensions with Toil, test cwltool:Loop with realistic loop counts before raising --cwl-loop-iteration-limit. A loop that is valid in a small fixture can still become expensive when every body run schedules real batch work.

If you depend on Toil stats for cluster sizing, compare Docker task CPU and memory before and after this change. Hidden container work may now show up more accurately, which can change how noisy or memory heavy a workflow appears.

For WDL operations, keep restart tests focused on exported files, not just process exit and JSON shape. The new bad worker case is a useful model because it isolates restart behavior from ordinary input validation errors.