Decaf Blog Data pipelines, open-source tooling, reverse engineering, and systems notes

Argo Workflows v4.1.0 - Device Claims and Scheduler Changes

5 min read

Argo Workflows v4.1.0 shipped on 11 August 2026 as a stable tag, not a release candidate. The main operator facing change is first class Kubernetes Dynamic Resource Allocation: workflow pods can request devices through a new resourceClaims field on the workflow spec and on templates. Data teams that run GPU or accelerator bound batch jobs can attach claims without stuffing the entire list into podSpecPatch.

The full release notes and downloads are on the GitHub release page. The tag page itself is a short install sheet. The project points at the changelog and the upgrading guide for the real delta.

Device claims, pod budgets, and pending pods

resourceClaims exists at workflow scope and at template scope. Template level claims replace the workflow list as a whole rather than merging with it. podSpecPatch still applies after that and merges by claim name. A patch that moves a claim from one source to the other has to set the old source to null, or the API server rejects the pod. Each entry names an existing ResourceClaim or a ResourceClaimTemplate in the workflow namespace. A container asks for one by name through resources.claims.

Declaring resourceClaims on a template that creates no pod is rejected at submit time. Steps, DAG, and Suspend templates orchestrate other templates. HTTP and Plugin templates run on the shared agent pod. Argo forwards the references. It does not allocate devices and it does not manage claim lifetime. The cluster must have the DynamicResourceAllocation feature gate enabled and a DRA driver installed.

A related field, podResources, sets a single resource budget for every container in the workflow pod (main, init, wait, and sidecars). It needs the PodLevelResources feature gate. If that gate is off, the API server strips the field and the controller emits a PodLevelResourcesDropped warning.

Templates also gain pendingTimeout. Unlike timeout, which covers the whole node lifecycle, this field caps time spent in Pending. When it fires, the node is marked Failed and the pending pod is deleted. Enforcement uses the controller’s last observed pod state, so it is approximate: a pod that starts running at almost exactly the deadline can still be failed and deleted.

Artifact writes and S3 addressing

Artifact drivers can save from an io.Reader through a new SaveStream method instead of only from a local file. Azure and HTTP or Artifactory stream the reader to the destination. S3, GCS, OSS, and HDFS still buffer to a temp file and reuse the existing save path, so bucket creation, key handling, and retries stay as they were. Artifact plugins get an optional client streaming SaveStream gRPC method plus GetCapabilities. Plugins that skip those methods keep the old Save path.

S3 upload throughput can be tuned with ARTIFACT_S3_UPLOAD_THREADS and ARTIFACT_S3_UPLOAD_PART_SIZE_MIB. Defaults remain 4 threads and a dynamic part size, often 16MiB for files up to 156GiB. A new addressingStyle field on S3 buckets accepts "" (auto), "path", or "virtual-hosted". The last value matters for S3 compatible stores that only speak virtual hosted style, including log streaming and artifact browsing.

Controller cache and live parallelism

The workflow controller now strips metadata.managedFields from objects before they enter informer caches. The project estimates that field can be about 20 percent of cached object memory, and at scale informer objects dominate controller RAM. Cluster objects are unchanged. This is internal cache policy, not a CRD change.

namespaceParallelism in the controller ConfigMap now reloads without a restart, matching existing parallelism behavior. Namespaces without an explicit parallelism label follow the live default.

The INFORMER_WRITE_BACK environment variable is gone. It used to choose between writing workflow updates back into the informer cache or sleeping for one second after persist. Other mechanisms now prevent reprocessing. If that variable is still set, drop it.

An opt in initlessPod mode (beta, off by default) removes the argoexec init container. The argoexec binary is mounted into main through a Kubernetes image volume. A new supervisor container takes over staging, input artifact download, readiness, and the work wait used to do after main exits. Enable with initlessPod.enabled: true in the controller ConfigMap. In flight pods keep their original layout. The project says this mode may still change in later minor releases before it is marked stable.

The controller also emits lock telemetry: locks_taken_total, locks_held, and locks_pending, labelled by type (mutex or semaphore), storage, lock_name, and namespace. That is the useful signal when a DAG sits blocked on a sync lock rather than on compute.

Upgrade notes

Read the upgrading guide before you roll this tag. argo archive get, delete, resubmit, and retry now accept a workflow name or a UID. A UUID shaped argument is treated as a UID. Anything else is a name in the selected namespace. Duplicate names fail and list matching UIDs. Force the parse with --uid or --name. Existing UID scripts keep working.

The GitHub tag is a stable v4.1.0 release. Search existing issues after you upgrade. Controller and server install from the GitHub release page via install.yaml:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v4.1.0/install.yaml

A new argo-workflows-crdinstaller image ships with the tag for installers that cannot server side apply the full CRDs themselves, including air gapped clusters.

Where to get it