Data processing with Argo Workflows: CAPA practice questions

Argo Workflows suits batch and data processing because each task is an ordinary container and the controller handles ordering, fan-out, retries and clean-up. A typical job lists its inputs, processes them in parallel and then merges the results, and each of those stages has a matching feature.

Fan-out uses loops. withItems runs a task once for every entry of a literal list, withSequence runs it for a range of numbers given with count or start and end, and withParam runs it for every item of a JSON array. The last form is the one used for dynamic jobs: an earlier step lists the files or partitions to process and prints them as JSON, and the next step loops over {{steps.list.outputs.result}}. Loop iterations run in parallel. A later step can collect the outputs of all iterations, and artifacts move the actual data through the artifact repository instead of through parameters.

Parallelism needs limits so that a large fan-out does not flood the cluster. parallelism on the workflow caps the Pods it runs at once, and parallelism on a template caps the tasks it starts. The controller can also cap the number of running workflows across the cluster or per namespace. Mutexes and semaphores in the synchronization block go further: a mutex allows a single holder at a time, and a semaphore, sized in a ConfigMap, allows a configured number, at either workflow or template level.

Long jobs fail in many small ways, so retries matter. retryStrategy sets a limit, a retryPolicy of OnFailure (the default), OnError, OnTransientError or Always, and an exponential backoff with a duration, a factor and a maximum. A failed run can be retried with argo retry, which reruns only the nodes that did not succeed. Volumes give steps a shared working directory: volumeClaimTemplates creates a claim for the run and removes it at the end.

Running thousands of Pods also leaves objects behind. podGC deletes completed Pods, ttlStrategy deletes finished workflows after a delay, and CronWorkflow history limits trim scheduled runs, which together keep the API server and the controller from slowing down.