Comprehensive Plugin-Based Monitoring of Nexflow Workflow Executions

arXiv:2603.28783v1 Announce Type: new
Abstract: Nextflow is a workflow management system commonly used in fields like bioinformatics and earth observation. It coordinates distributed data processing of various tools as an acyclic sequence of tasks while using, containerization (e.g., Docker), orchestration (e.g., Kubernetes), or batch processing (e.g., SLURM). Monitoring such workflow executions can be challenging but aids performance analysis, debugging, and data provenance. Besides Nexflow’s basic built-in monitoring, the wf-commons tool for creating wf-instances is widely regarded as the standard in the Nextflow community. The monitoring plugin we develpoed provides a more detailed and flexible alternative compatible with wf-instances while removing the need for a custom Nextflow fork by using Nextflow’s plug-in mechanism (version 21.10), optional direct .jar file changes of static artifacts without recompilation and allows online monitoring during execution.