We talk to Nextflow developer Ben Sherman about some exciting new features coming soon to Nextflow: a new CLI v2 interface; submission of job arrays and intermediate file cleanup during runs.
In this News and Views episode, Phil Ewels talks to Ben Sherman about upcoming features in core Nextflow development: a new CLI v2 interface; submission of job arrays and intermediate file cleanup during runs.
CLI v2
- The new command-line interface will be invoked with the
nfcommand - It will co-exist with the current
nextflowCLI, at least for a while - Expect argument / option parsing more in line with linux norms
- No longer a single-dash for core flags and double-dash for pipeline parameters
- Single dash for short flags, double for long-form flags
- Core and pipeline options separated with
--
See the pull-request here: Add CLI v2 #3600
Job Arrays
- Soon you’ll be able to submit job arrays to HPC and cloud
- Group the submission of any number (10s-1000s) of jobs
- Still independent jobs, but submitted in one go
See the pull-request here: Implement job array using TaskRun as carrier #3905
Intermediate file cleanup
- New ability to track which intermediate files are no longer needed
- Allows files to be deleted from the work directory during a run
- Reduces the peak storage required for large pipeline runs
See the pull-request here: Add support for temporary output paths #3818