5. Customizing the template¶
As a developer, you may want more flexibility when developing your pipelines.
While the nf-core template comes with specific standards, many of these can be altered to better suit your needs.
Here, you will learn how to use the nf-core tooling to customize different parts of the nf-core template and nf-core components without compromising their integration into the nf-core ecosystem.
5.1 Patching modules¶
Although modules are written to be flexible you may want to modify them to better fit your purpose. While you are free to edit modules locally, any changes
The nf-core lint
command will help manage nf-core components and test that they match the remote source they came from.
For example, if you modify an nf-core module, it will no longer match the remote and a linting test of this module will fail.
Exercise
Edit the fastp
module by adding an ed
to the end of reads_fail
. Check to see if your change has caused the linting test to fail.
modules/nf-core/fastp/main.nf | |
---|---|
Running nf-core lint
after you have modified an nf-core module will cause it to throw an error.
╭─ [✗] 1 Module Test Failed ────────────────────────────────────────────────────────────────╮
│ ╷ ╷ │
│ Module name │ File path │ Test message │
│╶─────────────┼───────────────────────────────┼────────────────────────────────────────────│
│ fastp │ modules/nf-core/fastp/main.nf │ Local copy of module does not match remote │
│ ╵ ╵ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
Changing a module does not mean you can't continue to use that module.
The nf-core modules patch
command allows you keep using the nf-core component without needing to make it into a local
module for tests to pass. Instead, nf-core modules patch
command creates a diff
file that will keep track of the changes you made. If you subsequently update the module using the nf-core tooling, the diff
file will be retained. If any subsequent changes to the module conflict with your diff
file, you will be prompted to resolve the conflicts.
The prompt can be followed to patch the fastp
module.
A patch file is created in the fastp module directory
...
INFO 'modules/nf-core/fastp/tests/main.nf.test.snap' is unchanged
INFO 'modules/nf-core/fastp/tests/tags.yml' is unchanged
INFO 'modules/nf-core/fastp/tests/nextflow.config' is unchanged
INFO 'modules/nf-core/fastp/tests/main.nf.test' is unchanged
INFO Patch file of 'modules/nf-core/fastp' written to 'modules/nf-core/fastp/fastp.diff'
Exercise
Patch the fastp
module to fix the linting error.
5.2 Ignoring linting tests¶
The linting tests in the nf-core template are primarily designed for pipelines that are shared as a part of the nf-core project and you may want to ignore certain linting tests if they are not required.
The .nf-core.yml
file can be modified to choose which linting tests you would like to skip.
For example, by default, there is a linting test that checks if the CODE_OF_CONDUCT.md
matches the template. If the CODE_OF_CONDUCT.md
has been modified the linting test will fail.
╭─ [✗] 1 Pipeline Test Failed ────────────────────────────────────╮
│ │
│ files_unchanged: CODE_OF_CONDUCT.md does not match the template │
│ │
╰─────────────────────────────────────────────────────────────────╯
If you were to alter the CODE_OF_CONDUCT.md
the nf-core lint
command will suggest a command to fix this warning.
Tip: Some of these linting errors can automatically be resolved with the following command:
nf-core lint --fix files_unchanged
However, if you wish to remove or modify this file you would need to ignore this test. To do this, you would need to list the CODE_OF_CONDUCT.md
as a files_unchanged:
in the .nf-core.yml
file.
If you run nf-core lint
again, you would see that the test is now ignored and there are no more failed tests.
╭─ [?] 1 Pipeline Test Ignored ────────────────────────────────────────╮
│ │
│ files_unchanged: File ignored due to lint config: CODE_OF_CONDUCT.md │
│ │
╰──────────────────────────────────────────────────────────────────────╯
Exercise
Edit the CODE_OF_CONDUCT.md
in your pipeline repository (e.g., add another bullet point). Use the nf-core lint
command to see if it passes or fails. Add the CODE_OF_CONDUCT.md
as a files_unchanged:
in the .nf-core.yml
file and lint your pipeline again to show that the test has been ignored.
A full list of checks, descriptions of how they work, and how they can be customized can be found on the tools documentation website.
Bonus Exercise
Modify the .nf-core.yml
file to prevent pipeline_todos
from showing as warnings in your lint tests.
5.3 Custom remote modules¶
As an individual or group, you may want to keep your own library of modules and/or subworkflows.
The nf-core modules command comes with two flags for specifying a custom remote:
--git-remote <git remote url>
: Specifies the repository from which the modules should be fetched as a git URL.--branch <branch name>
: Specifies the branch from which the modules should be fetched.
Note that a custom remote must follow a similar directory structure to that of nf-core/modules
for the nf-core commands to work properly.
The directory where modules are installed will be prompted or obtained from org_path
in the .nf-core.yml
file, if it is available. If a module was located at modules/my-folder/TOOL/SUBTOOL
your .nf-core.yml
should have:
The modules commands will, during initialization, try to pull changes from the remote repositories. If you want to disable this, for example, due to performance reasons, you can use the flag --no-pull
. Commands will still need to clone repositories that have previously not been used.
Private modules repositories
In order to browse private repositories you have to configure the GitHub CLI auth and provide your credentials with the command below.
5.4 TEMPLATE
syncs¶
The template evolves as the ecosystem evolves.
To keep nf-core pipelines up to date with improvements in the main template, you can use a method of synchronization with the TEMPLATE
branch.
To sync the template, you first need to commit and push your changes to GitHub. The nf-core sync
command can then be used to update the TEMPLATE
branch with the latest version of the nf-core template, so that these updates can be synchronized with the pipeline. It is run automatically for all pipelines whenever a new release of nf-core/tools (and the included template) is made.
The tooling merges updates suggesting a git command
Note
For a newly created pipeline the TEMPLATE
branch doesn't need to be synced.
5.4.1 Changes to the template structure¶
Occasionally, the structure of the nf-core pipeline template is updated during a new release of the nf-core tooling. Most of the time these changes are minor. However, sometimes, larger structural changes are adopted to align with changes in the wider ecosystem.
For example, as of nf-core tools 2.13, the groovy code that once lived in the lib
folder has been moved to subworkflows/
. Moving this code has made it easier to find, modify, and test the code. Importantly, it's modular nature is paving the way for a more flexible template in the future.
Note
The TEMPLATE
branch is essential for adopting these changes.
5.5 Update minimum Nextflow version¶
The template also includes a minimum Nextflow version that is required for the pipeline to run. You can also change the required version of Nextflow using the nf-core bump-version
command. These versions can also be used during GitHub Actions to test your pipeline with upper and lower version limits.
Exercise
Update the minimum version of Nextflow required to run your pipeline to 23.04.0
and push the changes to GitHub.
Tagged versions
Creating a tagged version release allows you to execute specific versions of your pipeline using the revision flag.