Nextflow is another workflow management system, and is (along with Snakemake) one of the most common such systems within the academic community. While Nextflow is based on the Groovy language, you don’t need to know how to code Groovy to be able to write good Nextflow workflows, just like you don’t need to know Python to write good Snakemake workflows.

Nextflow is built from the ground-up to be portable, scalable, reproducible and usable in a platform-agnostic sense. This means that any pipeline you write in Nextflow can be run locally on your laptop, a computer cluster or a cloud service. You can also define the compute environment in which each task is carried out, just like in Snakemake. Nextflow has a large community centered around it, including the nf-core curated collection of high quality pipelines used by e.g. the National Genomics Infrastructure.

Differences between Nextflow and Snakemake

There are several major differences between Snakemake and Nextflow, both in how they work and in their underlying philosophies, summarised by the following table:

	Snakemake	Nextflow
Language	Python	Groovy
Data	Everything is a file	Can use both files and values
Execution	Working directory	Each job in its own directory
Philosophy	“Pull”	“Push”
Dry runs	Yes	No
Track code changes	No	Yes

Starting from the top with the most obvious and perhaps superficial difference: language. Snakemake is, as you know, based on Python, whereas Nextflow is based on Groovy (which is a superset of the Java language). You don’t need to know Groovy to be able to use Nextflow, though, just as you don’t really need to know Python to use Snakemake.

Moving on to data: everything in Snakemake is a file, whereas Nextflow works on both files and arbitrary values and variables, stored in so-called channels. Channels are highly flexible streams of data that can be connected to a workflow’s various inputs and outputs. A common use-case is, for example, to define a channel to contain both sample data files and their corresponding sample names, which simplifies a lot of the coding. The flexibility of channels allows you to do many different things in powerful ways, but a downside is that they can be complicated when you first start working with them.

In Snakemake, the entire workflow and each rule is executed in the working directory, while Nextflow executes each individual process (the equivalent of a rule in Snakemake) using an isolated environment in a directory of its own. This greatly simplifies testing and debugging, as you can always go into a process’ directory and see exactly which files it has access to and which code was executed. This general structure means that you need to think less about full paths for all the workflow’s in- and output files, as the locations of all the files are fully taken care of by Nextflow - the only thing you need to care about are the final output file paths.

Snakemake uses a “pull”-philosophy similar to its inspiring predecessor make, meaning that you define a number of rules with inputs and outputs and then ask for the specific result you want, i.e. the final output files (usually defined in an all rule). Snakemake will work backwards from the final outputs you desired and find whatever combination of inputs and rules it needs to give them to you. This means that you always know exactly which files are going to be created and manipulated in all steps of the workflow even before it is executed, which is a nice thing to know. Nextflow works in the opposite way, i.e. with a “push”-philosophy: you define a number of processes with inputs and outputs, and then give the first inputs to Nextflow. It will run the first process using those inputs, pass them to the second process, then the third, and so on until it reaches the final outputs of the workflow. This means you don’t define file paths to each process’ input/output definitions like you do in Snakemake, only which files you want in the end. This can potentially remove some of the pitfalls and issues sometimes seen with e.g. wildcards in Snakemake, but it also front-loads some of the complexity to channel creation instead.

The above means means that Nextflow doesn’t actually know exactly which files are going to be created and manipulated during a run, which is both good and bad. The bad means that you can’t really do dry runs in Nextflow in the same simple manner as in Snakemake (there is, however, something similar called stubs). On the other hand, something good about this is that Nextflow handles variable inputs and outputs very well, making dynamic analyses easy to work with, e.g. processes where you don’t know the exact number of output files.

Lastly, both Snakemake and Nextflow can automatically determine which rules or processes need to be re-run when something has changed, but they do it in slightly different ways. Snakemake only checks if any of the input files are newer than the output files, while Nextflow also tracks code changes. This means that if you update a script that is run on some unchanged data in Nextflow, it will re-run the corresponding process automatically; the same is not true for Snakemake, where you need to specify that you want to re-run the workflow from that specific rule (i.e. using -R <rule>).

The aim of this tutorial

With all that said, both Nextflow and Snakemake are excellent systems for workflow management, and you can do basically everything in either: your choice is very much up to your personal preference and what you think is most important. We suggest that you try both and get a feel for them, and then decide which you like the most. The course material for Nextflow is, with this in mind, not as extensive as that for Snakemake. We have recreated the MRSA workflow from Snakemake in Nextflow and will, through it, give you an overview of how Nextflow does things. The idea is that you should have a rough idea of both Snakemake and Nextflow after the course, so that you may continue in what manner you think suits you the best.

This tutorial depends on files from the course GitHub repo. Take a look at the setup for instructions on how to set it up if you haven’t done so already, then open up a terminal and go to workshop-reproducible-research/tutorials/nextflow and activate your nextflow-env Conda environment.