Tools

Knowledge

Please prepare for the course by going through the following materials:

Intro to Conda

Contributors: Erik Fasterius, John Sund, Verena Kutschera

Introduction to conda

Intro to Git

Contributors: Erik Fasterius, John Sund, Verena Kutschera

Introduction to git

Intro to python

Contributors: Paulo Czarnewski

Python (part_1):https://swcarpentry.github.io/python-novice-inflammation/
Python (part_2): http://swcarpentry.github.io/python-novice-gapminder/

Intro to R

Contributors: Paulo Czarnewski

R (part_1): https://swcarpentry.github.io/r-novice-inflammation/
R (part_2): http://swcarpentry.github.io/r-novice-gapminder/

Intro to UNIX

Contributors: Paulo Czarnewski

UNIX (part_1): http://swcarpentry.github.io/shell-novice/
UNIX (part_2): https://carpentries-incubator.github.io/shell-extras/

Miscellaneous Unix setup

From NGSintro

Install tools for SSH & SFTP access

SSH

You need a program to connect to a remote cluster (UPPMAX and NSC). Linux and Mac users already have terminal on their systems. If you are on a Windows system, we recommend MobaXterm. It is recommended that you INSTALL the program and not use the portable version. MobaXterm also has an integrated SFTP file browser.

XQuartz

Mac users will need to download and install XQuartz for X11 forwarding. ie; to forward remotely opened windows to local machine.

SFTP Browser

When you need to transfer data between the remote cluster and your computer, you can use the tools SCP or SFTP through the terminal. Windows users can use the SFTP browser available with MobaXterm. If you prefer a GUI to upload and download files from the remote cluster, we recommend installing FileZilla.

Tools

Please install the following tools:

Conda installation

Contributors: Paulo Czarnewski

During this workshop, you will use conda environments to run the exercises. This is because conda environments allow all users to have the same computing environment, i.e. package versions. This enforces reproducibility for you to run this material without the need to re-install or change your local versions. See and graphical example below:

Conda environments are a self-contained directory that you can use in order to reproduce all your results.

Briefly, you need to:

Install Conda
Download the .yml environment file
Create and activate the environment
Deactivate the environment after running your analyses

You can read more about Conda environments and other important concepts to help you make your research reproducible.

1. Download and install Conda and Mamba

Start by installing Conda. We suggest installing Miniconda3 and NOT Anaconda. After installing Conda.

On Mac OS X

First, make sure you have Xcode and CommandLineTools installed and updated to latest version (in AppStore). If you have not already installed CommadLineTools, go to a terminal window and run:

  xcode-select --install

First download the latest version of Miniconda3 and run it to install.

  curl -o Miniconda3-latest-MacOSX-x86_64.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
  sh Miniconda3-latest-MacOSX-x86_64.sh

Follow the instructions on screen, scrolling down, pressing ENTER and replying yes when necessary. Install it in the default directory. Restart your terminal window to apply modifications. After restarting, you can type the command below to install Mamba:

  conda init
  conda install -n base -c conda-forge mamba

On Ubuntu

First download the latest version of Miniconda3 and run it to install.

  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  sh Miniconda3-latest-Linux-x86_64.sh

Follow the instructions on screen replying yes when necessary. Restart your terminal window to apply modifications. After restarting, you can type the command below to install Mamba:

  conda init
  conda install -n base -c conda-forge mamba

On Windows 10

Unfortunately, not all packages available on conda are compatible with windows machines. The good news is that Windows 10 offers native linux support via the Windows Subsystem for Linux (WSL2). This allows you to run linux/bash commands from within windows without the need of a virtual machine nor a dual-boot setup (i.e. having 2 operating systems). However, WSL does not offer a complete support for graphical interfaces (such as RStudio in our case), so we need additional steps to make that happen.

On Windows 10, install the WSL if you don’t have it. Follow the instructions here: https://docs.microsoft.com/en-us/windows/wsl/install-win10
Once you have that installed, you can download and install MobaXterm (which is the enhanced terminal with graphical capacity): https://mobaxterm.mobatek.net
It is recommended that you INSTALL the program and not use the portable version.
Inside MobaXterm, you will probably will see that your WSL is already listed on the left panel as an available connection. Just double-click it and you will be accessing it via MobaXterm. If by any chance you don’t see it there, close MobaXterm and go to the WSL terminal, because probably the WSL is not allowing SSH connections. You can follow this link for the instructions on how to do it. You need to complete until the step Start or restart the SSH service, while the further steps are optional, but might be useful.
Inside MobaXterm, download Conda with the command:

  wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Inside MobaXterm, type the commands below to install Conda. Follow the instructions for the installation there.

  cd ~/Downloads
  sh Miniconda3-latest-Linux-x86_64.sh

Inside MobaXterm, Follow the instructions on screen replying yes when necessary. Restart your terminal window to apply modifications. After restarting, you can type the command below to install Mamba:

  conda init
  conda install -n base -c conda-forge mamba

Inside MobaXterm, type the commands below to install the X-server graphical packages that will be used to launch RStudio. https://docs.anaconda.com/anaconda/install/linux

  sudo apt-get update
  sudo apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6

Close and open all application and Inside MobaXterm, you will probably will see that your WSL is already listed on the left panel as an available connection. Just double-click it and you will be accessing it via MobaXterm.

On VirtualBox

If by any means you see that the installations are not working as it should on your computer, you can try to create a virtual machine to run UBUNTU and install everything there. But please keep this alternative as the last temporary resourse, as we recommend troubleshooting the installation o the up-mentioned methods.

Download and install on your machine VIRTUALBOX https://www.virtualbox.org
Download the ISO disk of UBUNTU https://ubuntu.com/download/desktop
On VIRTUALBOX, click on Settings (yellow engine) > General > Advanced and make sure that both settings Shared Clipboard and Drag’n’Drop are set to Bidirectional.
Completely close VIRTUALBOX and start it again to apply changes.
On VIRTUALBOX, create a machine called Ubuntu and add the image above

set the memory to the maximum allowed in the GREEN bar
- set the hard disk to be dynamic allocated
- all other things can be default
1. Proceed with the Ubuntu installation as recommended. You can set to do “Minimal Installation” and deactivate to get updates during installation.
2. Inside Ubuntu, open TERMINAL and type the commands below to install the X-server graphical packages that will be used to launch RStudio. https://docs.anaconda.com/anaconda/install/linux
```
sudo apt-get update
sudo apt-get install libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6
```
1. Inside UBUNTU, Download conda:
```
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
```
1. Inside UBUNTU, open the TERMINAL and type the commands below. Follow the instructions for the installation there.
```
cd ~/Downloads
sh Miniconda3-latest-Linux-x86_64.sh
```
1. Close Terminal to apply the CONDA updates. Then you can create a course folder, download the environment file and create the environment:
```
mkdir ~/Desktop/course
cd ~/Desktop/course
wget https://raw.githubusercontent.com/NBISweden/workshop-your_workshop/master/path/to/environment.yml
conda env create -f environment.yml -n your_environment
```
1. You can then follow the instructions above to activate/deactivate the environment.
```
conda activate your_environment
rstudio &
```

2. Create a conda environment from file

To download the your_environment.yml file using the command on Terminal:

curl -o environment.yml https://raw.githubusercontent.com/NBISweden/workshop-your_workshop/master/path/to/environment.yml

After this, you should have a file named environment_scRNAseq2021.yml in your directory (it does not matter where). Next, type:

mamba env create -n your_environment -f environment.yml

Several messages will show up on your screen and will tell you about the installation process. This may take a few minutes depending on how many packages are to be installed.

##Collecting package metadata: done
##Solving environment: done
##
##Downloading and Extracting Packages
##libcblas-3.8.0       | 6 KB      | ############################################################################# | 100%
##liblapack-3.8.0      | 6 KB      | ############################################################################# | 100%
##...
##Preparing transaction: done
##Verifying transaction: done
##Executing transaction: done

3. Activate the environment

Once the environment is created, we need to activate it in order to use the softwares and packages inside it. To activate an environment type:

conda activate your_environment

From this point on you can run any of the contents from the workshop. For instance, you can directly launch RStudio by typing rstudio. Here it is important to add the & symbol in the end to be able to use the command line at the same time if needed. You can open other files from Rstudio later as well.

rstudio PATH/my_script.Rmd &

4. Deactivate the environment

After you’ve ran all your analyses, you can deactivate the environment by typing:

conda deactivate

Git installation

Contributors: Rui Benfeitas

Install git through mamba: mamba install -c anaconda git
Install git

Zoom installation

Contributors: Paulo Czarnewski

Make sure that you have the latest Zoom (version 5.4.0 or above) installed because we will use it a lot during the workshop for the lecture and groups discussions.

Previous Zoom versions will not work. If you already have Zoom installed, you can also update it by clicking in Check for Updates ... button in the main top menu.

Docker instructions

Contributors: Rui Benfeitas

To reproduce all analyses you will need to:

Install Docker;
Create the containers;
Launch rstudio or jupyter;
Stop the containers

If you have previously followed the conda instructions, we still recommend that you follow the instructions below as they may avoid some bugs.

Create the containers

At this point you need to create the two containers for all Rstudio or Jupyter notebooks. You can do so by choosing one of the following options:

> Download image from Dockerhub (recommended)

Pull and start the images

  ########### Rstudio image ###########
  # Your user is 'omics' (without the quotes)
  # Replace <yourpassword> with your desired password
  sudo docker run -d --rm -p 8787:8787 -e PASSWORD=<yourpassword> ruibenfeitas/rstudio:16_07_2021

  ########### Jupyter image ###########
  # Your user is 'jovyan' (without the quotes)
  # Replace <yourpassword> with your desired password
  sudo docker run -d --rm -p 8888:8888 -e JUPYTER_TOKEN=<yourpassword> ruibenfeitas/jupyter:16_07_2021

> Download the dockerfiles from github

On github you will find the dockerfiles necessary from the github repository. Download all files (Dockerfile_jupyter, Dockerfile_rstudio, docker-compose.yml and environment_jupyter), install docker compose and install git. Then:

## go to the dir where the dockerfiles are found
# for e.g.: 
# https://github.com/NBISweden/workshop_omicsint_ISMBECCB/tree/main/docker
## 
cd /path/to/dockerfiles

Build the containers

docker-compose build

At this point you can start and run the containers

########### Rstudio image ###########
# Your user is 'omics' (without the quotes)
# Replace <yourpassword> with your desired password
docker-compose up -d -p 8787:8787 -e PASSWORD=<yourpassword>

########### Jupyter image ###########
# Your user is 'jovyan' (without the quotes)
# Replace <yourpassword> with your desired password
docker-compose up -d -p 8888:8888 -e JUPYTER_TOKEN=<yourpassword>

Launch RStudio or Jupyter

Ensure you have followed all the instructions above and that your containers are running. If you have followed the instructions from the recommended solution you can simply access either RStudio or Jupyter from your browser with:

localhost:8888 to launch jupyter
localhost:8787 to launch rstudio

All notebooks are found within the folder workshop/ as indicated in the labs. If you want to verify that your containers are running use docker ps.

Stop the containers

To stop the containers write docker stop [container name].