Using SSH

ssh is an invaluable tool to securely login to remote systems to allow you to work there. You’re probably already using it. With a few extra steps can become even more powerful and useable.

For clarity, the machine at whose keyboard you are sitting is called the local machine and the machine you’d like to log in to is called the remote machine.

This guide is targeted at Unix systems, if you have experience with or would like to use windows, a merge request is welcomed.

Public Key Authentication

The basic way ssh authenticates you is by typing your password when you log in. Repeatedly typing your password can quickly become cumbersome, especially when using scp. To improve this, public key authentication can be used.

Generating a Key

Attention

Generate your keys on a trusted machine, keep them safe and use a strong password!

The ssh-keygen command can be used to generate a private-public key pair. Either an ED25519 key or a 4096 bits RSA key is fine. Use a strong password! As ED25519 is not supported by older ssh servers and 4096 bits RSA is considered safe, it is used throughout.

$> ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/username/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/username/.ssh/id_rsa.
Your public key has been saved in /home/username/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:P0aZKsGVvHnn/PCk7l+FFu6ev4AgcIbwH4sJbCIYmTM username@localhost
The key's randomart image is:
+---[RSA 4096]----+
|.o .             |
|E.. o .. .       |
|o+ + + ++     .  |
|. o ..B.oo o . o |
|     oo+S.= . + .|
|       ..=.+.o  .|
|      . . +.+.o .|
|       . . . B.o |
|           o+.*oo|
+----[SHA256]-----+

Or for an ED25519 key:

$> ssh-keygen -t ed25519

To make sure your ssh directory is only readable by you, update it’s permissions:

$> chmod -R go-rwx ~/.ssh

Installing your public key

Once your keys have been generated, you can start using them by installing the public key on the remote machine.

To do this, your public key should be appended to the file ~/.ssh/authorized_keys on the remote machine. The most convenient way to do this is usually to copy it using your password to log in:

$> ssh-copy-id username@remote

It can also be done manually by appending your key to the ~/.ssh/authorized_keys file on the remote machine. After copying the key to the clipboard, instead of using echo on the remote machine, an editor can also be used.

$> cd ~/.ssh
$> cat id_rsa.pub
your_key
$> ssh username@remote
$> mkdir -p ~/.ssh
$> chmod go-rwx ~/.ssh
$> echo "your_key" >> ~/.ssh/authorized_keys

Test it

You should now be able to authenticate using your private keys:

$> ssh -o PubkeyAuthentication=yes username@login.nikhef.nl

Should give:

Enter passphrase for key '/home/username/.ssh/id_rsa':

SSH Configuration

Your ssh configuration is stored (by default) in the file ~/.ssh/config. Using the your ssh config file can save you a lot of typing.

The file is organised by remote machine, defined by Host sections; wildcards are allowed. If a host matches multiple wildcards, all of them are considered. For each parameter, the first one is taken. Parameters can also be specified globally.

A few useful parameters as an example (make sure to modify username):

Host *
   ServerAliveInterval 60
   ServerAliveCountMax 5
   TCPKeepAlive yes

Host login.nikhef.nl login
   User username
   Protocol 2
   PubkeyAuthentication yes
   PasswordAuthentication yes

See ssh_config manual or $> man ssh_config for more details.

Agents

An ssh agent is an application that runs on a machine to which you can add your private key to avoid typing its password repeatedly. This does entail a security risk if somebody else can communicate with the agent application. If a machine is secure and you trust the system administrators, this is usually acceptable.

Once an agent has been setup, run the following to add your keys to it:

$> ssh-add -c

It searches for keys in the ~/.ssh directory generated with the default names by ssh-keygen. If you put your key somewhere different or named it differently, add the key file as an argument to ssh-add.

The -c flag causes you to be asked for confirmation every time your key is used, which is recommended for security reasons. See improved security for more details, or if no window pops up.

See the following sections on how to setup an agent.

MacOS

MacOS Keychain is automatically setup to act as an ssh agent.

Gnome

The Gnome keyring should be able to store your keys. If you use Gnome, ssh-add -c from a terminal should just work™.

KDE Plasma

Depending on the distribution, KDE Plasma doesn’t always have an agent started by default. If one is not available, an agent can be started when Plasma starts by saving the following as: ~/.config/plasma-workspace/env/ssh-agent-statup.sh:

#!/bin/sh
[ -n "$SSH_AGENT_PID" ] || eval "$(ssh-agent -s)"

and the following as ~/.config/plasma-workspace/env/ssh-agent-shutdown.sh

#!/bin/sh
[ -z "$SSH_AGENT_PID" ] || eval "$(ssh-agent -k)"

And log out and back in.

Terminal

You can also run the agent per terminal by executing (in bash):

$> eval `ssh-agent`

You’ll have to run ssh-add for every terminal, which makes this less convenient.

Improved Security with Agent

To increase the security when running an agent, it is recommended to add the -c flag when adding a key to the agent using ssh-add:

$> ssh-add -c

This will result in the agent asking for confirmation every time the key is used.

For this to work, the SSH_ASKPASS environment variable must point to a program that can pop-up a display. Depending on your desktop environment x11-ssh-askpass, gnome-ssh-askpass or kde-ssh-askpass may be installed. To enable use of one of these, ensure the following is set before the agent is started:

export SSH_ASKPASS=`which gnome-ssh-askpass`

Agent Forwarding

If you connect to a gateway machine such as login.nikhef.nl and would like to use the ssh agent when connecting from that machine to other machines, your ssh agent needs to be forwarded. This can be achieved by adding the following line to the configuration section of the gateway machine:

Host login.nikhef.nl
   ForwardAgent yes

X forwarding

It is possible to forward X11 connections over ssh to allow you to open GUI applictions on a remote machine and display them on your own. This requires a quite responsive network-connection to be workable; your kilometrage may vary.

To enable X11 forwarding, add the following to your ssh config file for the hosts where you want it enabled:

Host stbc-i*.nikhef.nl stbc-i*
   ForwardX11 yes
   ForwardX11Trusted yes

Test It

This should show a clock window:

$> ssh stbc-i5.nikhef.nl
$> xclock

Accessing Machines through a Gateway

Sometimes a machine you’d like to access cannot be reached directly fromt the network you are on. The interactive stoomboot nodes, for example, can only be reached from the Nikhef networks. If you’d like to connect to one of them from home or from a university network, you’d first have to log in to the gateway machine - login.nikhef.nl in this case - before being able to connect to the remote machine. Doing that by hand in two steps is cumbersome.

SSH allows this to be achieved using a single connection command by setting up a ProxyJump in your ssh configuration file. To connect to stbc-i5 through login.nikhef.nl add the following to the configuration section for the stoomboot nodes (create if it doesn’t exist):

Host stbc*.nikhef.nl stbc*
   User username
   ProxyJump login.nikhef.nl

To test this, ensure you’re on a network where the nodes cannot be directly contacted, for example by using the “Amsterdam Science Park” wifi network. Executing

$> ssh stbc-i5.nikhef.nl

Should now allow you to connect. If you are asked for your password multiple times, ensure your ssh agent is setup and running.

Port Forwarding

SSH allows tunnelling of traffic over the encrypted SSH connection. This can come in handy when dealing with firewalls or trying to access a machine that is only reachable through a gateway or a firewall.

Another use case is to connect to a service on another machine as if you were on that machine. This is very useful when connecting to a jupyter notebook running remotely; for example on an interactive stoomboot node. To show this, log in to an interactive stoomboot node and start a notebook:

$> ssh stbc-i5.nikhef.nl
$> source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_94
x86_64-centos7-gcc7-dbg
$> jupyter notebook --no-browser
.
.
.
    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8890/?token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXx

Note the port that was opened on the remote host, in this case 8890. The default port for a notebook server is 8888, but if a machine is shared, that port might be taken by another notebook server started by another user.

In a shell on your local machine run:

$> ssh -f -N -L 8890:localhost:8890 stbc-i5.nikhef.nl

The -f switch tells ssh to run in the background; -N tells it to not execute any command on the remote host. The systax of -L is local_port:remote_server:remote_port remote; local_port is opened on your local machine; remote_server can be any machine reachable from remote - which includes localhost; and remote is the machine ssh will connect to.

To view the notebook in your own browser, open it and point it to the link (including token) that jupyter printed when starting up.

If you are going to run the remote service you want to connect to yourself from an interactive ssh session, and you know the port number up front, the two separate ssh commands can be combined:

ssh -L 8888:localhost:8888 stbc-i5.nikhef.nl

Authenticating with Kerberos

Kerberos is an authentication framework based on tokens. It is for example used by Cern. To authenticate using Kerberos, there are two steps:

  • Obtain a ticket

  • Have ssh use the ticket to authenticate

To obtain a ticket, use (for Cern, use CERN.CH):

$> kinit username@SITE.COM

Then for your host add:

Host remote.site.org
   GSSAPIAuthentication yes
   GSSAPIDelegateCredentials yes

lxplus

CERN’s lxplus interactive cluster is used by many. There are a few things to note. As a result of the presence (not clear how much longer) of afs, public key authentication shouldn’t be used as you won’t have access to your home directory when it’s used to authenticate you. Instead, PubKeyAuthentication should be turned of and kerberos should be used.

lxplus uses a dns-based login to point you to a specifice node when you log in to the general address. For this to work, a patch must have been applied to your installation of openssh to allow this to work. In particular the option GSSAPITrustDns must be available and set. An example ssh config for lxplus is:

Host lxplus*.cern.ch lxplus*
   User cern_username
   ForwardX11 yes
   ForwardX11Trusted yes
   ForwardAgent yes
   PasswordAuthentication yes
   PubkeyAuthentication no
   GSSAPIAuthentication yes
   GSSAPIDelegateCredentials yes
   GSSAPITrustDns yes

More details about this issue here.

Using SSH as a Proxy Server

An SSH connection can serve as an HTTP(S) proxy server, which can be used to access web sites or services behind a firewall or gateway machine. To enable the proxy capabilities of ssh, add -D <port> to the command. A typical port is 8080:

ssh -D 8080 -f -N username@login.nikhef.nl

This starts ssh in the background. If you also require an interactive login sessions, the -f -N can be omitted.

Once the proxy is up, the browser needs to be configured to use it, see the following section for more details.

Firefox

To configure firefox’s proxy settings, open “Preferences -> Network Settings” and select “Manual Proxy Configuration”. Enter “localhost” and your chosen port. Also tick the “Proxy DNS with using SOCKS v5” box. See this guide for a more extensive explanation including pictures.

Alternatively, an addon such as FoxyProxy can be used to configure a per-site proxy server or per-site exception rules. See this guide for more information.

Chrome

Chrome (and Chromium) use the desktop environment’s proxy settings. To set the proxy server that chrome should use for everything, start chrome with:

$> google-chrome-stable --proxy-server="http://localhost:8080"

Alternatively, an addon such as Proxy-SwitchyOmega can be used to configure a per-site proxy server or per-site exception rules. See this guide for more information.