\(T\) | time normally spent on related tasks |
\(I\) | investment |
\(R\) | rate of productivity increase |
Learning a skill is worthwhile if \[ T ≥ I + \frac{T}{R} \]
(Yes, absolutely)
\(T\), \(I\), and \(R\) can only be learned from experience.
Getting Linux on your laptop:
Don't. But if you must:
.ssh/config
Host *.nikhef.nl
ControlMaster auto
ControlPath /tmp/%h-%p-%r.shared
Host *
ForwardAgent yes
User yournamehere
HashKnownHosts yes
ssh-keygen
cat ${HOME}/.ssh/id_rsa.pub > authorized_keys
scp authorized_keys login:.ssh/authorized_keys
Permissions:
drwxr-xr-x .ssh/ -rw-r--r-- .ssh/authorized_keys -r--r--r-- .ssh/id_rsa.pub -r-------- .ssh/id_rsa
ssh-add -l # list keys in the agent
ssh -A login # login with agent forwarding
Host stbci5.proxy
Hostname stbc-i5.nikhef.nl
user yournamehere
CheckHostIP no
ProxyCommand ssh -q -A login.nikhef.nl /usr/bin/nc %h %p 2>/dev/null
Fuse mount your remote home directory locally:
sshfs login.nikhef.nl: /tmp/login
ll /tmp/login/
fusermount -u /tmp/login
select your default shell at https://sso.nikhef.nl/chsh.
/bin/bash |
YES |
/bin/zsh |
YES |
/bin/csh |
NO! |
login shell | .bash_profile |
non-login shell | .bashrc |
This distinction is outmoded.
.bash_profile
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
.bashrc
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
TAB
will auto-complete your command linebash-completions
installed
.bashrc
# don't keep more than one copy of a repeated command
HISTCONTROL=ignoredups
# append to the history file, don't overwrite it
shopt -s histappend
# keep plenty of history
HISTSIZE=65000
# useful on systems with shared home directories
HISTFILE=${HOME}/.bash_history-$(hostname)
# keep track of time
HISTTIMEFORMAT='%F %T %Z # '
Ctrl-R
reverse search in historySeeing is believing.
stat /some/path/to/file
# now I want to run cat on the same file
cat <.>
cat /some/path/to/file
.bashrc
PS1='\u@\h:\w \A $(__git_ps1 " (%s)")\$ '
This shows:
a07@lena:/project/newton 11:24 (master)$
alias ls='ls --color=tty'
alias ll='ls -lhF'
alias rm='rm -i'
alias mv='mv -i'
script
to capture an entire session
Write myscript.sh
:
# my first script
echo "This is my first shellscript"
And then run it like
bash ./myscript.sh
Turn it into an executable like so:
#!/bin/bash
# my first script
echo "This is my first shellscript"
followed by
chmod +x myscript.sh
./myscript.sh
Make a habit out of always quoting variables like so:
"${var}"
and you will never go wrong.
Do not use eval
ever.
#!/bin/sh
proxyhost=login.nikhef.nl
proxyport=8888
while getopts :h:p: OPT; do
case $OPT in
h|+h) proxyhost="$OPTARG" ;;
p|+p) proxyport="$OPTARG" ;;
*) echo "usage: `basename $0`"\
"[+-h proxyhost] [+-p proxyport} [--] ARGS..."
exit 2 ;;
esac
done
shift `expr $OPTIND - 1`
OPTIND=1
ssh -n -N -f -D "$proxyport" "$proxyhost" "$@"
Jeff thoroughly tested the following code. Then he changed one line. What went wrong?
#!/bin/bash # clean up leftover files # echo 'running in test mode' echo 'now it's running in production' path=var/batch/jobs # it's ok to drop old file retention="30" find /$path -type f -mtime +$retention -exec rm {} +
#!/bin/bash # clean up leftover files # echo 'running in test mode' echo 'now it's running in production' path=var/batch/jobs # it's ok to drop old file retention="30" find /$path -type f -mtime +$retention -exec rm {} +
#!/bin/bash
# clean up leftover files
# echo 'running in test mode'
echo 'now it's running in production'
path=var/batch/jobs
# it's ok to drop old file
retention="30"
find /$path -type f -mtime +$retention -exec rm {} +
You will find yourself at times pondering why your shell script went south. Here is what you do next.
echo $?
set -e
trap 'fail $LINENO' ERR
fail() {
echo "error on line $1" >&2
}
input | stdin | 0 |
output | stdout | 1 |
output | stderr | 2 |
Redirect both output streams to separate files.
run=`date -u +%FT%T`
./analysis.sh > "output.$run" 2> "err.$run"
echo "now starting the frobnicator" >&2
set -x
foo=somevalue
echo $foo
set +x
echo done
Renders:
+ foo=somevalue + echo somevalue somevalue + set +x done
Dump the environment and check carefully:
PATH
LD_LIBRARY_PATH
LD_RUNPATH
PYTHONPATH
LANG
, LC_*
For completeness sake, here we compound stdout and stderr onto a single file.
./whatever.sh > all_the_output 2>&1
Mind the ordering. First you need to send stdout to a file, then you want to send stderr to the same stream.
“do one thing and do it well.”
Find e-mail addresses:
grep -E -o "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,6}\b"
cat | just listed here for the most useless use of cat award |
sed | streamline editor with regular expression powers |
awk | the duct tape of Unix tools |
grep | find strings in files |
sort | order lines |
cut | select fields from each line |
diff | show differences between files |
head/tail | tail -f is actually useful |
tar | roll directories into tarballs |
gzip | compress files or data streams |
ls |
swiss army knife of file listings |
find |
most of the time you want to use locate instead |
touch |
create files out of nowhere, update timestamps |
cp |
copy |
mv |
move or rename |
ln |
link |
rm |
really remove |
rsync |
copy on steroids |
which |
where is my executable? |
stat |
what can we tell about a file |
du |
disk usage |
ps |
list processes, like ps aux or ps -ef |
top |
who is eating my cpu and memory? |
kill |
sending signals |
bg/fg |
background/foreground programs |
lsof |
find open files |
vmstat |
memory, buffers and io |
free |
overview of memory |
ip |
swiss army knife of network tools |
ip addr |
show network addresses on this system |
ip route |
show the routing table |
ping |
see if we can reach a machine |
dig |
query DNS |
traceroute |
see which path takes us to a machine |
ssh |
secure shell |
nc |
netcat, less useless than cat |
apt/dpkg |
Debian’s package manager |
yum/rpm |
Red Hat’s package manager |
pip |
Python package tool |
Below are a few examples.
This traverses a directory and finds all files of a certain name and then tries to grep for a certain pattern in these files.
find . -type d \( -path \*/.svn \
-o -path \*/.git \) -prune -o \
-type f \( -name \*.txt \) \
-exec grep --color -i -nH -e searchterm {} +
for i in `seq -f file-%03g.txt 1 100` ; do
sort -t, -n -k2 $i | cut -d, -f2,4-8 | \
tail -n 1 > ${i%.*}.ord
done
A set of 100 comma-separated data files is numerically sorted on the second field, cut to only output fields 2, 4, 5, 6, 7, and 8, and then the last lines are saved to an output file.
du -s * | sort -n
Show which file/directory uses the most disk space.
ls -lrt # sort by timestamp
find . -mmin -10 -ls # find files changed in the last 10 minutes
At some point you will need to edit files: source code, LaTeX files, shell scripts, configuration files…
Modern Linux systems have plenty of editors to choose from.
Originally vi
, its pedigree going back to the original editor
called ed
.
Sometimes you remote session should last longer than your workday. Or your laptop’s battery.
The screen
utility allocates a pseudo terminal attached to
a background process independent of your session. You can run
multiple shells in a screen and manoeuvre around with the Ctrl-A
prefix. Type Ctrl-A ?
for a help screen.
The tmux utility is a remake of screen, with modernised session handling, scripting, split screen, and ease of use. It is still less ubiquitous than screen so you may not have the option to run it unless you bring your own.
(This may not be your choice to make.)
Security considerations are usually not at the top of everyone’s priority list. By following the guidelines here most bases are covered.
Ask the experts.
Treat passwords with extreme care.
Passwords are considered ‘something only you know’, but as soon as you write them down somewhere, on a piece of paper or in a file, you could inadvertently share this with others.
Never put passwords in a script. There is always a better way. Be aware that passwords typed on the command line will appear in your history file.
Established practice for safely creating temporary files is by
using mktemp
.
tmpfile=`mktemp`
tmpdir=`mktemp -d`
This takes care of creating a new file with a randomised name that is guaranteed to be owned by the user.
Sometimes scripts need to use a password to authenticate or
unlock. The script can read the password from stdin
and keep
it in a local variable for the time that it is needed.
stty -echo echo "enter password:" read passwd stty echo mkproxy --passin - <<<$passwd unset passwd
Be aware that putting passwords on the command-line means that it will show up in the process list.