2021-01-21 21:45:05 +09:00
---
2022-01-16 19:51:33 +09:00
title: Setting Up Arch Linux
2021-02-25 22:36:19 +09:00
date: 2021-02-12T00:00:00
2021-01-21 21:45:05 +09:00
---
2021-12-14 17:13:26 +09:00
This note includes all commands I typed when I set up Arch Linux on my new server.
2021-01-21 21:45:05 +09:00
2021-12-14 17:13:26 +09:00
> PSA: I published a toolchain for building AUR packages in a clean-room docker container
> <https://github.com/uetchy/archpkgs>
2021-01-21 21:45:05 +09:00
2021-06-06 21:56:40 +09:00
# Setup
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
## Wipe a device
2021-01-21 21:45:05 +09:00
```bash
2022-02-05 13:41:25 +09:00
wipefs -a /dev/sda # erase file-system (fast)
# or
shred -v -n 1 --random-source=/dev/urandom /dev/sda # write pseudo-random data to the device (takes longer)
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
## Create partitions
2021-01-21 21:45:05 +09:00
```bash
parted
2021-06-11 15:15:02 +09:00
```
2021-01-21 21:45:05 +09:00
2021-06-11 15:15:02 +09:00
```bash
2021-01-21 21:45:05 +09:00
select /dev/sda
mktable gpt
2022-02-05 13:41:25 +09:00
mkpart EFI fat32 0 512MB # EFI (Extensible Firmware Interface)
2021-01-21 21:45:05 +09:00
mkpart Arch ext4 512MB 100% # Arch
2022-02-05 13:41:25 +09:00
set 1 esp on # flag partition 1 as ESP (EFI System Partition)
2021-01-21 21:45:05 +09:00
quit
```
2022-03-04 15:03:48 +09:00
> NOTE: Since the server has 128GB of physical memory, I would rather let OOM Killer do its job than create a swap partition.
2022-05-20 13:18:51 +09:00
> Should the need for swap come up later, consider [swapfile](https://wiki.archlinux.org/title/Swap#Swap_file) (no difference in performance)
```bash
# create 8GB swap file
dd if=/dev/zero of=/swapfile bs=1M count=8000 status=progress
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
echo "/swapfile none swap defaults 0 0" >> /etc/fstab
```
2022-03-04 15:03:48 +09:00
2022-02-05 13:41:25 +09:00
## Install file-system for UEFI/GPT setup
2021-01-21 21:45:05 +09:00
```bash
mkfs.vfat -F 32 /dev/sda1 # EFI
mkfs.ext4 /dev/sda2 # Arch
```
2022-01-17 13:42:14 +09:00
## Mount disks
2021-01-21 21:45:05 +09:00
```bash
mkdir -p /mnt/boot
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
```
2022-02-05 13:41:25 +09:00
## Install the base system
2021-01-21 21:45:05 +09:00
```bash
2022-02-05 13:41:25 +09:00
# choose between 'linux' and 'linux-lts'
2021-07-28 15:22:05 +09:00
pacstrap /mnt base linux-lts linux-firmware
2021-01-21 21:45:05 +09:00
arch-chroot /mnt
```
2022-02-05 13:41:25 +09:00
## Configure pacman
2021-01-21 21:45:05 +09:00
```bash
2022-02-05 13:41:25 +09:00
# optimize mirrorlist
2021-07-28 15:22:05 +09:00
pacman -S reflector
reflector --protocol https --latest 30 --sort rate --save /etc/pacman.d/mirrorlist --verbose # optimize mirror list
2022-02-05 13:41:25 +09:00
# enable color output
sed '/Color/s/^#// ' -i /etc/pacman.conf
2021-07-28 15:22:05 +09:00
```
2021-12-14 17:13:26 +09:00
## Install essentials
2021-07-28 15:22:05 +09:00
```bash
pacman -S vim man-db man-pages git base-devel
```
2022-02-05 13:41:25 +09:00
## Configure fstab
```bash
pacman -S xfsprogs # for XFS
genfstab -U /mnt >> /etc/fstab # Generate fstab based off of /mnt
```
2021-06-08 16:50:58 +09:00
```ini /etc/fstab
# backup
UUID=< UUID > /mnt/backup ext4 defaults 0 2
# archive (do not prevent boot even if fsck fails)
UUID=< UUID > /mnt/archive ext4 defaults,nofail,x-systemd.device-timeout=4 0 2
2022-02-05 13:41:25 +09:00
# xfs drive
UUID=< UUID > /mnt/archive2 xfs defaults 0 0
2021-06-08 16:50:58 +09:00
```
2022-01-17 13:42:14 +09:00
You can find `<UUID>` from `lsblk -f` .
2021-06-08 16:50:58 +09:00
```bash
findmnt --verify --verbose # verify fstab
```
2022-02-05 13:41:25 +09:00
## Install bootloader (GRUB 2)
2021-01-21 21:45:05 +09:00
```bash
pacman -S \
grub \
efibootmgr \
amd-ucode # AMD microcode
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB
2021-06-06 21:56:40 +09:00
vim /etc/default/grub
2022-02-05 13:41:25 +09:00
sed -E \
-e 's/(GRUB_TIMEOUT=).+/\13/' \
-e 's/(GRUB_DISABLE_SUBMENU=).+/\1y/' \
-i /etc/default/grub
2021-01-21 21:45:05 +09:00
grub-mkconfig -o /boot/grub/grub.cfg
```
2021-06-06 21:56:40 +09:00
- [GRUB/Tips and tricks - ArchWiki ](https://wiki.archlinux.org/title/GRUB/Tips_and_tricks )
2022-02-05 13:41:25 +09:00
## Configure locales
```bash
ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
hwclock --systohc
vim /etc/locale.gen
locale-gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
```
2021-12-14 17:13:26 +09:00
## Setup network
2021-01-21 21:45:05 +09:00
```bash
2021-07-28 15:22:05 +09:00
hostnamectl set-hostname takos
2021-01-21 21:45:05 +09:00
hostnamectl set-chassis server
2021-02-25 22:36:19 +09:00
```
2021-01-21 21:45:05 +09:00
2021-02-25 22:36:19 +09:00
```ini /etc/hosts
127.0.0.1 localhost
::1 localhost
2021-07-28 15:22:05 +09:00
127.0.0.1 takos
2021-01-21 21:45:05 +09:00
```
2021-02-25 22:36:19 +09:00
```ini /etc/systemd/network/wired.network
2021-01-21 21:45:05 +09:00
[Match]
Name=enp5s0
[Network]
#DHCP=yes
Address=10.0.1.2/24
Gateway=10.0.1.1
2022-05-20 13:18:51 +09:00
DNS=10.0.1.100 # Self-hosted recursive DNS resolver
DNS=1.1.1.1 # Cloudflare for the fallback DNS resolver
MACVLAN=dns-shim # to route local DNS lookup to 10.0.1.100, which is managed by Docker MACVLAN driver
2021-01-21 21:45:05 +09:00
```
2021-02-25 22:36:19 +09:00
```ini /etc/systemd/network/dns-shim.netdev
2022-05-20 13:18:51 +09:00
# to route local dns lookups to 10.0.1.100
2021-01-21 21:45:05 +09:00
[NetDev]
Name=dns-shim
Kind=macvlan
[MACVLAN]
Mode=bridge
```
2021-02-25 22:36:19 +09:00
```ini /etc/systemd/network/dns-shim.network
2022-05-20 13:18:51 +09:00
# to route local dns lookups to 10.0.1.100
2021-01-21 21:45:05 +09:00
[Match]
Name=dns-shim
[Network]
IPForward=yes
[Address]
Address=10.0.1.103/32
Scope=link
[Route]
Destination=10.0.1.100/30
```
2021-12-14 17:13:26 +09:00
`ip` equivalent to the above config:
2021-01-21 21:45:05 +09:00
```bash
2021-12-14 17:13:26 +09:00
ip link add dns-shim link enp5s0 type macvlan mode bridge # add macvlan shim interface
ip a add 10.0.1.103/32 dev dns-shim # assign the interface an ip address
ip link set dns-shim up # enable the interface
ip route add 10.0.1.100/30 dev dns-shim # route macvlan subnet (.100 - .103) to the interface
2021-01-21 21:45:05 +09:00
```
```bash
systemctl enable --now systemd-networkd
networkctl status
2022-05-20 13:18:51 +09:00
# for self-hosted DNS resolver
sed -E 's/^DNS=.*/DNS=10.0.1.100 1.1.1.1/' -i /etc/systemd/resolved.conf
2021-01-21 21:45:05 +09:00
systemctl enable --now systemd-resolved
2022-05-20 13:18:51 +09:00
ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
2021-01-21 21:45:05 +09:00
resolvectl status
resolvectl query ddg.gg
2021-07-28 15:22:05 +09:00
drill ddg.gg
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
If `networkctl` keep showing `enp5s0` as `degraded` , then run `ip addr add 10.0.1.2/24 dev enp5s0` to manually assign static IP address for the workaround.
2021-01-21 21:45:05 +09:00
2022-05-20 13:18:51 +09:00
See also:
- [systemd.network ](https://systemd.network/systemd.network.html )
- [ArchWiki: systemd-networkd ](https://wiki.archlinux.org/title/Systemd-networkd )
- [ArchWiki: systemd-resolved ](https://wiki.archlinux.org/title/Systemd-resolved )
- [Ivan Smirnov's blog ](https://blog.ivansmirnov.name/set-up-pihole-using-docker-macvlan-network/ ).
2022-02-05 13:41:25 +09:00
## Leave chroot environment
2021-01-21 21:45:05 +09:00
```bash
2021-07-28 15:22:05 +09:00
exit # leave chroot
umount -R /mnt
reboot
2021-01-21 21:45:05 +09:00
```
2022-05-20 13:18:51 +09:00
## sysctl
```bash
2022-06-23 16:24:02 +09:00
# increase inotify limit
cat < < EOD > /etc/sysctl.d/97-inotify.conf
fs.inotify.max_user_watches=1048576
fs.inotify.max_user_instances=1000000
EOD
2022-05-20 13:18:51 +09:00
# reboot after 60s of kernel panic
2022-06-23 16:24:02 +09:00
echo "kernel.panic=60" > /etc/sysctl.d/98-kernel-panic.conf
2022-05-20 13:18:51 +09:00
# set swappiness
2022-06-23 16:24:02 +09:00
echo "vm.swappiness=10" > /etc/sysctl.d/99-swappiness.conf
sysctl --system
2022-05-20 13:18:51 +09:00
```
2021-07-28 15:22:05 +09:00
## NTP
```bash
timedatectl set-ntp true
timedatectl status
```
2021-01-21 21:45:05 +09:00
2022-01-17 13:42:14 +09:00
## AUR
```bash
git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si
```
2021-12-14 17:13:26 +09:00
## Shell
2021-01-21 21:45:05 +09:00
```bash
pacman -S zsh
chsh -s /bin/zsh
2022-01-17 13:42:14 +09:00
# Install useful utils (totally optional)
2022-02-05 13:41:25 +09:00
yay -S pyenv exa antibody direnv fd ripgrep fzy peco ghq-bin hub neofetch tmux git-delta lazygit jq lostfiles ncdu htop rsync youtube-dl prettier age
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
## Setup operator user (i.e., a user without a root privilege)
2021-01-21 21:45:05 +09:00
```bash
2022-01-17 13:42:14 +09:00
passwd # change root password
2021-01-21 21:45:05 +09:00
2022-01-17 13:42:14 +09:00
useradd -m -s /bin/zsh < user > # add operator user
passwd < user > # change operator user password
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
userdbctl # show users
userdbctl group # show groups
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
pacman -S sudo # install sudo
# add operator user to sudo group
2021-07-28 15:22:05 +09:00
groupadd sudo
2022-02-05 13:41:25 +09:00
usermod -aG sudo < user >
echo "%sudo ALL=(ALL) NOPASSWD:/usr/bin/pacman" > /etc/sudoers.d/pacman # allow users in sudo group to run pacman without password (optional)
visudo -c # validate sudoers
2021-01-21 21:45:05 +09:00
```
2021-12-14 17:13:26 +09:00
## SSH
2021-01-21 21:45:05 +09:00
```bash
pacman -S openssh
vim /etc/ssh/sshd_config
systemctl enable --now sshd
```
2022-02-05 13:41:25 +09:00
on the client machine:
```bash
# brew install ssh-copy-id
ssh-copy-id < user > @< serverIp >
```
### Make SSH forwarding work with tmux and sudo
2022-01-17 13:42:14 +09:00
```bash ~/.ssh/rc
2021-10-27 15:53:06 +09:00
if [ ! -S ~/.ssh/ssh_auth_sock ] & & [ -S "$SSH_AUTH_SOCK" ]; then
ln -sf $SSH_AUTH_SOCK ~/.ssh/ssh_auth_sock
fi
```
2022-01-17 13:42:14 +09:00
```bash ~/.tmux.conf
set -g update-environment -r
setenv -g SSH_AUTH_SOCK $HOME/.ssh/ssh_auth_sock
```
2021-10-27 15:53:06 +09:00
2022-01-17 13:42:14 +09:00
```bash /etc/sudoers
Defaults env_keep += SSH_AUTH_SOCK
```
2022-02-05 13:41:25 +09:00
See also: [Happy ssh agent forwarding for tmux/screen · Reboot and Shine ](https://werat.dev/blog/happy-ssh-agent-forwarding/ )
## MTA
### mail
Needed for `smartd` .
2021-01-21 21:45:05 +09:00
```bash
2022-02-05 13:41:25 +09:00
pacman -S s-nail
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
- [S-nail - ArchWiki ](https://wiki.archlinux.org/title/S-nail )
### sendmail
Needed for `cfddns` .
```
yay -S sendmail
```
2022-01-17 13:42:14 +09:00
2021-12-14 17:13:26 +09:00
## S.M.A.R.T.
2021-01-21 21:45:05 +09:00
```bash
2021-07-28 15:22:05 +09:00
pacman -S smartmontools
2022-02-05 13:41:25 +09:00
```
### Setup automated checkups
```ini /etc/smartd.conf
# scan all but removable devices and notify any test failures
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../6/03) -m me@example .com
```
```bash
2021-07-28 15:22:05 +09:00
systemctl enable --now smartd
2022-02-05 13:41:25 +09:00
```
### Manual testing
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
```bash
2021-07-28 15:22:05 +09:00
smartctl -t short /dev/sda
smartctl -l selftest /dev/sda
```
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
See: [S.M.A.R.T. - ArchWiki ](https://wiki.archlinux.org/title/S.M.A.R.T. )
## lm_sensors
```bash
pacman -S lm_sensors
sensors-detect
systemctl enable --now lm_sensors
```
See: [lm_sensors - ArchWiki ](https://wiki.archlinux.org/title/Lm_sensors )
2021-12-14 17:13:26 +09:00
## NVIDIA drivers
2021-01-21 21:45:05 +09:00
```bash
2022-01-17 13:42:14 +09:00
pacman -S nvidia-lts # 'nvidia' for 'linux'
2021-07-28 15:22:05 +09:00
reboot
2022-02-05 13:41:25 +09:00
nvidia-smi # validate runtime
2021-01-21 21:45:05 +09:00
```
2021-12-14 17:13:26 +09:00
## Docker
2021-01-21 21:45:05 +09:00
```bash
pacman -S docker docker-compose
2021-05-06 13:53:02 +09:00
yay -S nvidia-container-runtime
2021-01-21 21:45:05 +09:00
```
2021-02-25 22:36:19 +09:00
```json /etc/docker/daemon.json
2021-01-21 21:45:05 +09:00
{
2021-06-06 21:56:40 +09:00
"log-driver": "json-file",
2021-01-21 21:45:05 +09:00
"log-opts": {
2021-06-06 21:56:40 +09:00
"max-size": "10m", // default: -1 (unlimited)
"max-file": "3" // default: 1
2021-01-21 21:45:05 +09:00
},
"runtimes": {
// for docker-compose
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
}
```
```bash
2022-02-05 13:41:25 +09:00
systemctl enable --now docker
2021-01-21 21:45:05 +09:00
2021-07-28 15:22:05 +09:00
usermod -aG docker < user >
# to create mandatory device files on /dev
docker run --gpus all nvidia/cuda:10.2-cudnn7-runtime nvidia-smi
2021-01-21 21:45:05 +09:00
2021-06-06 21:56:40 +09:00
GPU_OPTS=(--gpus all --device /dev/nvidia0 --device /dev/nvidiactl --device /dev/nvidia-modeset --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools)
docker run --rm -it ${GPU_OPTS} nvidia/cuda:10.2-cudnn7-runtime nvidia-smi
docker run --rm -it ${GPU_OPTS} tensorflow/tensorflow:1.14.0-gpu-py3 bash
2021-01-21 21:45:05 +09:00
```
2021-06-06 21:56:40 +09:00
### Use `journald` log driver in Docker Compose
```yaml
services:
web:
logging:
driver: "journald"
options:
tag: "{{.ImageName}}/{{.Name}}/{{.ID}}" # default: "{{.ID}}"
```
- [Configure logging drivers | Docker Documentation ](https://docs.docker.com/config/containers/logging/configure/ )
2022-01-17 13:42:14 +09:00
- [Architecture Overview — NVIDIA Cloud Native Technologies documentation ](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/arch-overview.html )
2021-06-06 21:56:40 +09:00
2022-02-05 13:41:25 +09:00
## Backup (restic)
```bash
yay -S restic
```
```ini /etc/systemd/system/restic.service
[Unit]
Description=Restic Backup Service
[Service]
Nice=19
IOSchedulingClass=idle
KillSignal=SIGINT
ExecStart=/etc/restic/run.sh
```
```ini /etc/systemd/system/restic.timer
[Unit]
Description=Restic Backup Timer
[Timer]
OnCalendar=*-*-* 0,6,12,18:0:0
RandomizedDelaySec=15min
Persistent=true
[Install]
WantedBy=timers.target
```
```bash /etc/restic/config.sh
export RESTIC_REPOSITORY=/path/to/backup
export RESTIC_PASSWORD=< passphrase >
export RESTIC_PROGRESS_FPS=1
```
```bash /etc/restic/run.sh
#!/bin/bash -ue
# usage: run.sh
# https://restic.readthedocs.io/en/latest/040_backup.html#
source /etc/restic/config.sh
date
# system
restic backup --tag system -v \
--one-file-system \
--exclude .cache \
--exclude .vscode-server \
--exclude .vscode-server-insiders \
--exclude TabNine \
--exclude /var/lib/docker/overlay2 \
/ /boot
# data
restic backup --tag data -v \
--exclude 'appdata_*/preview' \ # Nextcloud cache
--exclude 'appdata_*/dav-photocache' \ # Nextcloud cache
/mnt/data
# prune
restic forget --prune --group-by tags \
--keep-last 4 \
--keep-within-daily 7d \
--keep-within-weekly 1m \
--keep-within-monthly 3m
# verify
restic check
```
```bash /etc/restic/show.sh
#!/bin/bash
# usage: show.sh <file|directory>
# https://restic.readthedocs.io/en/latest/050_restore.html
source /etc/restic/config.sh
2021-06-06 21:56:40 +09:00
2022-02-05 13:41:25 +09:00
TARGET=${1:-$(pwd)}
MODE="ls -l"
if [[ -f $TARGET ]]; then
TARGET=$(realpath ${TARGET})
MODE=dump
fi
TAG=$(restic snapshots --json | jq -r '[.[].tags[0]] | unique | .[]' | fzy)
ID=$(restic snapshots --tag $TAG --json | jq -r ".[] | [.time, .short_id] | @tsv " | fzy | awk '{print $2}')
>& 2 echo "Command: restic ${MODE} ${ID} ${TARGET}"
restic $MODE $ID ${TARGET}
```
```bash /etc/restic/restore.sh
#!/bin/bash
# https://restic.readthedocs.io/en/latest/050_restore.html
source /etc/restic/config.sh
TARGET=${1:?Specify TARGET}
TARGET=$(realpath ${TARGET})
TAG=$(restic snapshots --json | jq -r '[.[].tags[0]] | unique | .[]' | fzy)
ID=$(restic snapshots --tag $TAG --json | jq -r ".[] | [.time, .short_id] | @tsv " | fzy | awk '{print $2}')
>& 2 echo "Command: restic restore ${ID} -i ${TARGET} -t /"
read -p "Press enter to continue"
restic restore $ID -i ${TARGET} -t /
```
2022-01-17 13:42:14 +09:00
```bash
2022-02-05 13:41:25 +09:00
chmod 700 /etc/restic/config.sh
ln -sf /etc/restic/restic.{service,timer} /etc/systemd/system/
systemctl enable --now restic.timer
2022-01-17 13:42:14 +09:00
```
2022-02-05 13:41:25 +09:00
- [Restic Documentation — restic 0.12.1 documentation ](https://restic.readthedocs.io/en/stable/ )
## VPN (WireGuard)
2022-01-17 13:42:14 +09:00
```bash
2022-02-05 13:41:25 +09:00
pacman -S wireguard-tools
# gen private key
(umask 0077; wg genkey > server.key)
# gen public key
wg pubkey < server.key > server.pub
# gen preshared key for each client
(umask 0077; wg genpsk > secret1.psk)
(umask 0077; wg genpsk > secret2.psk)
...
```
```ini /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.10.1/24
ListenPort = 12345
PrivateKey = < content of server . key >
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o dns-shim -d 10.0.1.100/32 -j MASQUERADE; iptables -t nat -A POSTROUTING -o enp5s0 ! -d 10.0.1.100/32 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o dns-shim -d 10.0.1.100/32 -j MASQUERADE; iptables -t nat -D POSTROUTING -o enp5s0 ! -d 10.0.1.100/32 -j MASQUERADE
[Peer]
PublicKey = < public key >
PresharedKey = < content of secret1 . psk >
AllowedIPs = 10.0.10.2/32
[Peer]
PublicKey = < public key >
PresharedKey = < content of secret2 . psk >
AllowedIPs = 10.0.10.3/32
```
```bash
sysctl -w net.ipv4.ip_forward=1
ufw allow 12345/udp # if ufw is running
systemctl enable --now wg-quick@wg0
wg show # show active settings
2022-01-17 13:42:14 +09:00
```
2022-02-05 13:41:25 +09:00
- https://wiki.archlinux.org/title/WireGuard
- https://github.com/linuxserver/docker-wireguard
- https://philipdeljanov.com/posts/2019/03/21/setting-up-a-wireguard-vpn/
2021-12-14 17:13:26 +09:00
## Fail2ban
2021-01-21 21:45:05 +09:00
```
pacman -S fail2ban
2021-06-11 15:15:02 +09:00
```
```ini /etc/fail2ban/filter.d/bad-auth.conf
[Definition]
failregex = .* client login failed: .+ client:\ < HOST >
ignoreregex =
2022-02-05 13:41:25 +09:00
journalmatch = CONTAINER_NAME=mail-front-1
2021-01-21 21:45:05 +09:00
```
2021-02-25 22:36:19 +09:00
```ini /etc/fail2ban/jail.local
2021-01-21 21:45:05 +09:00
[DEFAULT]
ignoreip = 127.0.0.1/8 10.0.1.0/24
[sshd]
enabled = true
port = 22,10122
2021-07-28 15:22:05 +09:00
bantime = 1h
2021-06-11 15:15:02 +09:00
mode = aggressive
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
# https://mailu.io/1.9/faq.html?highlight=fail2ban#do-you-support-fail2ban
2021-01-21 21:45:05 +09:00
[mailu]
enabled = true
backend = systemd
2021-06-11 15:15:02 +09:00
filter = bad-auth
2022-02-05 13:41:25 +09:00
findtime = 15m
maxretry = 10
2021-07-28 15:22:05 +09:00
bantime = 1w
2022-02-05 13:41:25 +09:00
action = docker-action
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
```ini /etc/fail2ban/action.d/docker-action.conf
[Definition]
actionstart = iptables -N f2b-bad-auth
iptables -A f2b-bad-auth -j RETURN
iptables -I DOCKER-USER -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth
actionstop = iptables -D DOCKER-USER -p tcp -m multiport --dports 1:1024 -j f2b-bad-auth
iptables -F f2b-bad-auth
iptables -X f2b-bad-auth
actioncheck = iptables -n -L DOCKER-USER | grep -q 'f2b-bad-auth[ \t]'
actionban = iptables -I f2b-bad-auth 1 -s < ip > -j DROP
actionunban = iptables -D f2b-bad-auth -s < ip > -j DROP
2021-01-21 21:45:05 +09:00
```
```bash
2021-06-11 15:15:02 +09:00
systemctl enable --now fail2ban
2022-02-05 13:41:25 +09:00
fail2ban-client status
2021-01-21 21:45:05 +09:00
```
2022-02-05 13:41:25 +09:00
## cfddns
Dynamic DNS for Cloudflare.
> Star [the GitHub repository](https://github.com/uetchy/cfddns) if you like it :)
```
yay -S cfddns
```
```yml /etc/cfddns/cfddns.yml
token: < token >
notification:
enabled: true
from: cfddns@localhost
to: me@example .com
```
```ini /etc/cfddns/domains
example.com
dev.example.com
example.org
```
```
systemctl enable --now cfddns
```
## Reverse proxy (nginx-proxy)
```bash
git clone --recurse-submodules https://github.com/evertramos/nginx-proxy-automation.git /srv/proxy
cd /srv/proxy
./fresh-start.sh --yes -e your_email@domain --skip-docker-image-check
```
## Nextcloud
```bash
git clone https://github.com/uetchy/docker-nextcloud.git /srv/cloud
cd /srv/cloud
cp .env.sample .env
vim .env # fill the blank variables
make # pull, build, start
make applypatches # run only once
```
## Monitor (Telegraf + InfluxDB + Grafana)
### Grafana + InfluxDB (Docker)
```bash
git clone https://github.com/uetchy/docker-monitor.git /srv/monitor
cd /srv/monitor
```
### Telegraf (Host)
2021-07-28 15:22:05 +09:00
```bash
yay -S telegraf
```
```ini /etc/telegraf/telegraf.conf
# Global tags can be specified here in key="value" format.
[global_tags]
# Configuration for telegraf agent
[agent]
2021-10-11 12:24:57 +09:00
interval = "15s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
2021-07-28 15:22:05 +09:00
# Read InfluxDB-formatted JSON metrics from one or more HTTP endpoints
[[outputs.influxdb]]
2021-10-11 12:24:57 +09:00
urls = ["http://127.0.0.1:8086"]
database = "< db > "
username = "< user > "
password = "< password > "
2021-07-28 15:22:05 +09:00
# Read metrics about cpu usage
[[inputs.cpu]]
2021-10-11 12:24:57 +09:00
percpu = true
totalcpu = true
collect_cpu_time = false
report_active = false
2021-07-28 15:22:05 +09:00
# Read metrics about disk usage by mount point
[[inputs.disk]]
2021-10-11 12:24:57 +09:00
ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
2021-07-28 15:22:05 +09:00
# Read metrics about disk IO by device
[[inputs.diskio]]
# Get kernel statistics from /proc/stat
[[inputs.kernel]]
# Read metrics about memory usage
[[inputs.mem]]
# Get the number of processes and group them by status
[[inputs.processes]]
# Read metrics about system load & uptime
[[inputs.system]]
# Read metrics about network interface usage
[[inputs.net]]
2021-10-11 12:24:57 +09:00
interfaces = ["enp5s0"]
2021-07-28 15:22:05 +09:00
# Read metrics about docker containers
[[inputs.docker]]
2021-10-11 12:24:57 +09:00
endpoint = "unix:///var/run/docker.sock"
perdevice = false
total = true
2021-07-28 15:22:05 +09:00
[[inputs.fail2ban]]
2021-10-11 12:24:57 +09:00
interval = "15m"
use_sudo = true
2021-07-28 15:22:05 +09:00
# Pulls statistics from nvidia GPUs attached to the host
[[inputs.nvidia_smi]]
2021-10-11 12:24:57 +09:00
timeout = "30s"
2021-07-28 15:22:05 +09:00
[[inputs.http_response]]
2021-10-11 12:24:57 +09:00
interval = "5m"
urls = [
"https://example.com"
]
2021-07-28 15:22:05 +09:00
# Monitor sensors, requires lm-sensors package
[[inputs.sensors]]
2021-10-11 12:24:57 +09:00
interval = "60s"
remove_numbers = false
2021-07-28 15:22:05 +09:00
```
```ini /etc/sudoers.d/telegraf
Cmnd_Alias FAIL2BAN = /usr/bin/fail2ban-client status, /usr/bin/fail2ban-client status *
telegraf ALL=(root) NOEXEC: NOPASSWD: FAIL2BAN
Defaults!FAIL2BAN !logfile, !syslog, !pam_session
```
```bash
chmod 440 /etc/sudoers.d/telegraf
usermod -aG docker telegraf
telegraf -config /etc/telegraf/telegraf.conf -test
systemctl enable --now telegraf
```
2021-12-14 17:13:26 +09:00
## Kubernetes
2021-01-21 21:45:05 +09:00
2021-12-14 17:13:26 +09:00
```bash
pacman -S minikube kubectl
minikube start --cpus=max
2022-01-17 13:42:14 +09:00
kubectl taint nodes --all node-role.kubernetes.io/master- # to allow the control plane to allocate pods to the master node
2021-12-14 17:13:26 +09:00
minikube ip
kubectl cluster-info
kubectl get cm -n kube-system kubeadm-config -o yaml
2021-01-21 21:45:05 +09:00
```
2021-12-14 17:13:26 +09:00
- [Kubernetes - ArchWiki ](https://wiki.archlinux.org/index.php/Kubernetes )
- [Kubernetes Ingress Controller with NGINX Reverse Proxy and Wildcard SSL from Let's Encrypt - Shogan.tech ](https://www.shogan.co.uk/kubernetes/kubernetes-ingress-controller-with-nginx-reverse-proxy-and-wildcard-ssl-from-lets-encrypt/ )
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
## Firewall (ufw)
```bash
pacman -S ufw
systemctl enable --now ufw
```
See:
- https://wiki.archlinux.org/title/Uncomplicated_Firewall
- [Introduction to Netfilter – To Linux and beyond ! ](https://home.regit.org/netfilter-en/netfilter/ )
2021-12-14 17:13:26 +09:00
## Audio
2021-01-21 21:45:05 +09:00
```bash
2022-02-05 13:41:25 +09:00
pacman -S alsa-utils # may require rebooting system
2021-07-28 15:22:05 +09:00
usermod -aG audio < user >
2021-06-06 21:56:40 +09:00
# list devices as root
aplay -l
arecord -L
cat /proc/asound/cards
# test speaker
speaker-test -c2
# test mic
arecord -vv -Dhw:2,0 -fS32_LE mic.wav
aplay mic.wav
# gui mixer
alsamixer
# for Mycroft.ai
pacman -S pulseaudio pulsemixer
pulseaudio --start
pacmd list-cards
```
```conf /etc/pulse/default.pa
# INPUT/RECORD
load-module module-alsa-source device="default" tsched=1
# OUTPUT/PLAYBACK
load-module module-alsa-sink device="default" tsched=1
# Accept clients -- very important
load-module module-native-protocol-unix
load-module module-native-protocol-tcp
2021-02-25 22:36:19 +09:00
```
2021-01-21 21:45:05 +09:00
2021-02-25 22:36:19 +09:00
```conf /etc/asound.conf
2021-06-06 21:56:40 +09:00
pcm.mic {
2021-01-21 21:45:05 +09:00
type hw
card M96k
rate 44100
format S32_LE
}
2021-06-06 21:56:40 +09:00
pcm.speaker {
2021-01-21 21:45:05 +09:00
type plug
2021-06-06 21:56:40 +09:00
slave {
pcm "hw:1,0"
}
2021-01-21 21:45:05 +09:00
}
2021-06-06 21:56:40 +09:00
pcm.!default {
type asym
capture.pcm "mic"
playback.pcm "speaker"
}
#defaults.pcm.card 1
#defaults.ctl.card 1
2021-01-21 21:45:05 +09:00
```
2021-06-06 21:56:40 +09:00
- [PulseAudio as a minimal unintrusive dumb pipe to ALSA ](https://wiki.archlinux.org/title/PulseAudio/Examples#PulseAudio_as_a_minimal_unintrusive_dumb_pipe_to_ALSA )
2021-01-21 21:45:05 +09:00
- [SoundcardTesting - AlsaProject ](https://www.alsa-project.org/main/index.php/SoundcardTesting )
- [Advanced Linux Sound Architecture/Troubleshooting - ArchWiki ](https://wiki.archlinux.org/index.php/Advanced_Linux_Sound_Architecture/Troubleshooting#Microphone )
- [ALSA project - the C library reference: PCM (digital audio) plugins ](https://www.alsa-project.org/alsa-doc/alsa-lib/pcm_plugins.html )
2021-06-06 21:56:40 +09:00
- [Asoundrc - AlsaProject ](https://www.alsa-project.org/wiki/Asoundrc )
2021-01-21 21:45:05 +09:00
2022-02-05 13:41:25 +09:00
## Telegram notifier
2021-07-28 15:22:05 +09:00
2022-02-05 13:41:25 +09:00
```bash /usr/local/bin/telegram-notifier
#!/bin/bash
BOT_TOKEN=< your bot token >
CHAT_ID=< your chat id >
PAYLOAD=$(ruby -r json -e "print ({text: ARGF.to_a.join, chat_id: $CHAT_ID}).to_json" < /dev/stdin)
OK=$(curl -s -X "POST" \
-H "Content-Type: application/json; charset=utf-8" \
-d "$PAYLOAD" \
https://api.telegram.org/bot${BOT_TOKEN}/sendMessage | jq .ok)
if [[ $OK == true ]]; then
exit 0
else
exit 1
fi
2021-07-28 15:22:05 +09:00
```
2022-02-05 13:41:25 +09:00
## LUKS
<!-- I've published a Vagrantfile for playing around LUKS. -->
### Encrypt existing un-encrypted root partition
Boot Arch Linux from archiso USB device, then:
```bash
pacman -Syu
resize2fs -p -M /dev/sdaX
# resize2fs -p /dev/sdaX 1838M
cryptsetup reencrypt --encrypt --reduce-device-size 32M /dev/sdaX
cryptsetup --allow-discards --perf-no_read_workqueue --perf-no_write_workqueue --persistent open /dev/sdaX root
resize2fs /dev/mapper/root
mkdir /mnt/{root,boot}
mount /dev/sda1 /mnt/boot
mount /dev/mapper/root /mnt/root
systemd-nspawn --boot --bind=/mnt/boot:/boot --directory=/mnt/root
# add sysroot line
vim /etc/mkinitcpio-systemd-tool/config/crypttab
vim /etc/mkinitcpio-systemd-tool/config/fstab
mkinitcpio -P
reboot
```
2021-07-28 15:22:05 +09:00
2021-01-21 21:45:05 +09:00
# Maintenance
2021-12-14 17:13:26 +09:00
## Quick checkups
2021-06-06 21:56:40 +09:00
2021-01-21 21:45:05 +09:00
```bash
2021-07-28 15:22:05 +09:00
htop # show task overview
2021-06-06 21:56:40 +09:00
systemctl --failed # show failed units
free -h # show memory usage
lsblk -f # show disk usage
networkctl status # show network status
userdbctl # show users
nvidia-smi # verify nvidia cards
2021-07-28 15:22:05 +09:00
ps aux | grep "defunct" # find zombie processes
2021-06-06 21:56:40 +09:00
```
2021-12-14 17:13:26 +09:00
## Delve into system logs
2021-06-06 21:56:40 +09:00
```bash
journalctl -p err -b-1 -r # show error logs from previous boot in reverse order
2021-07-28 15:22:05 +09:00
journalctl -u sshd -f # tail logs from sshd unit
journalctl --no-pager -n 25 -k # show latest 25 logs from the kernel without pager
journalctl --since=yesterday --until "2020-07-10 15:10:00" # show logs within specific time range
2022-02-05 13:41:25 +09:00
journalctl CONTAINER_NAME=service_web_1 # show error from the docker container named 'service_web_1'
2021-07-28 15:22:05 +09:00
journalctl _PID=2434 -e # filter logs based on PID and jump to the end of the logs
2022-02-05 13:41:25 +09:00
journalctl -g 'timed out' # filter logs based on a regular expression. if the pattern is all lowercase, it will become insensitive mode
2021-07-28 15:22:05 +09:00
```
2021-12-14 17:13:26 +09:00
- g - go to the first line
- G - go to the last line
- / - search for the string
2021-07-28 15:22:05 +09:00
2021-12-14 17:13:26 +09:00
## Force overriding installation
2021-07-28 15:22:05 +09:00
```bash
pacman -S < pkg > --overwrite '*'
```
2021-12-14 17:13:26 +09:00
## Check memory modules
```bash
pacman -S lshw dmidecode
lshw -short -C memory # lists installed mems
dmidecode # shows configured clock speed
```
## File-system related issues checklist
2021-07-28 15:22:05 +09:00
```bash
smartctl -H /dev/sdd
2021-12-14 17:13:26 +09:00
# umount the drive before this ops
2022-02-05 13:41:25 +09:00
# [!] Never perform `fsck` on an unmounted LUKS drive, as it may lead to data loss
2021-07-28 15:22:05 +09:00
e2fsck -C 0 -p /dev/sdd1 # preen
e2fsck -C 0 -cc /dev/sdd1 # badblocks
2021-01-21 21:45:05 +09:00
```
2021-06-06 21:56:40 +09:00
2022-02-05 13:41:25 +09:00
# Troubleshooting
2021-06-06 21:56:40 +09:00
## Longer SSH login (D-bus glitch)
```bash
systemctl restart systemd-logind
systemctl restart polkit
```
- [A comprehensive guide to fixing slow SSH logins – JRS Systems: the blog ](https://jrs-s.net/2017/07/01/slow-ssh-logins/ )
2022-02-05 13:41:25 +09:00
## Annoying `systemd-homed is not available` flooding journald logs
2021-06-06 21:56:40 +09:00
Move `pam_unix` before `pam_systemd_home` .
```ini /etc/pam.d/system-auth
#%PAM-1.0
auth required pam_faillock.so preauth
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth [success=2 default=ignore] pam_unix.so try_first_pass nullok
-auth [success=1 default=ignore] pam_systemd_home.so
auth [default=die] pam_faillock.so authfail
auth optional pam_permit.so
auth required pam_env.so
auth required pam_faillock.so authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.
account [success=1 default=ignore] pam_unix.so
-account required pam_systemd_home.so
account optional pam_permit.so
account required pam_time.so
password [success=1 default=ignore] pam_unix.so try_first_pass nullok shadow
-password required pam_systemd_home.so
password optional pam_permit.so
session required pam_limits.so
session required pam_unix.so
session optional pam_permit.so
```
- [[solved] pam fails to find unit dbus-org.freedesktop.home1.service / Newbie Corner / Arch Linux Forums](https://bbs.archlinux.org/viewtopic.php?id=258297)
## Annoying systemd-journald-audit log
```ini /etc/systemd/journald.conf
Audit=no
```
## Missing `/dev/nvidia-{uvm*,modeset}`
2022-02-05 13:41:25 +09:00
This usually happens right after updating the Linux kernel.
2021-06-11 15:15:02 +09:00
2021-10-11 12:24:57 +09:00
- Run `docker run --rm --gpus all --device /dev/nvidia0 --device /dev/nvidiactl --device /dev/nvidia-modeset --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools -it nvidia/cuda:10.2-cudnn7-runtime nvidia-smi` once.
2021-06-06 21:56:40 +09:00
2021-12-14 17:13:26 +09:00
## `[sudo] Incorrect password` while password is correct
```bash
faillock --reset
```
2021-06-06 21:56:40 +09:00
# Useful links
- [General recommendations ](https://wiki.archlinux.org/index.php/General_recommendations#Users_and_groups )
- [System maintenance ](https://wiki.archlinux.org/index.php/System_maintenance )
- [Improving performance ](https://wiki.archlinux.org/index.php/Improving_performance#Know_your_system )
2021-07-28 15:22:05 +09:00
- [General troubleshooting - ArchWiki ](https://wiki.archlinux.org/title/General_troubleshooting )
- [Stress testing - ArchWiki ](https://wiki.archlinux.org/title/Stress_testing#Stressing_memory )
- [udev - ArchWiki ](https://wiki.archlinux.org/title/Udev#Debug_output )
- [[HOWTO] Repair Broken system, system without a kernel / Forum & Wiki discussion / Arch Linux Forums](https://bbs.archlinux.org/viewtopic.php?id=18066)
- [Archboot - ArchWiki ](https://wiki.archlinux.org/title/Archboot )