Unix Services

This chapter covers a number of basic services that are common to many Unix systems. All administrators should be familiar with them.

System Boot

booting

the system

BIOS

MBR

ESP

Compatibility Support Mode

CSM

CSM

When you boot the computer, the many messages scrolling by on the console display many automatic initializations and configurations that are being executed. Sometimes you may wish to slightly alter how this stage works, which means that you need to understand it well. That is the purpose of this section.

On systems with a BIOS, first, the BIOS takes control of the computer, initializes the controllers and hardware, detects the disks, and bridges everything together. Then it looks up the Master Boot Record (MBR) of the first disk in the boot order and loads the code stored there (first stage). This code then launches the second stage and finally executes the bootloader.

In contrast to the BIOS, UEFI is more sophisticated, it knows filesystems and can read the partition tables. The interface searches the system storage for a partition labeled with a specific globally unique identifier (GUID) that marks it as the EFI System Partition (ESP), where the bootloaders, boot managers, UEFI shell, etc., are located, and launches the desired bootloader. If Secure Boot is enabled, the boot process will verify authenticity of the EFI binaries there by signature (thus grub-efi-<arch>-signed is required in this case). The UEFI specification also defines support for booting in legacy BIOS mode. This is called the Compatibility Support Module (CSM). If CSM is enabled, it will attempt to boot from a drive's MBR. However, many new systems do no longer support the CSM mode.

In both cases then the actual bootloader takes over, finds either a chained bootloader or the kernel on the disk, loads, and executes it. The kernel is then initialized, and starts to search for and mount the partition containing the root filesystem, and finally executes the first program — init. Frequently, this “root partition” and this init are, in fact, located in a virtual filesystem that only exists in RAM (hence its name, “initramfs”, formerly called “initrd” for “initialization RAM disk”). This filesystem is loaded in memory by the bootloader, often from a file on a hard drive or from the network. It contains the bare minimum required by the kernel to load the “true” root filesystem: this may be driver modules for the hard drive, or other devices without which the system cannot boot, or, more frequently, initialization scripts and modules for assembling RAID arrays, opening encrypted partitions, activating LVM volumes, etc. Once the root partition is mounted, the initramfs hands over control to the real init, and the machine goes back to the standard boot process.

The systemd init system

init

The “real init” is currently provided by systemd and this section documents this init system.

Systemd executes several processes, in charge of setting up the system: keyboard, drivers, filesystems, network, services. It does this while keeping a global view of the system as a whole, and the requirements of the components. Each component is described by a “unit file” (sometimes more); the general syntax is derived from the widely-used “*.ini files“ syntax, with key = value pairs grouped between [section] headers. Unit files are stored under /lib/systemd/system/ and /etc/systemd/system/; they come in several flavors, but we will focus on “services” and “targets” here.

systemd

.service

file

/etc

/etc/systemd/system/

systemd

service

ssh.service

A systemd “.service file” describes a process managed by systemd. It contains roughly the same information as old-style init-scripts, but expressed in a declaratory (and much more concise) way. Systemd handles the bulk of the repetitive tasks (starting and stopping the process, checking its status, logging, dropping privileges, and so on), and the service file only needs to fill in the specifics of the process. For instance, here is the service file for SSH:

[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

The [Unit] section contains generic information about the service, like its description and manual page resources, as well as relations (dependency and order) to other services. The [Service] part contains the declarations related to the service execution (starting, stopping, killing, restarting), directories and configuration file(s) used. The last section, [Install], again carries generic information into which targets to install the service and, in this case, the alias that can be used instead of the service name. As you can see, there is very little code in there, only declarations. Systemd takes care of displaying progress reports, keeping track of the processes, and even restarting them when needed. The syntax of these files is fully described in several manual pages (e.g. systemd.service(5), systemd.unit(5), systemd.exec(5), etc.).

systemd

.target

file

A systemd “.target file” describes a state of the system where a set of services are known to be operational. It can be thought of as an equivalent of the old-style runlevel. One of the pre-defined targets is local-fs.target; when it is reached, the rest of the system can assume that all local filesystems are mounted and accessible. Other targets include network-online.target and sound.target (for a full list of special targets see systemd.special(7)). The dependencies of a target can be listed either within the target file (in the Requires= line), or using a symbolic link to a service file in the /lib/systemd/system/targetname.target.wants/ directory. For instance, /etc/systemd/system/printer.target.wants/ contains a link to /lib/systemd/system/cups.service; systemd will therefore ensure CUPS is running in order to reach printer.target.

Since unit files are declarative rather than scripts or programs, they cannot be run directly, and they are only interpreted by systemd; several utilities therefore allow the administrator to interact with systemd and control the state of the system and of each component.

systemd

systemctl

systemctl

systemctl

status

The first such utility is systemctl. When run without any arguments, it lists all the unit files known to systemd (except those that have been disabled), as well as their status. systemctl status gives a better view of the services, as well as the related processes. If given the name of a service (as in systemctl status ntp.service), it returns even more details, as well as the last few log lines related to the service (more on that later).

systemctl

start

systemctl

stop

systemctl

reload

systemctl

restart

Starting a service by hand is a simple matter of running systemctl start servicename.service. As one can guess, stopping the service is done with systemctl stop servicename.service; other subcommands include reload and restart.

systemctl

enable

systemctl

disable

systemctl

is-enabled

To control whether a service is active (i.e. whether it will get started automatically on boot), use systemctl enable servicename.service (or disable). is-enabled allows checking the status of the service.

systemd

journald

journald

systemd

journalctl

journalctl

journald

syslogd

journalctl

An interesting feature of systemd is that it includes a logging component named journald. It comes as a complement to more traditional logging systems such as syslogd, but it adds interesting features such as a formal link between a service and the messages it generates, and the ability to capture error messages generated by its initialization sequence. The messages can be displayed later on, with a little help from the journalctl command. Without any arguments, it simply spews all log messages that occurred since system boot; it will rarely be used in such a manner. Most of the time, it will be used with a service identifier:

# journalctl -u ssh.service

May 20 19:28:59 debian systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 20 19:28:59 debian sshd[532]: Server listening on 0.0.0.0 port 22.
May 20 19:28:59 debian sshd[532]: Server listening on :: port 22.
May 20 19:28:59 debian systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
May 20 19:34:17 debian sshd[532]: Received signal 15; terminating.
May 20 19:34:17 debian systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
May 20 19:34:17 debian systemd[1]: ssh.service: Deactivated successfully.
May 20 19:34:17 debian systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
-- Boot 539ead7492344c2baddeea6bba357eee --
May 20 19:36:56 debian systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 20 19:36:56 debian sshd[532]: Server listening on 0.0.0.0 port 22.
May 20 19:36:56 debian sshd[532]: Server listening on :: port 22.
May 20 19:36:56 debian systemd[1]: Started ssh.service - OpenBSD Secure Shell server.

Another useful command-line flag is -f, which instructs journalctl to keep displaying new messages as they are emitted (much in the manner of tail -f file).

systemctl

status

If a service doesn't seem to be working as expected, the first step to solve the problem is to check that the service is actually running with systemctl status; if it is not, and the messages given by the first command are not enough to diagnose the problem, check the logs gathered by journald about that service. For instance, assume the SSH server doesn't work:

# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
   Active: failed (Result: start-limit) since Tue 2015-03-31 17:30:36 CEST; 1s ago
  Process: 1023 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 1188 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
 Main PID: 1188 (code=exited, status=255)

Mar 31 17:30:36 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:36 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:36 mirtuel systemd[1]: ssh.service start request repeated too quickly, refusing to start.
Mar 31 17:30:36 mirtuel systemd[1]: Failed to start OpenBSD Secure Shell server.
Mar 31 17:30:36 mirtuel systemd[1]: Unit ssh.service entered failed state.
# journalctl -u ssh.service
-- Logs begin at Tue 2015-03-31 17:29:27 CEST, end at Tue 2015-03-31 17:30:36 CEST. --
Mar 31 17:29:27 mirtuel sshd[424]: Server listening on 0.0.0.0 port 22.
Mar 31 17:29:27 mirtuel sshd[424]: Server listening on :: port 22.
Mar 31 17:29:29 mirtuel sshd[424]: Received SIGHUP; restarting.
Mar 31 17:29:29 mirtuel sshd[424]: Server listening on 0.0.0.0 port 22.
Mar 31 17:29:29 mirtuel sshd[424]: Server listening on :: port 22.
Mar 31 17:30:10 mirtuel sshd[1147]: Accepted password for roland from 192.168.1.129 port 38742 ssh2
Mar 31 17:30:10 mirtuel sshd[1147]: pam_unix(sshd:session): session opened for user roland by (uid=0)
Mar 31 17:30:35 mirtuel sshd[1180]: /etc/ssh/sshd_config line 28: unsupported option "yess".
Mar 31 17:30:35 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:35 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:35 mirtuel sshd[1182]: /etc/ssh/sshd_config line 28: unsupported option "yess".
Mar 31 17:30:35 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:35 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:35 mirtuel sshd[1184]: /etc/ssh/sshd_config line 28: unsupported option "yess".
Mar 31 17:30:35 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:35 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:36 mirtuel sshd[1186]: /etc/ssh/sshd_config line 28: unsupported option "yess".
Mar 31 17:30:36 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:36 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:36 mirtuel sshd[1188]: /etc/ssh/sshd_config line 28: unsupported option "yess".
Mar 31 17:30:36 mirtuel systemd[1]: ssh.service: main process exited, code=exited, status=255/n/a
Mar 31 17:30:36 mirtuel systemd[1]: Unit ssh.service entered failed state.
Mar 31 17:30:36 mirtuel systemd[1]: ssh.service start request repeated too quickly, refusing to start.
Mar 31 17:30:36 mirtuel systemd[1]: Failed to start OpenBSD Secure Shell server.
Mar 31 17:30:36 mirtuel systemd[1]: Unit ssh.service entered failed state.
# vi /etc/ssh/sshd_config
# systemctl start ssh.service
# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
   Active: active (running) since Tue 2015-03-31 17:31:09 CEST; 2s ago
  Process: 1023 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 1222 (sshd)
   CGroup: /system.slice/ssh.service
           └─1222 /usr/sbin/sshd -D
# 

After checking the status of the service (failed), we went on to check the logs; they indicate an error in the configuration file. After editing the configuration file and fixing the error, we restart the service, then verify that it is indeed running.

The System V init system

init

System V init

System V init

The System V init system (which we'll call init for brevity) executes several processes, following instructions from the /etc/inittab file. The first program that is executed (which corresponds to the sysinit step) is /etc/init.d/rcS, a script that executes all of the programs in the /etc/rcS.d/ directory.

/etc

/etc/init.d/rcS

/etc

/etc/rcS.d/

rcS

rcS.d

Among these, you will find successively programs in charge of:

• configuring the console's keyboard;

• loading drivers: most of the kernel modules are loaded by the kernel itself as the hardware is detected; extra drivers are then loaded automatically when the corresponding modules are listed in /etc/modules;

  /etc /etc/modules

• checking the integrity of filesystems;

• mounting local partitions;

• configuring the network;

• mounting network filesystems (NFS).

/etc

/etc/init.d/rc

/etc

/etc/rcX.d/

insserv

After this stage, init takes over and starts the programs enabled in the default runlevel (which is usually runlevel 2). It executes /etc/init.d/rc 2, a script that starts all services which are listed in /etc/rc2.d/ and whose names start with the “S” letter. The two-figures number that follows had historically been used to define the order in which services had to be started, but nowadays the default boot system uses insserv, which schedules everything automatically based on the scripts' dependencies. Each boot script thus declares the conditions that must be met to start or stop the service (for example, if it must start before or after another service); init then launches them in the order that meets these conditions. The static numbering of scripts is therefore no longer taken into consideration (but they must always have a name beginning with “S” followed by two digits and the actual name of the script used for the dependencies). Generally, base services (such as logging with rsyslog, or port assignment with portmap) are started first, followed by standard services and the graphical interface (gdm3).

This dependency-based boot system makes it possible to automate re-numbering, which could be rather tedious if it had to be done manually, and it limits the risks of human error, since scheduling is conducted according to the parameters that are indicated. Another benefit is that services can be started in parallel when they are independent from one another, which can accelerate the boot process.

runlevel

level, runlevel

telinit

init distinguishes several runlevels, so it can switch from one to another with the telinit new-level command. Immediately, init executes /etc/init.d/rc again with the new runlevel. This script will then start the missing services and stop those that are no longer desired. To do this, it refers to the content of the /etc/rcX.d (where <X> represents the new runlevel). Scripts starting with “S” (as in “Start”) are services to be started; those starting with “K” (as in “Kill”) are the services to be stopped. The script does not start any service that was already active in the previous runlevel.

System V init

runlevels

By default, System V init in Debian uses four different runlevels:

• Level 0 is only used temporarily, while the computer is powering down. As such, it only contains many “K” scripts.

• Level 1, also known as single-user mode, corresponds to the system in degraded mode; it includes only basic services, and is intended for maintenance operations where interactions with ordinary users are not desired.

• Level 2 is the level for normal operation, which includes networking services, a graphical interface, user logins, etc.

• Level 6 is similar to level 0, except that it is used during the shutdown phase that precedes a reboot.

Other levels exist, especially 3 to 5. By default they are configured to operate the same way as level 2, but the administrator can modify them (by adding or deleting scripts in the corresponding /etc/rcX.d directories) to adapt them to particular needs.

initialization script

update-rc.d

rcconf

All the scripts contained in the various /etc/rcX.d directories are really only symbolic links — created upon package installation by the update-rc.d program — pointing to the actual scripts which are stored in /etc/init.d/. The administrator can fine tune the services available in each runlevel by re-running update-rc.d with adjusted parameters. The update-rc.d(1) manual page describes the syntax in detail. Please note that removing all symbolic links (with the remove parameter) is not a good method to disable a service. Instead you should simply configure it to not start in the desired runlevel (while preserving the corresponding calls to stop it in the event that the service runs in the previous runlevel). Since update-rc.d has a somewhat convoluted interface, you may prefer using rcconf (from the rcconf package) which provides a more user-friendly interface.

Finally, init starts control programs for various virtual consoles (getty). It displays a prompt, waiting for a username, then executes login user to initiate a session.

getty

service

getty@.service

Remote Login

It is essential for an administrator to be able to connect to a computer remotely. Servers, confined in their own room, are rarely equipped with permanent keyboards and monitors — but they are connected to the network.

login

remote login

remote login

Secure Remote Login: SSH

SSH

server

SSH

ssh

Secure Shell

SSH

The SSH (Secure SHell) protocol was designed with security and reliability in mind. Connections using SSH are secure: the partner is authenticated and all data exchanges are encrypted.

scp

SSH also offers two file transfer services. scp is a command line tool that can be used like cp, except that any path to another machine is prefixed with the machine's name, followed by a colon.

$ scp file machine:/tmp/

sftp

sftp is an interactive command, similar to ftp. In a single session, sftp can transfer several files, and it is possible to manipulate remote files with it (delete, rename, change permissions, etc.). Many FTP clients, including filezilla, support it.

OpenSSH

SSH

SSH

OpenSSH

Debian uses OpenSSH, a free version of SSH maintained by the OpenBSD project (a free operating system based on the BSD kernel, focused on security) and fork of the original SSH software developed by the SSH Communications Security Corp company, of Finland. This company initially developed SSH as free software, but eventually decided to continue its development under a proprietary license. The OpenBSD project then created OpenSSH to maintain a free version of SSH.

OpenSSH is split into two packages: the client part is in the openssh-client package, and the server is in the openssh-server package. The ssh meta-package depends on both parts and facilitates installation of both (apt install ssh), while the task-ssh-server, often chosen during the initial installation, depends on the server package only.

Key-Based Authentication

SSH

key creation

Each time someone logs in over SSH, the remote server asks for a password to authenticate the user. This can be problematic if you want to automate a connection, or if you use a tool that requires frequent connections over SSH. This is why SSH offers a key-based authentication system.

ssh-keygen

ssh-copy-id

The user generates a key pair on the client machine with ssh-keygen -t rsa; the so generated public key is stored in ~/.ssh/id_rsa.pub, while the corresponding private key is stored in ~/.ssh/id_rsa. The user can then use ssh-copy-id server to add their public key to the ~/.ssh/authorized_keys file on the server, or, if SSH access hasn't been enabled yet, they have to ask the administrator to add their key manually.

SSH

passphrase cache

SSH

passwordless

SSH

authentication agent

ssh-agent

X.org

/etc/X11/Xsession.options

If the private key was not protected with a “passphrase” at the time of its creation, all subsequent logins on the server will work without a password. Otherwise, the private key must be decrypted each time by entering the passphrase. Fortunately, ssh-agent allows us to keep private keys in memory to not have to regularly re-enter the password. For this, you simply use ssh-add (once per work session) provided that the session is already associated with a functional instance of ssh-agent. Debian activates it by default in graphical sessions, but this can be deactivated by changing /etc/X11/Xsession.options and commenting out use-ssh-agent. For a console session, you can manually start the agent with eval $(ssh-agent).

Cert-Based Authentication

SSH

signed key

SSH

TrustedUserCAKeys

SSH

HostCertificate

/etc

/etc/ssh/sshd_config

/etc

/etc/ssh/known_hosts

SSH keys cannot just be protected by a password (or not). An often unknown feature is that they can also be signed via certificate, both the host as well as the client keys. This approach comes with several advantages. Instead of maintaining an authorized_keys file per user as described in the previous section, the SSH server can be configured to trust all client keys signed by the same certificate (see also ???) by using the TrustedUserCAKeys and HostCertificate directives in /etc/ssh/sshd_config.

TrustedUserCAKeys /etc/ssh/ssh_users_ca.pub

HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub

Vice-versa the clients can also be configured to trust the host key signed by the same authority, making it easier to maintain the known_hosts file (even system wide via /etc/ssh/known_hosts).

@cert-authority *.falcot.com ssh-rsa AAAA[..]

Both, public key and certificate authentication, can be used alongside each other.

Combining authentication methods

SSH

AuthenticationMethods

SSH

PreferredAuthentications

Besides the already mentioned password based, key based, and the certificate based authentication, other methods like using a TOTP exist as well. Not only is this an abundance of options. They can also be combined. On the server side, the AuthenticationMethods directive can define an order of authentication methods a user has to successfully pass, before they are allowed to enter the system. It is even possible to define multiple alternative sequences:

AuthenticationMethods publickey,password publickey,keyboard-interactive:pam

This setting, for example, allows users to login after initially completing the public key authentication, followed by either a successful password authentication or entering a valid TOTP set up via PAM. On the user side, the sequence of methods can be defined using the PreferredAuthentications directive.

Using Remote X11 Applications

SSH

X11Forwarding

SSH

remote graphical applications

remote graphical application

application, remote graphical application

/etc

/etc/ssh/sshd_config

The SSH protocol allows forwarding of graphical data (“X11” session, from the name of the most widespread graphical system in Unix); the server then keeps a dedicated channel for those data. Specifically, a graphical program executed remotely can be displayed on the X.org server of the local screen, and the whole session (input and display) will be secure. Since this feature allows remote applications to interfere with the local system, it is disabled by default. You can enable it by specifying X11Forwarding yes in the server configuration file sshd_config(5) or by prepending the public key with the X11-forwarding keyword in the authorized_keys(5) file. Finally, the user must also request it by adding the -X option to the ssh command-line.

Creating Encrypted Tunnels with Port Forwarding

port forwarding

SSH

port forwarding

SSH

tunnel

Its -R and -L options allow ssh to create “encrypted tunnels” between two machines, securely forwarding a local TCP port (see sidebar ???) to a remote machine or vice versa.

ssh -L 8000:server:25 intermediary establishes an SSH session with the <intermediary> host and listens to local port 8000 (see Forwarding a local port with SSH). For any connection established on this port, ssh will initiate a connection from the <intermediary> computer to port 25 on the <server>, and will bind both connections together.

ssh -R 8000:server:25 intermediary also establishes an SSH session to the <intermediary> computer, but it is on this machine that ssh listens to port 8000 (see Forwarding a remote port with SSH). Any connection established on this port will cause ssh to open a connection from the local machine on to port 25 of the <server>, and to bind both connections together.

In both cases, connections are made to port 25 on the <server> host, which pass through the SSH tunnel established between the local machine and the <intermediary> machine. In the first case, the entrance to the tunnel is local port 8000, and the data move towards the <intermediary> machine before being directed to the <server> on the “public” network. In the second case, the input and output in the tunnel are reversed; the entrance is port 8000 on the <intermediary> machine, the output is on the local host, and the data are then directed to the <server>. In practice, the server is usually either the local machine or the intermediary. That way SSH secures the connection from one end to the other.

Using Remote Graphical Desktops

VNC (Virtual Network Computing) allows remote access to graphical desktops.

VNC

Virtual Network Computing

VNC

graphical desktop

remote

remote graphical desktop

desktop, remote graphical desktop

This tool is mostly used for technical assistance; the administrator can see the errors that the user is facing, and show them the correct course of action without having to stand by them.

VNC

server

server

VNC

VNC

VNC

VNC

VNC

VNC

First, the user must authorize sharing their session. The GNOME graphical desktop environment includes that option via Settings > Sharing (contrary to previous versions of Debian, where the user had to install and run vino). For this to work network-manager must be managing the network used (e.g. enable the managed mode for devices handled by ifupdown in /etc/NetworkManager/NetworkManager.conf). KDE Plasma still requires using krfb to allow sharing an existing session over VNC. For other graphical desktop environments, the x11vnc or tightvncserver commands (from the Debian packages of the same name) or tigervncserver (tigervnc-standalone-server) serve the same purpose and provide the vnc-server virtual package; you can make either of them available to the user with an explicit menu or desktop entry.

VNC

client

VNC

VNC

VNC

VNC

When the graphical session is made available by VNC, the administrator must connect to it with a VNC client. GNOME has vinagre and remmina for that, while the KDE project provides krdc (in the menu at K > Internet > Remote Desktop Client). There are other VNC clients that use the command line, such as xtightvncviewer from the homonym package or xtigervncviewer from the tigervnc-viewer Debian package. Once connected, the administrator can see what is going on, work on the machine remotely, and show the user how to proceed.

/etc

/etc/gdm3/daemon.conf

/etc

/etc/inetd.conf

Managing Rights

permissions

rights

permissions

device

files

Linux is definitely a multi-user system, so it is necessary to provide a permission system to control the set of authorized operations on files and directories, which includes all the system resources and devices (on a Unix system, any device is represented by a file or directory). This principle is common to all Unix systems, but a reminder is always useful, especially as there are some interesting and relatively unknown advanced uses.

Owners and Permissions

user

owner

group

owner

owner

user

owner

group

Each file or directory has specific permissions for three categories of users:

• permissions u its owner (symbolized by u as in “user”);
• permissions g its owner group (symbolized by g as in “group”), representing all the members of the group;
• permissions o the others (symbolized by o as in “other”).

Three basic types of rights can be combined:

• permissions r reading (symbolized by r as in “read”);
• permissions w writing (or modifying, symbolized by w as in “write”);
• permissions x executing (symbolized by x as in “eXecute”).

read, right

write, right

modification, right

execution, right

permissions

read

permissions

write

permissions

execute

In the case of a file, these rights are easily understood: read access allows reading the content (including copying), write access allows changing it, and execute access allows you to run it (which will only work if it is a program).

A directory is handled differently. Read access gives the right to consult the list of its entries (files and directories), write access allows creating or deleting files, and execute access allows “traversal“, the possibility crossing through it (especially to go there with the cd command). Being able to cross through a directory without being able to read it gives permission to access the entries therein that are known by name, but not to find them if you do not know their existence or their exact name.

Three commands control the permissions associated with a file:

• chown chown user file changes the owner of the file;
• chgrp chgrp group file alters the owner group;
• chmod chmod rights file changes the permissions for the file.

symbolic representation of rights

rights

symbolic representation

permissions

symbolic representation

There are two ways of presenting rights. Among them, the symbolic representation is probably the easiest to understand and remember. It involves the letter symbols mentioned above. You can define rights for each category of users (u/g/o), by setting them explicitly (with =), by adding (+), or subtracting (-). Thus the u=rwx,g+rw,o-r formula gives the owner read, write, and execute rights, adds read and write rights for the owner group, and removes read rights for other users. Rights not altered by the addition or subtraction in such a command remain unmodified. The letter a, for “all”, covers all three categories of users, so that a=rx grants all three categories the same rights (read and execute, but not write).

octal representation of rights

rights

octal representation

permissions

octal representation

The (octal) numeric representation associates each right with a value: 4 for read, 2 for write, and 1 for execute. We associate each combination of rights with the sum of the figures. Each value is then assigned to different categories of users by putting them end to end in the usual order (owner, group, others).

For instance, the chmod 754 file command will set the following rights: read, write and execute for the owner (since 7 = 4 + 2 + 1); read and execute for the group (since 5 = 4 + 1); read-only for others. The 0 (zero) means no rights; thus chmod 600 file allows for read/write rights for the owner, and no rights for anyone else. The most frequent right combinations are 755 for executable files and directories, and 644 for data files.

setuid

setgid

sticky bit

To represent special rights, you can prefix a fourth digit to this number according to the same principle, where the setuid, setgid and sticky bits are 4, 2 and 1, respectively. chmod 4754 will associate the setuid bit with the previously described rights.

Note that the use of octal notation only permits setting all file permissions at once; it does not allow you to change a single permission in a simple way, such as adding read access for the group owner, since you must take into account the existing permissions and compute the new corresponding numerical value.

$ ls test.txt
  -rw-r--r-- 1 daniel daniel 0 18. Nov 01:07 test.txt
  $ chmod u+X test.txt
  $ ls test.txt
  -rw-r--r-- 1 daniel daniel 0 18. Nov 01:07 test.txt
  $ chmod o+x test.txt
  $ ls test.txt
  -rw-r--r-x 1 daniel daniel 0 18. Nov 01:07 test.txt
  $ chmod u+X test.txt
  $ ls test.txt
  -rwxr--r-x 1 daniel daniel 0 18. Nov 01:07 test.txt

ACLs - Access Control Lists

ACL

Access Control List

ACL

Many filesystems, e.g. Btrfs, Ext3, Ext4, JFS, XFS, etc., support the use of Access Control Lists (ACLs). These extend the basic features of file ownership and permission, described in the previous section, and allow for a more fine-grained control of each (file) object. For example: A user wants to share a file with another user and that user should only be able to read the file, but not write or change it.

/etc

/etc/fstab

ACL

acl

ACL

noacl

For some of the filesystems, the usage of ACLs is enabled by default (e.g. Btrfs, Ext3, Ext4). For other filesystems or older systems it must be enabled using the acl mount option - either in the mount command directly or in /etc/fstab. In the same way the usage of ACLs can be disabled by using the noacl mount option. For Ext* filesystems one can also use the tune2fs -o [no]acl /dev/device command to enable/disable the usage of ACLs by default. The default values for each filesystem can usually be found in their homonym manual pages in section 5 ((5)) or in mount(8).

After enabling ACLs, permissions can be set using the setfacl(1) command, while getfacl(1) allows one to retrieve the ACLs for a given object or path. These commands are part of the acl package. With setfacl one can also configure newly created files or directories to inherit permissions from the parent directory. It is important to note that ACLs are processed in their order and that an earlier entry that fits the situation has precedence over later entries.

If a file has ACLs set, the output of the ls -l command will show a plus-sign after the traditional permissions. When using ACLs, the chmod command behaves slightly different, and umask might be ignored. The extensive documentation, e.g. acl(5) contains more information.

Administration Interfaces

interface

administration interface

administration, interfaces

Using a graphical interface for administration is interesting in various circumstances. An administrator does not necessarily know all the configuration details for all their services, and doesn't always have the time to go seeking out the documentation on the matter. A graphical interface for administration can thus accelerate the deployment of a new service. It can also simplify the setup of services which are hard to configure.

Such an interface is only an aid, and not an end in itself. In all cases, the administrator must master its behavior in order to understand and work around any potential problem.

Since no interface is perfect, you may be tempted to try several solutions. This is to be avoided as much as possible, since different tools are sometimes incompatible in their work methods. Even if they all aim to be very flexible and try to adopt the configuration file as a single reference, they are not always able to integrate external changes.

Browser-based Administration: cockpit

administration, interfaces

cockpit

Cockpit is a web-based graphical interface. It is secured by HTTPS by default and can be used with your own trusted certificates. It allows an administrator to access not only data and basic administration functions, but also a terminal to run commands directly on the target system. The interface is provided in form of multiple packages which provide module-like functionality. For example, cockpit-pcp provides functionality to log and access resource usage statistics. It is important to know that the cockpit-networkmanager should not be installed if network-manager is not installed either. Some of the modules provide network services, and it should be carefully checked if they should really be accessible by the public.

Administrating on a Web Interface: webmin

webmin

administration, interfaces

webmin

This is, without a doubt, one of the most successful administration interfaces. It is a modular system managed through a web browser, covering a wide array of areas and tools. Furthermore, it is internationalized and available in many languages.

Sadly, webmin is no longer part of Debian. Its Debian maintainer removed the packages created because they no longer had the time required to maintain them at an acceptable quality level. Nobody has officially taken over, so Debian does not provide the webmin package.

There is, however, an unofficial package distributed on the webmin.com website. Contrary to the original Debian packages, this package is monolithic; all of its configuration modules are installed and activated by default, even if the corresponding service is not installed on the machine. Users should be aware that webmin had its fair share of vulnerabilities. It should therefore be kept up-to-date, and additional measurements might be in order for public systems using it.

Webmin is used through a web interface, but it does not require Apache to be installed. Essentially, this software has its own integrated mini web server. This server listens by default on port 10000 and accepts secure HTTP connections.

Included modules cover a wide variety of services, among which:

• all base services: creation of users and groups, management of crontab files, service scripts/files, viewing of logs, etc.

• bind: DNS server configuration (name service);

• postfix: SMTP server configuration (e-mail);

• network services: configuration of the xinetd super-server;

• disk quota: user quota management;

• dhcpd: DHCP server configuration;

• proftpd: FTP server configuration;

• samba: Samba file server configuration;

• software: installation or removal of software from Debian packages and system updates.

The administration interface is available in a web browser at https://localhost:10000. Beware! Not all the modules are directly usable. Sometimes they must be configured by specifying the locations of the corresponding configuration files and some executable files (program). Frequently the system will politely prompt you when it fails to activate a requested module.

Configuring Packages: debconf

debconf

dpkg-reconfigure

dpkg-reconfigure

debconf

Many packages are automatically configured after asking a few questions during installation through the Debconf tool. These packages can be reconfigured by running dpkg-reconfigure -plevel package.

For most cases, these settings are very simple; only a few important variables in the configuration file are changed. These variables are often grouped between two “demarcation” lines so that reconfiguration of the package only impacts the enclosed area. In other cases, reconfiguration will not change anything if the script detects a manual modification of the configuration file, in order to preserve these human interventions (because the script can't ensure that its own modifications will not disrupt the existing settings).

syslog System Events

rsyslogd

syslogd

journalctl

file

log file

log files

dispatching

log files

/var/log

Principle and Mechanism

Starting with Debian Bookworm, the rsyslog package is no longer installed by default, which in the past provided the default logging functionality. Instead, the log is provided by systemd's journal, which can be accessed by the journalctl(5) command.

There are situations, where it is necessary or desired to install rsyslog, though, e.g. to send all log messages to a central server. After installation, the rsyslogd daemon is responsible for collecting service messages coming from applications and the kernel, then dispatching them into log files (usually stored in the /var/log/ directory). It obeys the /etc/rsyslog.conf configuration file.

/etc

/etc/rsyslog.conf

rsyslogd

Each log message is associated with an application subsystem (called “facility” in the documentation):

rsyslogd

facility

• auth and authpriv: for authentication;

• cron: comes from task scheduling services, cron and atd;

• daemon: affects a daemon without any special classification (DNS, NTP, etc.);

• ftp: concerns the FTP server;

• kern: message coming from the kernel;

• lpr: comes from the printing subsystem;

• mail: comes from the e-mail subsystem;

• news: Usenet subsystem message (especially from an NNTP — Network News Transfer Protocol — server that manages newsgroups);

  Network News Transfer Protocol NNTP

• syslog: messages from the syslogd server, itself;

  syslogd

• user: user messages (generic);

• uucp: messages from the UUCP server (Unix to Unix Copy Program, an old protocol notably used to distribute e-mail messages);

• local0 to local7: reserved for local use.

Each message is also associated with a priority level. Here is the list in decreasing order:

rsyslogd

priority

• emerg: “Help!” There is an emergency, the system is probably unusable.

• alert: hurry up, any delay can be dangerous, action must be taken immediately;

• crit: conditions are critical;

• err: error;

• warn: warning (potential error);

• notice: conditions are normal, but the message is important;

• info: informative message;

• debug: debugging message.

The Configuration File

rsyslogd

configuration

/etc/rsyslog.conf

/etc

/etc/rsyslog.conf

rsyslogd

The syntax of the /etc/rsyslog.conf file is detailed in the rsyslog.conf(5) manual page, but there is also HTML documentation available in the rsyslog-doc package (/usr/share/doc/rsyslog-doc/html/index.html). The overall principle is to write “selector” and “action” pairs. The selector defines all relevant messages, and the action describes how to deal with them.

Syntax of the Selector

rsyslogd

selector

The selector is a semicolon-separated list of subsystem.priority pairs (example: auth.notice;mail.info). An asterisk may represent all subsystems or all priorities (examples: *.alert or mail.*). Several subsystems can be grouped, by separating them with a comma (example: auth,mail.info). The priority indicated also covers messages of equal or higher priority; thus auth.alert indicates the auth subsystem messages of alert or emerg priority. Prefixed with an exclamation point (!), it indicates the opposite, in other words the strictly lower priorities; auth.!notice, thus, indicates messages issued from auth, with info or debug priority. Prefixed with an equal sign (=), it corresponds to precisely and only the priority indicated (auth.=notice only concerns messages from auth with notice priority).

Each element in the list on the selector overrides previous elements. It is thus possible to restrict a set or to exclude certain elements from it. For example, kern.info;kern.!err means messages from the kernel with priority between info and warn. The none priority indicates the empty set (no priorities), and may serve to exclude a subsystem from a set of messages. Thus, *.crit;kern.none indicates all the messages of priority equal to or higher than crit not coming from the kernel.

Syntax of Actions

rsyslogd

action

The various possible actions are:

• add the message to a file (example: /var/log/messages);

• send the message to a remote syslog server (example: @log.falcot.com);

• send the message to an existing named pipe (example: |/dev/xconsole);

• send the message to one or more users, if they are logged in (example: root,rhertzog);

• send the message to all logged in users (example: *);

• write the message in a text console (example: /dev/tty8).

The inetd Super-Server

Inetd (often called “Internet super-server”) is a server of servers. It executes rarely used servers on demand, so that they do not have to run continuously.

inetd

super-server

super-server

The /etc/inetd.conf file lists these servers and their usual ports. The inetd command listens to all of them; when it detects a connection to any such port, it executes the corresponding server program.

Each significant line of the /etc/inetd.conf file describes a server through seven fields (separated by spaces):

super-server

configuration

• The TCP or UDP port number, or the service name (which is mapped to a standard port number with the information contained in the /etc/services file).

  /etc /etc/services

• The socket type: stream for a TCP connection, dgram for UDP datagrams.

• The protocol: tcp, tcp6, udp, or udp6.

• The options: two possible values: wait or nowait, to tell inetd whether it should wait or not for the end of the launched process before accepting another connection. For TCP connections, easily multiplexable, you can usually use nowait. For programs responding over UDP, you should use nowait only if the server is capable of managing several connections in parallel. You can suffix this field with a period, followed by the maximum number of connections authorized per minute (the default limit is 256).

• The user name of the user under whose identity the server will run. Optionally one can add the group too via user.group syntax.

• The full path to the server program to execute.

• The arguments: this is a complete list of the program's arguments, including its own name (argv[0] in C).

The following example illustrates some use-cases after installing talkd, nullidentd (ident-server), and fingerd:

#:BSD: Shell, login, exec and talk are BSD protocols.
talk   dgram   udp     wait    nobody.tty   /usr/sbin/in.talkd      in.talkd
ntalk  dgram   udp     wait    nobody.tty   /usr/sbin/in.ntalkd     in.ntalkd

#:INFO: Info services
ident  stream  tcp     nowait  nobody       /usr/sbin/nullidentd    nullidentd
finger stream  tcp     nowait  nobody       /usr/sbin/tcpd          /usr/sbin/in.fingerd

tcpd

super-server

access limitation

The tcpd program is frequently used in the /etc/inetd.conf file. It allows limiting incoming connections by applying access control rules, documented in the hosts_access(5) manual page, and which are configured in the /etc/hosts.allow and /etc/hosts.deny files. Once it has been determined that the connection is authorized, tcpd executes the real server (like in.fingerd in our example). It is worth noting that tcpd relies on the name under which it was invoked (that is the first argument, argv[0]) to identify the real program to run. So you should not start the arguments list with tcpd but with the program that must be wrapped.

/etc

/etc/hosts.allow

/etc

/etc/hosts.deny

Scheduling Tasks with cron and atd

cron

atd

scheduled commands

command scheduling

cron is the daemon responsible for executing scheduled and recurring commands (every hour, every day, every week, etc.). atd deals with commands to be executed a single time, but at a specific moment in the future.

In a Unix system, many tasks are scheduled for regular execution:

• rotating the logs;

• updating the database for the locate program;

• back-ups;

• maintenance scripts (such as cleaning out temporary files).

crontab

By default, all users can schedule the execution of tasks. Each user has thus their own crontab in which they can record scheduled commands. It can be edited by running crontab -e (its content is stored in the /var/spool/cron/crontabs/user file).

/etc

/etc/crontab

/etc

/etc/cron.d/

The root user has their own crontab, but can also use the /etc/crontab file, or write additional crontab files in the /etc/cron.d directory. These last two solutions have the advantage of being able to specify the user identity to use when executing the command.

The cron package includes by default some scheduled commands that execute:

• /etc /etc/cron.hourly/ programs in the /etc/cron.hourly/ directory once per hour;
• /etc /etc/cron.daily/ programs in /etc/cron.daily/ once per day;
• /etc /etc/cron.weekly/ programs in /etc/cron.weekly/ once per week;
• /etc /etc/cron.monthly/ programs in /etc/cron.monthly/ once per month.

Many Debian packages rely on this service: by putting maintenance scripts in these directories, they ensure optimal operation of their services.

Format of a crontab File

crontab

crontab

file format

/etc

/etc/crontab

Each significant line of a crontab entry describes a scheduled command with the six (or seven) following fields:

• the value for the minute (from 0 to 59);

• the value for the hour (from 0 to 23);

• the value for the day of the month (from 1 to 31);

• the value for the month (from 1 to 12);

• the value for the day of the week (from 0 to 7, 1 corresponding to Monday, Sunday being represented by both 0 and 7; it is also possible to use the first three letters of the name of the day of the week in English, such as Sun, Mon, etc.);

• the user name under whose identity the command must be executed (in the /etc/crontab file and in the fragments located in /etc/cron.d/, but not in the users' own crontab files);

• the command to execute (when the conditions defined by the first five columns are met).

All these details are documented in the crontab(5) man page.

Each value can also be expressed in the form of a list of possible values (separated by commas). The syntax a-b describes the interval of all the values between a and b. The syntax a-b/c describes the interval with an increment of c (example: 0-10/2 means 0,2,4,6,8,10). An asterisk * is a wildcard, representing all possible values.

#Format
#min hour day mon dow  command

# Download data every night at 7:25 pm
 25  19   *   *   *    $HOME/bin/get.pl

# 8:00 am, on weekdays (Monday through Friday)
 00  08   *   *   1-5  $HOME/bin/dosomething

# every two hours
 *  */2   *   *   *    $HOME/bin/dosomethingelse

# Restart the IRC proxy after each reboot
@reboot /usr/bin/dircproxy

Using the at Command

at

The at executes a command at a specified moment in the future. It takes the desired time and date as command-line parameters, and the command to be executed in its standard input. The command will be executed as if it had been entered in the current shell. at even takes care to retain the current environment, in order to reproduce the same conditions when it executes the command. The time is indicated by following the usual conventions: 16:12 or 4:12pm represents 4:12 pm. The date can be specified in several European and Western formats, including DD.MM.YY (27.07.22 thus representing 27 July 2022), YYYY-MM-DD (this same date being expressed as 2022-07-27), MM/DD/[CC]YY (i.e., 12/25/22 or 12/25/2022 will be December 25, 2022), or simple MMDD[CC]YY (so that 122522 or 12252022 will, likewise, represent December 25, 2022). Without it, the command will be executed as soon as the clock reaches the time indicated (the same day, or tomorrow if that time has already passed on the same day). You can also simply write “today” or “tomorrow”, which is self-explanatory.

$ at 09:00 27.07.22 <<END
> echo "Don't forget to wish a Happy Birthday to Raphaël!" \
>   | mail lolando@debian.org
> END
warning: commands will be executed using /bin/sh
job 1 at Wed Jul 27 09:00:00 2022

An alternative syntax postpones the execution for a given duration: at now + number period. The <period> can be minutes, hours, days, or weeks. The <number> simply indicates the number of said units that must elapse before execution of the command.

To cancel a task scheduled by cron, simply run crontab -e and delete the corresponding line in the crontab file. For at tasks, it is almost as easy: run atrm task-number. The task number is indicated by the at command when you scheduled it, but you can find it again with the atq command, which gives the current list of scheduled tasks.

crontab

atrm

atq

Scheduling Asynchronous Tasks: anacron

anacron

cron

/etc

/etc/anacrontab

anacron is the daemon that completes cron for computers that are not on at all times. Since regular tasks are usually scheduled for the middle of the night, they will never be executed if the computer is off at that time. The purpose of anacron is to execute them, taking into account periods in which the computer is not working.

nice

Please note that anacron will frequently execute such activity a few minutes after booting the machine, which can render the computer less responsive. This is why the tasks in the /etc/anacrontab file are started with the nice command, which reduces their execution priority and thus limits their impact on the rest of the system. Beware, the format of this file is not the same as that of /etc/crontab; if you have particular needs for anacron, see the anacrontab(5) manual page.

/etc

/etc/cron.hourly/

/etc

/etc/cron.daily/

/etc

/etc/cron.weekly/

/etc

/etc/cron.monthly/

Installation of the anacron package deactivates execution by cron of the scripts in the /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/, and /etc/cron.monthly/ directories. This avoids their double execution by anacron and cron. The latter command remains active, though, and will continue to handle the other scheduled tasks (especially those scheduled by users).

Quotas

quota

The quota system allows limiting disk space allocated to a user or group of users. To set it up, you must have a kernel that supports it (compiled with the CONFIG_QUOTA option) — as is the case with Debian kernels. The quota management software is found in the quota Debian package.

/etc

/etc/fstab

usrquota

quota

grpquota

quota

To activate quota in a filesystem, you have to indicate the usrquota and grpquota options in /etc/fstab for the user and group quotas, respectively. Rebooting the computer will then update the quotas in the absence of disk activity (a necessary condition for proper accounting of already used disk space).

edquota

quota

edquota

The edquota user (or edquota -g group) command allows you to change the limits while examining current disk space usage.

The quota system allows you to set four limits:

• two limits (called “soft” and “hard”) refer to the number of blocks consumed. If the filesystem was created with a block-size of 1 kibibyte, the 1024 available bytes of each block can only be assigned to one file. Unsaturated blocks thus induce losses of disk space. A quota of 100 blocks, which theoretically allows storage of 102,400 bytes, will, however, be saturated with just 100 files of 500 bytes each, only representing 50,000 bytes in total.

• two limits (“soft” and “hard”) refer to the number of inodes used. Each file occupies at least one inode to store information about it (permissions, owner, timestamp of last access, etc.). It is thus a limit on the number of user files.

  inode quota soft limit quota hard limit

A “soft” limit can be temporarily exceeded; the user will simply be warned that they are exceeding the quota by the warnquota command, which is usually invoked by cron. A “hard” limit can never be exceeded: the system will refuse any operation that will cause a hard quota to be exceeded.

quota

warnquota

quota

edquota

With the edquota -t command, you can define a maximum authorized “grace period” within which a soft limit may be exceeded. After this period, the soft limit will be treated like a hard limit, and the user will have to reduce their disk space usage to within this limit in order to be able to write anything to the hard drive.

Backup

backup

restoration

Making backups is one of the main responsibilities of any administrator, but it is a complex subject, involving powerful tools which are often difficult to master. The importance of backups cannot be stressed enough, though. Backups are also not to be confused with the availability of data. Both can complement each other, but they are essentially different.

amanda

bacula

dd

duplicity

fsarchiver

BackupPC

rdiff-backup

rsnapshot

timeshift

Many programs exist, such as amanda, bacula, or BackupPC. Those are client/server systems featuring many options, whose configuration is rather difficult. Some of them provide user-friendly web interfaces to mitigate this. For non-enterprise systems, administrators might want to check out rsnapshot or rdiff-backup. Users can easily create backups of their filesystems with timeshift, fsarchiver, duplicity, or even dd.

Debian contains dozens of other backup software covering all possible use cases, as you can easily confirm with apt-cache search backup.

Rather than detailing some of them, this section will present the thoughts of the Falcot Corp administrators when they defined their backup strategy.

At Falcot Corp, backups have two goals: recovering erroneously deleted files, and quickly restoring any computer (server or desktop) whose hard drive has failed.

Backing Up with rsync

rsync

Backups on tape having been deemed too slow and costly, data will be backed up on hard drives on a dedicated server, on which the use of software RAID (see ???) will protect the data from hard drive failure. Desktop computers are not backed up individually, but users are advised that their personal account on their department's file server will be backed up. The rsync command (from the package of the same name) is used daily to back up these different servers.

The available hard drive space prohibits implementation of a complete daily backup. As such, the rsync command is preceded by a duplication of the content of the previous backup with hard links, which prevents usage of too much hard drive space. The rsync process then only replaces files that have been modified since the last backup. With this mechanism a great number of backups can be kept in a small amount of space. Since all backups are immediately available and accessible (for example, in different directories of a given share on the network), you can quickly make comparisons between two given dates.

copy, backup copy

backup

copy

This backup mechanism is easily implemented with the dirvish program. It uses a backup storage space (“bank” in its vocabulary) in which it places timestamped copies of sets of backup files (these sets are called “vaults” in the dirvish documentation).

/etc

/etc/dirvish/master.conf

The main configuration is in the /etc/dirvish/master.conf file. It defines the location of the backup storage space, the list of “vaults” to manage, and default values for expiration of the backups. The rest of the configuration is located in the bank/vault/dirvish/default.conf files and contains the specific configuration for the corresponding set of files.

bank:
    /backup
exclude:
    lost+found/
    core
    *~
Runall:
    root    22:00
expire-default: +15 days
expire-rule:
#   MIN HR    DOM MON       DOW  STRFTIME_FMT
    *   *     *   *         1    +3 months
    *   *     1-7 *         1    +1 year
    *   *     1-7 1,4,7,10  1

The bank setting indicates the directory in which the backups are stored. The exclude setting allows you to indicate files (or file types) to exclude from the backup. The Runall is a list of file sets to backup with a time-stamp for each set, which allows you to assign the correct date to the copy, in case the backup is not triggered at precisely the assigned time. You have to indicate a time just before the actual execution time (according to /etc/cron.d/dirvish). Finally, the expire-default and expire-rule settings define the expiration policy for backups. The above example keeps forever backups that are generated on the first Sunday of each quarter, deletes after one year those from the first Sunday of each month, and after 3 months those from other Sundays. Other daily backups are kept for 15 days. The order of the rules does matter, Dirvish uses the last matching rule, or the expire-default one if no other expire-rule matches.

client: rivendell.falcot.com
tree: /
xdev: 1
index: gzip
image-default: %Y%m%d
exclude:
    /var/cache/apt/archives/*.deb
    /var/cache/man/**
    /tmp/**
    /var/tmp/**
    *.bak

The above example specifies the set of files to back up: these are files on the machine rivendell.falcot.com (for local data backup, simply specify the name of the local machine as indicated by hostname), especially those in the root tree (tree: /), except those listed in exclude. The backup will be limited to the contents of one filesystem (xdev: 1). It will not include files from other mount points. An index of saved files will be generated (index: gzip), and the image will be named according to the current date (image-default: %Y%m%d).

There are many options available, all documented in the dirvish.conf(5) manual page. Once these configuration files are setup, you have to initialize each file set with the dirvish --vault vault --init command. From there on the daily invocation of dirvish-runall will automatically create a new backup copy just after having deleted those that expired.

Restoring Machines without Backups

Desktop computers, which are not backed up, will be easy to reinstall from custom DVD-ROMs/USB sticks prepared with simple-cdd (see ???). Since this performs an installation from scratch, it loses any customization that can have been made after the initial installation. This is fine since the systems are all hooked to a central LDAP directory for accounts and most desktop applications are preconfigured thanks to dconf (see ??? for more information about this).

The Falcot Corp administrators are aware of the limits in their backup policy. Since they can't protect the backup server as well as a tape in a fireproof safe, they have installed it in a separate room so that a disaster such as a fire in the server room won't destroy backups along with everything else. Furthermore, they do an incremental backup on DVD-ROM once per week — only files that have been modified since the last backup are included.

Hot Plugging: hotplug

Introduction

hotplug

kernel

hotplug subsystem

udevd

/dev

device

/dev

USB

IEEE 1394

PCMCIA

SATA

The hotplug kernel subsystem dynamically handles the addition and removal of devices, by loading the appropriate drivers and by creating the corresponding device files (with the help of udevd). With modern hardware and virtualization, almost everything can be “hotplugged“: from the usual USB/PCMCIA/IEEE 1394 peripherals to SATA hard drives, but also the CPU and the memory.

The kernel has a database that associates each device ID with the required driver. This database is used during boot to load all the drivers for the peripheral devices detected on the different buses, but also when an additional hotplug device is connected. Once the device is ready for use, a message is sent to udevd so it will be able to create the corresponding entry in /dev/.

The Naming Problem

Before the appearance of hotplug connections, it was easy to assign a fixed name to a device. It was based simply on the position of the devices on their respective bus. But this is not possible when such devices can come and go on the bus. The typical case is the use of a digital camera and a USB key, both of which appear to the computer as disk drives. The first one connected may be /dev/sdb and the second /dev/sdc (with /dev/sda representing the computer's own hard drive). The device name is not fixed; it depends on the order in which devices are connected.

Additionally, more and more drivers use dynamic values for devices' major/minor numbers, which makes it impossible to have static entries for the given devices, since these essential characteristics may vary after a reboot.

udev

udev was created precisely to solve this problem.

How udev Works

udev

/sys

/etc

/etc/udev/rules.d/

/lib

/lib/udev/rules.d/

When udev is notified by the kernel of the appearance of a new device, it collects various information on the given device by consulting the corresponding entries in /sys/, especially those that uniquely identify it (MAC address for a network card, serial number for some USB devices, etc.).

udev

rules

Armed with all of this information, udev then consults all of the rules contained in /etc/udev/rules.d/ and /lib/udev/rules.d/. In this process it decides how to name the device, what symbolic links to create (to give it alternative names), and what commands to execute. All of these files are consulted, and the rules are all evaluated sequentially (except when a file uses “GOTO” directives). Thus, there may be several rules that correspond to a given event.

The syntax of rules files is quite simple: each row contains selection criteria and variable assignments. The former are used to select events for which there is a need to react, and the latter defines the action to take. They are all simply separated with commas, and the operator implicitly differentiates between a selection criterion (with comparison operators, such as == or !=) or an assignment directive (with operators such as =, += or :=).

Comparison operators are used on the following variables:

• KERNEL: the name that the kernel assigns to the device;

• ACTION: the action corresponding to the event (“add” when a device has been added, “remove” when it has been removed);

• DEVPATH: the path of the device's /sys/ entry;

• SUBSYSTEM: the kernel subsystem which generated the request (there are many, but a few examples are “usb”, “ide”, “net”, “firmware”, etc.);

• ATTR{attribute}: file contents of the <attribute> file in the /sys/$devpath/ directory of the device. This is where you find the MAC address and other bus specific identifiers;

• KERNELS, SUBSYSTEMS and ATTRS{attributes} are variations that will try to match the different options on one of the parent devices of the current device;

• PROGRAM: delegates the test to the indicated program (true if it returns 0, false if not). The content of the program's standard output is stored so that it can be reused by the RESULT test;

• RESULT: execute tests on the standard output stored during the last call to PROGRAM.

The right operands can use pattern expressions to match several values at the same time. For instance, * matches any string (even an empty one); ? matches any character, and [] matches the set of characters listed between the square brackets (or the opposite thereof if the first character is an exclamation point, and contiguous ranges of characters are indicated like a-z).

Regarding the assignment operators, = assigns a value (and replaces the current value); in the case of a list, it is emptied and contains only the value assigned. := does the same, but prevents later changes to the same variable. As for +=, it adds an item to a list. The following variables can be changed:

• NAME: the device filename to be created in /dev/. Only the first assignment counts; the others are ignored;

• SYMLINK: the list of symbolic links that will point to the same device;

• OWNER, GROUP and MODE define the user and group that owns the device, as well as the associated permission;

• RUN: the list of programs to execute in response to this event.

The values assigned to these variables may use a number of substitutions:

• $kernel or %k: equivalent to KERNEL;

• $number or %n: the order number of the device, for example, for sda3, it would be “3”;

• $devpath or %p: equivalent to DEVPATH;

• $attr{attribute} or %s{attribute}: equivalent to ATTRS{attribute};

• $major or %M: the kernel major number of the device;

• $minor or %m: the kernel minor number of the device;

• $result or %c: the string output by the last program invoked by PROGRAM;

• and, finally, %% and $$ for the percent and dollar sign, respectively.

The above lists are not complete (they include only the most important parameters), but the udev(7) manual page should be exhaustive.

A concrete example

Let us consider the case of a simple USB key and try to assign it a fixed name. First, you must find the elements that will identify it in a unique manner. For this, plug it in and run udevadm info -a -n /dev/sdc (replacing </dev/sdc> with the actual name assigned to the key).

# udevadm info -a -n /dev/sdc
[...]
  looking at device '/devices/pci0000:00/0000:00:10.0/usb2/2-1/2-1:1.0/host4/target4:0:0/4:0:0:0/block/sdc':
    KERNEL=="sdc"
    SUBSYSTEM=="block"
    DRIVER==""
    ATTR{hidden}=="0"
    ATTR{events}=="media_change"
    ATTR{ro}=="0"
    ATTR{discard_alignment}=="0"
    ATTR{removable}=="1"
    ATTR{events_async}==""
    ATTR{alignment_offset}=="0"
    ATTR{capability}=="51"
    ATTR{events_poll_msecs}=="-1"
    ATTR{stat}=="130  0  6328  435  0  0  0  0  0  252  252  0  0  0  0"
    ATTR{size}=="15100224"
    ATTR{range}=="16"
    ATTR{ext_range}=="256"
    ATTR{inflight}=="0  0"
[...]

  looking at parent device '/devices/pci0000:00/0000:00:10.0/usb2/2-1/2-1:1.0/host4/target4:0:0/4:0:0:0':
[...]
    ATTRS{max_sectors}=="240"
[...]
  looking at parent device '/devices/pci0000:00/0000:00:10.0/usb2/2-1':
    KERNELS=="2-1"
    SUBSYSTEMS=="usb"
    DRIVERS=="usb"
    ATTRS{bDeviceProtocol}=="00"
    ATTRS{bNumInterfaces}==" 1"
    ATTRS{busnum}=="2"
    ATTRS{quirks}=="0x0"
    ATTRS{authorized}=="1"
    ATTRS{ltm_capable}=="no"
    ATTRS{speed}=="480"
    ATTRS{product}=="TF10"
    ATTRS{manufacturer}=="TDK LoR"
[...]
    ATTRS{serial}=="07032998B60AB777"
[...]

To create a new rule, you can use tests on the device's variables, as well as those of one of the parent devices. The above case allows us to create two rules like these:

KERNEL=="sd?", SUBSYSTEM=="block", ATTRS{serial}=="07032998B60AB777", SYMLINK+="usb_key/disk"
KERNEL=="sd?[0-9]", SUBSYSTEM=="block", ATTRS{serial}=="07032998B60AB777", SYMLINK+="usb_key/part%n"

Once these rules are set in a file, named for example /etc/udev/rules.d/010_local.rules, you can simply remove and reconnect the USB key. You can then see that /dev/usb_key/disk represents the disk associated with the USB key, and /dev/usb_key/part1 is its first partition.

Power Management: Advanced Configuration and Power Interface (ACPI)

power management

ACPI

management, power management

ACPI

ACPI

Advanced Configuration and Power Interface

ACPI

acpid

The topic of power management is often problematic. Indeed, properly suspending the computer requires that all the computer's device drivers know how to put them to standby, and that they properly reconfigure the devices upon waking. Unfortunately, there are still a few devices unable to sleep well under Linux, because their manufacturers have not provided the required specifications.

Linux supports ACPI (Advanced Configuration and Power Interface) — the most recent standard in power management. The acpid package provides a daemon that looks for power management related events (switching between AC and battery power on a laptop, etc.) and that can execute various commands in response.

After this overview of basic services common to many Unix systems, we will focus on the environment of the administered machines: the network. Many services are required for the network to work properly. They will be discussed in the next chapter.