CharmingGroot

Production Server Setup Notes, Ep. 4 - What Is SELinux?

infra5 min한국어

Also available in Korean: 운영계 세팅 회고 ep4 - SELinux란?

Overview

The production server was set to run SELinux in Enforcing mode.
The development server runs in Permissive mode, so containers that worked fine there end up blocked with Permission denied in production.
With SELinux in force, access can be blocked even after opening file permissions with chmod.
That is because access is checked once more, separately from file permissions.

DAC and MAC

Linux’s basic permission model is DAC (Discretionary Access Control).
The file owner decides with rwx who may do what.

SELinux adds MAC (Mandatory Access Control) on top.
Here the system policy, not the file owner, decides what is allowed, and root cannot bypass the check.
With SELinux on, every access request must pass DAC and then the SELinux policy to be allowed.

Process file access request DAC owner · rwx SELinux (MAC) policy · labels Allow pass pass Permission denied root is no exception
Passing DAC is not enough; if the SELinux policy blocks the request, the result is Permission denied.

Labels: what SELinux looks at

SELinux attaches a label (context) to every process and file.
A label has four fields, user:role:type:level, and you can see them with ls -Z (files) and ps -Z (processes).
Decisions are mostly made on the third field, the type.

The policy is an allow list of the form “processes of this type may do these actions on files of that type”.
This is called type enforcement.
Any action without an allow rule in the policy is denied.

The four fields of a label (context) system_u user object_r role httpd_sys_content_t type s0 level ← decisions use the type type enforcement: an allow list between types Web server process httpd_t Web content file httpd_sys_content_t Home directory file user_home_t allow rule for read no rule → denied
A web server (httpd_t) can read web content (httpd_sys_content_t), but cannot read home directory files (user_home_t), which have no allow rule, regardless of rwx.

The three modes

modebehavior
Enforcingallow/deny by policy && log denials
Permissiveallow even policy violations && log violations
Disabledno checks

getenforce shows which mode a server is running in.
In Permissive mode policy violations are only logged, not actually blocked.
So if the development server is Permissive, actions that production (Enforcing) would block never show up during development.

A mode changed with setenforce reverts on reboot. The persistent setting lives in /etc/selinux/config.

Identifying denials

Actions SELinux blocks are recorded in the audit log as denial records of type AVC.
ausearch -m AVC -ts today shows today’s denials.

The AVC (Access Vector Cache) is where SELinux caches the answer to "may this type perform this action on that type". Denial records carry its name.

avc:  denied  { read } for  pid=4121 comm="otelcol" name="pods"
  scontext=system_u:system_r:container_t:s0:c12,c34
  tcontext=system_u:object_r:container_log_t:s0  tclass=dir
  • scontext: who (the label of the requesting process)
  • tcontext: what (the label of the target)
  • { read }, tclass=dir: which action, on what kind of target

This record shows a container (container_t) being denied a read on the pod log directory (container_log_t).

The policy also has deny rules that are deliberately not logged (dontaudit). If something is blocked but no AVC record appears, suspect this.

Fixing denials

Red Hat’s official documentation says to fix SELinux denials by checking for a labeling problem first, then booleans, and only then a policy module.
Each step allows more than the one before.

1. Fix labels semanage fcontext restorecon 2. Turn on a boolean setsebool -P prebuilt switches 3. Add a policy module audit2allow semodule -i Allows more, and a mistake costs more Turning it off (setenforce 0) or switching to Permissive is not a fix
Labels → boolean → policy module; each step allows more.

1. Fix labels

Most denials come from files carrying the wrong label.
Register a labeling rule for the path with semanage fcontext and apply it with restorecon.

semanage fcontext -a -t container_file_t "/srv/data(/.*)?"
restorecon -Rv /srv/data

chcon can change a label too, but it is not registered as a rule, so a later restorecon or full relabel puts the old label back.

2. Turn on a boolean

The policy includes prebuilt switches (booleans) for commonly needed permissions.
For example, if a web server has to make outbound connections, turn it on with setsebool -P httpd_can_network_connect on.
getsebool -a lists them.

3. Add a policy module

If labels and booleans are not enough, build a policy module from the denial records.
audit2allow reads the AVC records and generates allow rules, and semodule -i installs the module.
It allows everything that was logged, so read the generated rules and keep only what is needed.

Turning SELinux off or switching to Permissive is not a fix.
Even if you use it briefly while investigating, once you find the cause, fix it with one of the three steps above and go back to Enforcing.

Containers and SELinux

Processes running inside a container get the type container_t.
The policy lets container_t processes read and write files of type container_file_t.
So when a directory on the host is mounted into a container as a volume and its type is not container_file_t, access is denied even if rwx is wide open.
For example, a directory on a newly attached disk has the type unlabeled_t (no label), so containers can neither read nor write it.

Data disk directory Container container_t unlabeled_t (no label) container_file_t denied allowed after semanage + restorecon Log collector reading pod logs on the node container_t → reading log directory denied container_logreader_t → log reading added, nothing else spc_t lifts almost everything · last resort
Relabel unlabeled directories as container_file_t, and run the log collector as container_logreader_t, which adds read access only.

Setting up the production server ran into three of these.

  • Directories on the data disk had no label. Every directory on the separately attached disk was unlabeled_t, so containers could not write to it. Using method 1, container_file_t rules were registered for the DB and backup directories.
  • The log collector could not read pod logs. This is the AVC example from Identifying denials. Instead of spc_t, which has no restrictions at all, the collector runs as container_logreader_t, which only adds reading container logs.
  • A postStart hook could not write to the container log. The postStart hook, which runs when the pod starts, was blocked writing to /proc/1/fd/1, so its output was moved to a file inside the pod.

With Docker and Podman, the volume options :z (shared by several containers) and :Z (private to one container) relabel the mounted directory for you.

k3s ships a separate SELinux policy package (k3s-selinux) and turns SELinux on with selinux: true in its config.
The official k3s documentation states that a custom --data-dir is not supported under SELinux.
So instead of moving the data directory, a directory on the data disk was bind-mounted onto k3s’s default data path (/var/lib/rancher).
Then semanage fcontext -e registered the data-disk directory to receive the same labels as the default path.
That way, running restorecon on the data-disk side does not break the k3s labels.

References