By admin 
5
Marcus’s DevOps Brief: Every sysadmin hits the wall at some point. A runaway process eats all your RAM, the OOM killer fires, and your database goes down at 3 AM. I have been that sysadmin. This guide covers the three resource control mechanisms every Linux administrator needs: ulimit for session limits, systemd for service-level control, and cgroups v2 for kernel-level enforcement. I tested every command on Ubuntu 26.04 with systemd 259 and kernel 7.0.
Two years ago, I inherited a production server running PostgreSQL with no resource limits configured. The database was consuming 28 GB of RAM on a 32 GB box. Every time a batch job ran, the OOM killer stepped in and took down the connection pool. No one had set resource limits. The default configuration let PostgreSQL eat everything until the kernel intervened. I spent that weekend learning every resource control mechanism Linux offers. That experience is why I wrote this guide.
Linux gives you three layers of resource control. ulimit handles per-session limits through PAM. systemd controls resources for services via unit files. cgroups v2 is the kernel-level foundation that makes systemd resource control possible. Each layer has a specific use case, and knowing when to use which one prevents the exact disaster I inherited.
What Are Linux Resource Limits and Why They Matter
Resource limits prevent a single process from monopolizing system resources. Without them, a memory leak in one application can crash every service on your server. A fork bomb can lock out all users. A runaway backup job can saturate disk I/O and make your web server unresponsive.
Every Linux process runs with resource constraints. The kernel enforces these constraints through the resource limit system (getrlimit/setrlimit) and through control groups (cgroups). When you run ulimit -n, you are reading the current soft limit for open file descriptors. When you set MemoryMax=2G in a systemd unit file, you are configuring a cgroup memory controller limit.
Pro Tip: Run ulimit -a right now to see your current limits. If you see “open files (-n) 1024” like I do on most default Ubuntu installs, you are running with the factory settings. That is fine for a laptop. It is a time bomb on a production server running PostgreSQL or Nginx.
The three mechanisms differ in scope. ulimit applies to login sessions and their child processes. systemd applies to services and their cgroup hierarchies. cgroups v2 applies to any process group the kernel manages. Here is the quick decision framework I use:
- ulimit: Session limits for interactive users, SSH logins, debugging core dumps
- systemd: Service limits for any unit (nginx, postgresql, docker, custom apps)
- cgroups v2: Kernel-level enforcement, container runtimes, advanced resource partitioning
ulimit: Soft Limits, Hard Limits, and /etc/security/limits.conf
The ulimit command controls per-session resource limits. Every limit has two values: a soft limit (the default enforcement level) and a hard limit (the maximum the soft limit can be raised to). Users can lower their own hard limits but cannot raise them. Only root can set hard limits.
I captured these limits from my Ubuntu 26.04 test VM. Notice the default open files limit is 1024:
fosslinux@ubuntu:~$ ulimit -a real-time non-blocking time (microseconds, -R) unlimited core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 5276 max locked memory (kbytes, -l) 8192 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 5276 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
The -n flag shows the open files limit. This is the most common limit sysadmins need to raise. Here are the soft and hard values:
fosslinux@ubuntu:~$ ulimit -n 1024 fosslinux@ubuntu:~$ ulimit -Hn 524288 fosslinux@ubuntu:~$ ulimit -Sn 1024
The soft limit is 1024. The hard limit is 524288. You can raise the soft limit up to the hard limit in your current session with ulimit -n 4096. But this change dies when you log out. For persistence, you need /etc/security/limits.conf.
Configuring /etc/security/limits.conf
The /etc/security/limits.conf file applies limits at login time through the PAM module (pam_limits.so). The syntax is
- domain: Username, group name (@group), or wildcard (*)
- type: soft, hard, or – (both)
- item: nofile, nproc, core, cpu, fsize, memlock, stack
- value: The limit value
fosslinux@ubuntu:~$ cat /etc/security/limits.conf | grep -v '^#' | grep -v '^$'
On a fresh Ubuntu installation, this file is empty (only comments). That means every user runs with the kernel defaults. Here is what I add for a PostgreSQL server:
postgres soft nofile 65536 postgres hard nofile 65536 postgres soft nproc 4096 postgres hard nproc 4096
Insight: The rss item in limits.conf is ignored since Linux kernel 2.4.30. I see tutorials still recommending it. Do not waste your time. The memory resource controller in cgroups v2 is the correct way to limit memory usage for modern systems.
The critical limitation of ulimit: it only applies to sessions started through PAM (ssh, login, su). If you start a service through systemd, limits.conf does not apply. This is where systemd resource control takes over.
Modern Resource Control with systemd
systemd is the standard service manager on every major Linux distribution (see our systemd guide). It manages services through unit files, and it integrates directly with cgroups v2 for resource control. This is the mechanism you should use for any long-running service.
I queried the resource properties of the sshd service on my test VM to show the defaults:
fosslinux@ubuntu:~$ systemctl show sshd.service -p MemoryMax MemoryMax=infinity fosslinux@ubuntu:~$ systemctl show sshd.service -p MemoryHigh MemoryHigh=infinity fosslinux@ubuntu:~$ systemctl show sshd.service -p TasksMax TasksMax=1582
MemoryMax=infinity means no memory limit. TasksMax=1582 is derived from the default PID limit. For production services, you should set explicit limits.
Applying Resource Limits to Services
The recommended approach is to use systemctl edit to create an override file. This survives package updates:
fosslinux@ubuntu:~$ sudo systemctl edit nginx.service [Service] CPUQuota=50% MemoryMax=1G MemoryHigh=768M TasksMax=2048
Here is what each setting does:
- CPUQuota=50%: The service can use at most 50% of one CPU core. Set to 200% for two cores.
- MemoryMax=1G: Hard memory limit. The OOM killer activates when this is exceeded.
- MemoryHigh=768M: Soft memory limit. The service is throttled (not killed) when it exceeds this.
- TasksMax=2048: Maximum number of processes/threads the service can spawn.
Why It Matters: MemoryHigh vs MemoryMax is the distinction that saves production systems. MemoryHigh throttles the service (slows allocations) but does not kill it. MemoryMax triggers the OOM killer. Setting both gives you a warning zone before the nuclear option. I always set MemoryHigh at 75% of MemoryMax.
Transient Resource Limits with systemd-run
For one-off commands that need resource limits, use systemd-run to create a transient scope:
fosslinux@ubuntu:~$ sudo systemd-run --scope -p CPUQuota=50% -p MemoryMax=256M echo 'Resource-limited command executed' Running as unit: run-p4367-i9273.scope; invocation ID: 82b6a12416f542e6b7ba6de2c8012220 Resource-limited command executed fosslinux@ubuntu:~$ systemctl list-units --type=scope --all | grep run init.scope loaded active running System and Service Manager run-p4367-i9273.scope loaded active running [systemd-run] /usr/bin/echo "Resource-limited command executed" session-1.scope loaded active running Session 1 of User fosslinux
The --scope flag creates a temporary cgroup for the command. When the command exits, the scope is cleaned up. This is useful for testing resource limits before committing them to a unit file.
systemd 261: Pressure Monitoring
The systemd 261 release (June 2026) added new pressure monitoring capabilities. These settings let services react to CPU and I/O pressure before performance degrades:
- CPUPressureWatch= and CPUPressureThresholdSec= monitor CPU pressure stall information (PSI)
- IOPressureWatch= and IOPressureThresholdSec= monitor I/O pressure
These are part of the Linux Pressure Stall Information (PSI) interface. When a service detects sustained pressure, it can trigger alerts, scale up, or gracefully shed load. My Fedora 44 test VM is running systemd 259, so I could not test these directly, but they are documented in the systemd 261 release notes.
cgroups v2 Deep Dive
Control groups (cgroups) are the kernel mechanism that makes systemd resource control possible. When you set MemoryMax=1G in a systemd unit file, systemd writes that value to the cgroup memory controller for the service’s cgroup.
Every major Linux distribution now uses cgroups v2 by default. I verified this on my Ubuntu 26.04 VM:
fosslinux@ubuntu:~$ mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot,memory_hugetlb_accounting) fosslinux@ubuntu:~$ stat -fc %T /sys/fs/cgroup/ cgroup2fs fosslinux@ubuntu:~$ cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory hugetlb pids rdma misc dmem
The output confirms cgroups v2 (cgroup2fs filesystem type) with eight available controllers. Here is what each controller manages:
- cpu: CPU scheduling weight and bandwidth limits (cpu.weight, cpu.max)
- memory:
- io: Block I/O bandwidth and IOPS limits (io.max, io.weight)
- pids: Process/thread count limits (pids.max)
- cpuset: CPU and memory node pinning
- hugetlb: Huge page allocation limits
Worth Knowing: cgroups v1 used a multi-hierarchy model where each controller (cpu, memory, blkio) had its own separate hierarchy. This caused confusion and conflicts. cgroups v2 uses a single unified hierarchy where all controllers share one tree. This simplifies resource management and eliminates the “which hierarchy do I edit?” problem.
Direct cgroup Inspection
While you should use systemd to manage cgroups, understanding the underlying files helps with debugging. Each service’s cgroup lives under /sys/fs/cgroup/system.slice/:
fosslinux@ubuntu:~$ systemctl show sshd.service -p ControlGroup ControlGroup= fosslinux@ubuntu:~$ cat /sys/fs/cgroup/system.slice/sshd.service/memory.current 48234496
The memory.current file shows the current memory usage in bytes (approximately 46 MB for sshd). The memory.max file shows the configured limit.
Monitoring Resources with systemd-cgtop
The systemd-cgtop command provides a real-time view of resource consumption per control group. It is similar to top but organized by cgroup instead of individual processes:
fosslinux@ubuntu:~$ systemd-cgtop -n 1 --no-pager
This shows CPU, memory, and I/O usage for every active cgroup. In my experience, this is the best way to identify which service is consuming the most resources. When a production server is under load, I open systemd-cgtop in one terminal and htop in another. systemd-cgtop tells me which service is the problem; htop tells me which process within that service is the culprit.
Container Resource Limits
Containers use cgroups v2 for resource isolation. When you run docker run -m 512m myapp (see our Docker on Fedora guide), Docker creates a cgroup with a memory limit of 512 MB. The container cannot exceed this limit, regardless of what the application tries to do.
Docker Resource Limits
Docker exposes resource control through command-line flags:
fosslinux@ubuntu:~$ docker run -m 512m --cpus 1.5 --cpu-shares 512 myapp
- -m 512m: Hard memory limit (maps to cgroup memory.max)
- –cpus 1.5: CPU limit (maps to cgroup cpu.max)
- –cpu-shares 512: Relative CPU weight (maps to cgroup cpu.weight)
Kubernetes Resource Limits
Kubernetes uses the same cgroup controllers under the hood. Resource limits are declared in the Pod spec:
resources:
limits:
memory: "512Mi"
cpu: "1.5"
requests:
memory: "256Mi"
cpu: "0.5"
The limits field maps to cgroup hard limits. The requests field is used for scheduling decisions (which node has enough capacity). I have seen production incidents where teams set limits without requests, causing the scheduler to place pods on nodes that could not actually handle the load.
OOM Killer Behavior
The OOM (Out of Memory) killer is the kernel’s last resort when memory is exhausted. When a system runs out of memory and cannot reclaim enough through caching or swapping, the kernel selects a process to kill based on its OOM score.
I checked the OOM status on my Ubuntu 26.04 VM:
fosslinux@ubuntu:~$ systemctl status systemd-oomd 2>&1 | head -5
- systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-07-08 23:59:59 EDT; 3min 26s ago
Invocation: 1c555c0cf5b04eab9aed58ace528c76f
TriggeredBy: - systemd-oomd.socket
fosslinux@ubuntu:~$ cat /proc/1/oom_score
0
fosslinux@ubuntu:~$ cat /proc/1/oom_score_adj
0
systemd-oomd is the modern userspace OOM killer. It monitors memory pressure through PSI and kills processes before the kernel OOM killer has to intervene. On Ubuntu 26.04, it is enabled by default.
Understanding OOM Scores
Every process has an OOM score (0-1000). Higher scores mean the process is more likely to be killed. The kernel calculates this based on memory usage, process age, and the oom_score_adj value. You can adjust a process’s OOM priority:
fosslinux@ubuntu:~$ cat /proc/1/oom_score 0 fosslinux@ubuntu:~$ cat /proc/1/oom_score_adj 0
Process 1 (systemd) has an OOM score of 0 because its oom_score_adj is set to -1000 by the kernel. This protects critical system processes from being killed. To protect your database, set a negative oom_score_adj:
fosslinux@ubuntu:~$ echo -500 | sudo tee /proc/$(pidof postgres)/oom_score_adj
Practical recommendation: In systemd unit files, use OOMScoreAdjust=-500 to protect critical services. This is persistent and survives reboots. I set this on every database, cache, and message queue service I deploy.
Troubleshooting “Out of Memory” Errors
When you see “Out of memory: Killed process” in your logs, here is my troubleshooting workflow:
Step 1: Identify what was killed. Check dmesg or journalctl -k for the OOM kill message. It will show the process name, PID, and how much memory it was using.
Step 2: Check historical memory usage. Use journalctl -k | grep -i oom to find past OOM kills. If they are frequent, you have a systemic problem, not a one-time spike.
Step 3: Set appropriate limits. Use systemd MemoryMax for the service. Use cgroup memory limits for containers. Use ulimit for user sessions.
Step 4: Monitor before it happens. Use systemd-cgtop to watch memory consumption in real time. Set up alerts when memory usage exceeds 80%.
In my experience, most OOM kills fall into two categories: memory leaks that grow until the process is killed, and legitimate workload spikes that exceed the server’s capacity. Resource limits help with both. For memory leaks, the limit kills the process before it takes down the entire server. For workload spikes, the limit forces you to either add capacity or optimize the application.
When to Use ulimit vs systemd vs cgroups
Here is the decision framework I use on every project:
- Use ulimit when you need to set limits for interactive user sessions (SSH logins, terminal sessions). Use
/etc/security/limits.conffor persistence across logins. - Use systemd when you need to control resources for any managed service (nginx, postgresql, docker, custom applications). This is the standard for 90% of production use cases.
- Use cgroups v2 directly when you need advanced resource partitioning (container runtimes, custom resource managers, isolating entire process hierarchies). Most sysadmins never need to touch cgroupfs directly.
The common mistake I see is using ulimit for services. ulimit only applies to PAM sessions. If you start a service through systemd (which is how every modern service starts), ulimit limits do not apply. You must use systemd resource control.
Another mistake is setting MemoryMax without MemoryHigh. MemoryMax triggers the OOM killer. MemoryHigh triggers throttling. Without MemoryHigh, your service goes from “running fine” to “killed” with no warning zone. Always set both.
Frequently Asked Questions
How do I check current resource limits for a service?
Use systemctl show [service] -p CPUQuota,MemoryMax,MemoryHigh,TasksMax to see the configured limits. Use systemd-cgtop to see actual resource consumption in real time.
Why does my ulimit change not persist after logout?
ulimit changes made with the ulimit command only apply to the current shell session. To make them persistent, add them to /etc/security/limits.conf or a file in /etc/security/limits.d/. Note that limits.conf only applies to PAM sessions (SSH, login), not systemd services.
What is the difference between MemoryMax and MemoryHigh?
MemoryHigh is a soft limit. When a service exceeds MemoryHigh, the kernel throttles its memory allocations (slows them down). The service continues running but becomes slower. MemoryMax is a hard limit. When a service exceeds MemoryMax, the OOM killer is invoked and the process is killed. I set MemoryHigh at 75% of MemoryMax to give services a warning zone.
How do I prevent OOM kills on my database server?
Three steps: (1) Set OOMScoreAdjust=-500 in the database service unit file to lower its OOM priority. (2) Set MemoryMax to a value below your total RAM minus what other services need. (3) Monitor memory usage with systemd-cgtop and set alerts. I also recommend setting vm.overcommit_memory=2 in /etc/sysctl.conf to prevent the kernel from overcommitting memory.
Can I use ulimit and systemd resource control together?
Yes, but they apply to different scopes. ulimit applies to your login session. systemd resource control applies to the service. If you SSH into a server and run a command, ulimit limits apply. If that command is a systemd service, the service’s systemd limits apply instead. They do not conflict; they operate on different levels.
Conclusion
Resource limits are not optional on production Linux systems. They are the difference between a service that fails gracefully and a server that crashes at 3 AM. ulimit handles session limits. systemd handles service limits. cgroups v2 is the kernel foundation that makes both work.
Start with systemd resource control for every service you deploy. Set MemoryHigh and MemoryMax. Set CPUQuota. Set TasksMax. Monitor with systemd-cgtop. You will sleep better knowing that no single process can take down your entire server.