https://www.endpointdev.com/blog/tags/systemd/2024-02-03T00:00:00+00:00End Point DevSetting User-Specific Resource Limits with systemdhttps://www.endpointdev.com/blog/2024/02/setting-user-specific-resource-limits-in-linux/2024-02-03T00:00:00+00:00Seth Jensen
<p><img src="/blog/2024/02/setting-user-specific-resource-limits-in-linux/sun-through-trees.webp" alt="In the center of a dim street lined with houses and cars, the sun illuminates and sillhouettes green foliage. It projects a triangle of light on the ground, first the street and then the grass, centered in the frame. On either side of the light, centered verticaly, a white car catches the light."></p>
<!-- Photo by Seth Jensen, 2023. -->
<p>We recently encountered an issue on one of our servers where some processes were hogging CPU time and slowing down the entire machine to unusable speeds. It turned out that a couple of our developers were using the VS Code extension “Remote - SSH,” which gets SSH credentials to a server from you, logs in by SSH, figures out what kind of server it’s running on, and runs some server-side components that it talks to.</p>
<p>This is fine in principle, but with larger projects it frequently spins out of control and eats up all available CPU and RAM on the server. The developer using VS Code on their desktop likely won’t even notice, leaving a system administrator to kill the VS Code processes after it causes issues for other users.</p>
<p>Our answer to this problem was setting resource limits. In many cases, you could use <a href="https://linuxcommand.org/lc3_man_pages/ulimith.html">ulimit</a> to control a user’s resource limits. But ulimit only controls resources for the shell it’s running in, and since VS Code was connecting through SSH and spawning a bunch of processes, we didn’t have a reasonable way to do this.</p>
<p>Instead of using ulimit, you can use control groups (cgroups) to set overall user limits, not just limits on a specific process or in a specific shell. This is good practice for any multi-user server so that different parts of the server are protected from each other. We will do this by adding a systemd controller to control a specific user slice.</p>
<blockquote>
<p>On PAM systems, you can also use the <code>pam_limits</code> module to set user-wide resource limits. O’Reilly has a <a href="https://www.oreilly.com/library/view/network-security-hacks/0596006438/ch01s20.html">guide on configuring limits</a> this module.</p>
</blockquote>
<h3 id="a-closer-look-at-setting-limits-with-cgroups">A closer look at setting limits with cgroups</h3>
<p>You can add systemd controllers—including resource limits—via drop-in files in <code>/etc/systemd/system/</code>.</p>
<p>To set limits for a specific user, we need the user’s UID. You can find this in the third column of <code>/etc/passwd</code>:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">seth:x:1000:1000:Seth Jensen:/home/seth:/bin/bash
testy:x:1001:1001:Testy Tester:/home/testy:/bin/bash
</code></pre></div><p>For the <code>seth</code> user, the UID is 1000, so we will create a file called <code>50-limits.conf</code> in the <code>/etc/systemd/system/user-1000.slice.d/</code> directory to limit that user’s resources.</p>
<p>The <code>50-</code> in the filename just determines the order in which the config files will be applied, so it shouldn’t matter if this is the only file you’re adding.</p>
<p>Now, in our <code>50-limits.conf</code> file we can add limits:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain">[Slice]
CPUQuota=50%
MemoryLimit=1G
TasksMax=40
</code></pre></div><p>Here we’re just limiting CPU time, memory, and the number of tasks run by the <code>seth</code> user. You can see a full list of what’s available with <a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html">man systemd.resource-control</a>.</p>
<p>It’s worth noting that <code>CPUQuota=50%</code> will limit <code>seth</code> to 50% of a <em>single</em> CPU, which would be 25% of total CPU on a dual-core system, etc.</p>
<p>Finding the right values for your limits is a bit of a balancing act: if you set the resource limits too tight, your programs may not be able to run. But, if you set them too loose, the misbehaving program may still end up hogging too many resources. Try different values while running the resource-heavy program, adjusting as needed to find the sweet spot for your case.</p>
systemd: a primer from the trencheshttps://www.endpointdev.com/blog/2018/06/systemd-primer-from-the-trenches/2018-06-18T00:00:00+00:00Ian Neilsen
<p><img src="/blog/2018/06/systemd-primer-from-the-trenches/6095265888_a27b664798_o-crop.jpg" width="1540" alt="gears" /><br><a href="https://www.flickr.com/photos/guysie/6095265888/">Gears image by Guy Sie, CC BY-SA 2.0, cropped & scaled</a></p>
<h3 id="systemctl-lets-get-back-to-basics">systemctl: Let’s get back to basics</h3>
<p>‘‘Help me systemd, you are my only hope.’’</p>
<p>Sometimes going back to day zero brings clarity to what seems like hopeless or frustrating situation for users from the Unix SysV init world. Caveat: I previously worked at Red Hat for many years before joining the excellent team at End Point and I have been using systemd for as long. I quite honestly have forgotten most of the SysV init days. Although at End Point we work daily on Debian, Ubuntu, CentOS, and BSD variants.</p>
<p>Here is a short and sweet primer to get your fingers wet, before we dive into some of the heavier subjects with systemd.</p>
<p>Did you know that systemd has many utilities you can run?</p>
<ul>
<li>systemctl</li>
<li>timedatectl</li>
<li>journalctl</li>
<li>loginctl</li>
<li>systemd-notify</li>
<li>systemd-analyze - analyze system</li>
<li>systemd-cgls - show cgroup tree</li>
<li>systemd-cgtop</li>
<li>systemd-nspawn</li>
</ul>
<p>And systemd consists of several daemons:</p>
<ul>
<li>systemd</li>
<li>journald</li>
<li>networkd</li>
<li>logind</li>
<li>timedated</li>
<li>udevd</li>
<li>system-boot</li>
<li>tmpfiles</li>
<li>session</li>
</ul>
<p>That’s a long way from the old SysV init days. But in all essence it’s not that different. The one thing that stands out to me is we have more information with less typing then previously. That can only be a good thing, right?</p>
<p>Well, let’s see! There are many many web pages out there that list systemd or systemctl switches/flags. However in everyday use I want to speed up the work I do, I want information at my fingertips, and I find flags and switches which mean something sure do make it easier.</p>
<h3 id="pro-tip-1-tab-completion">Pro Tip 1: Tab completion</h3>
<p>Before you begin playing with the commands, you should install <code>bash-completion</code>. Some distros don’t auto-complete with systemd until you install that, and without tab auto-completion you miss out on <strong>a lot</strong> of systemctl.</p>
<p>As an example when you tab for completion you will see many of the systemctl options:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># systemctl
add-requires enable is-system-running preset show
add-wants exit kexec preset-all show-environment
cancel force-reload kill reboot start
cat get-default link reenable status
condreload halt list-dependencies reload stop
condrestart help list-jobs reload-or-restart suspend
condstop hibernate list-machines rescue switch-root
daemon-reexec hybrid-sleep list-sockets reset-failed try-reload-or-restart
daemon-reload import-environment list-timers restart try-restart
default is-active list-unit-files revert unmask
disable is-enabled list-units set-default unset-environment
edit is-failed mask set-environment
emergency isolate poweroff set-property
</code></pre></div><h3 id="systemctl-vs-old-school-commands">systemctl vs. old school commands</h3>
<p>Here we will list out new systemctl commands and the corresponding old SysV command, followed by systemctl flags and explanation.</p>
<p>Go ahead and run each command to get a feel for what it displays. Remember each command usually has switches/flags you can use.</p>
<p>Let’s start at the top and work down:</p>
<h4 id="systemctl">systemctl</h4>
<p>Formerly: <code>service</code></p>
<p>Used in conjunction with ABRT it can show you some great debug info and runtime metadata, categorized by their respective groupings of loaded, active, running and a description of the unit.</p>
<h4 id="systemctl-status">systemctl status</h4>
<p>Formerly: <code>service --status-all</code> or <code>initctl list</code>.</p>
<p>Check all system services’ status. Normally during a server update I will run this and output it to a file. When the server reboots I can run it again and diff this file to ensure all things started.</p>
<p>The output is great. It shows me the PID path and potentially the arguments which were run for the process or service. Saves me <code>ps</code>ing the process.</p>
<p>Note that each distro deals differently with <code>service --status-all</code> output.</p>
<h4 id="systemctl-status-servicename--l">systemctl status serviceName -l</h4>
<p>Formerly: <code>service serviceName status</code></p>
<p>Good flags: <code>is-active</code>, <code>-a</code>, <code>-l</code></p>
<p>As it suggests, show me the status and information related to the service unit file. Other good info included is whether the service is enabled or chkconfig is on, uptime, PID and cgroup info, and any other information associated with the service.</p>
<p>Tips:</p>
<ul>
<li>The <code>-l</code> flag will usually output enough information to diagnose a service start or reload problem without having to go into the logs.</li>
<li>You can view more than one service by separating them with spaces, e.g.: <code>systemctl status httpd mysql postfix</code>.</li>
</ul>
<h4 id="systemctl-enabledisable-nameofservice">systemctl enable|disable NameofService</h4>
<p>Formerly: <code>chkconfig ServiceName on|off</code></p>
<p>You might find on some distros that <code>chkconfig</code> is still present. It doesn’t do what you think it does with systemd systems.</p>
<h4 id="systemctl-startstoprestart-httpd">systemctl start|stop|restart httpd</h4>
<p>Formerly: <code>service httpd start|stop|restart</code></p>
<p>Good unit commands: reload-or-restart</p>
<p>As it suggests, start, stop, or restart services/processes/units.</p>
<p>The <code>reload-or-restart</code> command tells the services to reload if it is able and if not then restart, similar to the old <code>service serviceName force-reload</code>. Some services don’t allow a reload. Nagios is one example where a <code>reload-or-restart</code> works because it doesn’t allow reloads.</p>
<h4 id="systemctl-reload-httpd">systemctl reload httpd</h4>
<p>Formerly: <code>service httpd reload</code></p>
<p>Perform a graceful reload of a configuration you may have just changed. Example: I’ve just made some changes to httpd conf and need to gracefully reload them without restarting the web service.</p>
<h4 id="systemctl-daemon-reload">systemctl daemon reload</h4>
<p>Formerly: <code>chkconfig serviceName --add</code></p>
<p>Graceful reloads configuration files on a running service/process. See below for an explanation of “daemon reload”. Basically, if you have added in a new service and made many config changes, use <code>daemon-reload</code>.</p>
<h4 id="systemctl-list-unit-files">systemctl list-unit-files</h4>
<p>Good flags: <code>--type=service</code></p>
<p>Formerly: <code>ls /etc/rc.d/init.d/ /etc/rc.d/rc.local</code></p>
<p>Prints unit files from <code>/usr/lib/systemd/system/</code> and <code>/etc/systemd/system/</code>. Slightly different to <code>list-units</code>; rarely used but has any interesting output. You may want to use this in monitoring scripts you write.</p>
<h4 id="systemctl-list-units">systemctl list-units</h4>
<p>Good flags: <code>-a (--all)</code>, <code>-t serviceName</code></p>
<p>Formerly:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">chkconfig --list
ntsysv
ls /etc/rc.d/init.d/ /etc/rc.d/rc.local
sysv-rc-conf
initctl list
</code></pre></div><p>I prefer to use <code>list-units</code> over <code>list-unit-files</code>. It shows more information and is shorter to type. Or you could install the <code>sysvinit-utils</code> package, which by default is not installed on systemd distros.</p>
<h4 id="systemctl-list-sockets">systemctl list-sockets</h4>
<p>Formerly:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">lsof
ss -s
cat /proc/net/*
</code></pre></div><p>Sockets of all types can be viewed from one command. Although time to time I will still use <code>lsof</code> or <code>ss</code> depending on the socket type I want to look at.</p>
<h4 id="systemctl-list-timers">systemctl list-timers</h4>
<p>Formerly: <code>crontab -e</code></p>
<p>systemd offers a way to schedule tasks like crontab. I’m going to go out on a limb here and say they both do the same thing, except systemd timers may be more readable by human eyes, are logged to the journal, easier to debug and enable or disable.</p>
<p>My caveat is that I still use cron jobs in my daily work, because I’m familiar with them and emailing is still an issue from a timer.</p>
<p>Tip:</p>
<ul>
<li>set a systemd timer to a calendar day, month, year to trigger.</li>
</ul>
<h4 id="systemctl-list-jobs">systemctl list-jobs</h4>
<p>Requires further explanation in another blog post.</p>
<h4 id="systemctl---failed">systemctl –failed</h4>
<p>Show me failed services. <code>systemctl status</code> will highlight at the top if units have failed, especially useful after a reboot.</p>
<h4 id="systemctl-get-default">systemctl get-default</h4>
<p>Formerly:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">runlevel
chkconfig --list
</code></pre></div><p>To list targets run <code>systemctl list-units -t target</code>.</p>
<p>Gets the run level default for the system. Not often used, but good to know when you start having boot issues or need to change to a different run level to fix things.</p>
<p>Tips:</p>
<ul>
<li>There are many tables comparing the SysV runlevels to systemd. Get your Google on and search.</li>
<li><code>systemctl set-default graphical.target</code> will set a graphical user shell. For all those that like a good desktop.</li>
</ul>
<h4 id="systemctl-set-default-multi-usertarget">systemctl set-default multi-user.target</h4>
<p>Formerly: <code>telinit runlevel</code></p>
<p>The systemd <code>multi-user.target</code> is equivalent to runlevels 2, 3, and 4.</p>
<h4 id="systemctl-shutdown-or-reboot">systemctl shutdown or reboot</h4>
<p>Formerly: <code>shutdown -r|-h now</code></p>
<p>Reboot/shutdown and poweroff the system. Personally I still use <code>shutdown</code>.</p>
<h4 id="systemctl-cat-servicename">systemctl cat serviceName</h4>
<p>Formerly: <code>cat /etc/init.d/$init_file</code></p>
<p>Shows me the system service (unit) file contents and options. We will go more into these commands later as we work through building and maintaining our own unit files.</p>
<p>Tips:</p>
<ul>
<li><code>systemctl cat</code> shows all unit file information and snippets involved with the unit file.</li>
<li>If you use Vim and Arch Linux, you can enable <code>vim-systemd</code> to help with syntax highlighting.</li>
</ul>
<h4 id="systemctl-list-dependencies-servicename">systemctl list-dependencies serviceName</h4>
<p>What’s really depending on a given service. The <code>--all</code> flag will show everything from <code>--before</code>, <code>--after</code>, <code>--reverse</code>.</p>
<h4 id="systemctl-show-servicename">systemctl show serviceName</h4>
<p>Flags: <code>-p</code> shows a single property of a service.</p>
<p>Shows more than using <code>systemctl cat servicename</code>. Don’t forget tab completion is your friend. Running the <code>-p</code> flag and using tab will help you.</p>
<h4 id="systemctl-mask-servicename">systemctl mask serviceName</h4>
<p>Formerly: <code>update-rc.d serviceName disable</code></p>
<p>Never want someone starting a service <strong>ever</strong>? <code>mask</code> is your friend and a little sneaky. See if your system admins pick this one up. Good April Fool’s day trick on them. Use <code>unmask</code> to return it to its normal state.</p>
<h4 id="systemctl-edit-servicename">systemctl edit serviceName</h4>
<p>Formerly: edit <code>/etc/init.d/scriptName</code></p>
<p>Good options: <code>--full</code></p>
<p>Yes, that’s right! Edit the service file without having to go find it on disk. That has saved me a bit of time.</p>
<p>Tips:</p>
<ul>
<li>Careful with this. The plain edit creates an override file in /etc/systemd/system to complement the original unit file.</li>
<li>If you need to edit the original unit file use the <code>--full</code> flag, which allows you to edit the unit file without creating a snippet.</li>
<li>If you make a mistake in your unit file: <code>systemctl revert serviceName</code></li>
</ul>
<h4 id="systemctl--o">systemctl -o</h4>
<p>Formerly: Edit Apache config to set log level to warn or debug, <code>/etc/init.d/httpd reload</code>, view logs</p>
<p>Good options: <code>--output=verbose</code></p>
<p>Particularly good if you have a service acting up. Outputs a short standard message or a very verbose message using different flags.</p>
<p>Tip:</p>
<ul>
<li><code>journalctl -u serviceName</code> can help you here, but I often find it easier to include <code>--output=verbose</code></li>
</ul>
<h4 id="systemctl-isolate">systemctl isolate</h4>
<p>This deserves its own small blog post. Isolate can be used to rescue systems automagically following kernel reboot failures, but requires some special work.</p>
<h4 id="systemd-delta">systemd-delta</h4>
<p>Check your unit files to see if someone has been changing things on you. Especially useful if you are writing your own unit files.</p>
<p>Tip:</p>
<ul>
<li>Used in conjunction with <code>systemctl cat</code> or <code>edit</code>, <code>delta</code> can help you see what was what.</li>
</ul>
<h3 id="gotchas">Gotchas</h3>
<ol>
<li>Sometimes you need to use the suffix such as ‘config_<a href="mailto:file@openvpn.service">file@openvpn.service</a>’. systemd will always think services are services unless you use the suffix like .target or .socket, so make sure you tell the system so.</li>
<li>If you want multiple services running use a prefix, such as; <a href="mailto:ssh1@sshd.service">ssh1@sshd.service</a> <a href="mailto:ssh2@sshd.service">ssh2@sshd.service</a> with different configs. Handy for multiple openvpn servers.</li>
<li>Mount points will always be determined as mount points.</li>
<li>If you have made a lot of configuration changes and want to gracefully load these without restarting everything try <code>systemctl daemon reload</code>. This is not the same as the above <code>reload</code> action. Daemon reload is for systemd and not the unit files it controls. This is a safe command to run since it keeps sockets open while it does its thing.</li>
<li>Reboots on newer systems only really need to be done when new kernels are presented. Systemd is gracious and good at processing new packages and enabling these changes during a yum update.</li>
<li>Autocomplete on CentOS is not present until you install <code>bash-completion</code>. systemctl takes on a life of its own when you install this utility. Tabbing out will list all systemctl options. Very handy!</li>
<li>Systemd does not use the /etc/inittab file even if you have it present.</li>
<li>Converting your init scripts to systemd is easier than you think. Create a systemd unit file and add in 10 basic lines to call the bin or init script. You have basically created a systemd managed init script. Don’t forget to go back and one day convert it completely.</li>
<li>Targets vs. runlevels: A target in systemd is a runlevel in sysV, names replace numbers; runlevel 3 = multi-user.target, runlevel 1 = rescue.target</li>
</ol>
<h3 id="whos-got-the-goods-on-speed">Who’s got the goods on speed?</h3>
<p>Is systemd faster than SysV init? Parallel processing says it is? You be the judge!</p>
<p>One great test is to build a new machine which doesn’t have systemd installed. Reboot the machine and check your boot time. On an older SysV system you may have use “tuned”, “systemtap”, “numastat” etc. to gather performance information.</p>
<p>Then install systemd. Better still, upgrade from a old version of Linux to a new version of linux from ‘init’ to ‘systemd’ and then run ‘systemd-analyze’.</p>
<p><code>systemd-analyze</code> will show you boot times. Notice that systemd starts fewer services at boot because it only starts what is necessary to get the server booting.</p>
<p>Not a bad way to collect a baseline on a newly-built server. Add it to your Ansible facts for the server so you have a historical view and collection of boot times. In fact add it to your monitoring system and be proactive in your monitoring of server boot times while performing your maintenance cycles.</p>
<p>How do you see what is taking its sweet time during boot or what is borking your beautiful server following an update/upgrade? Wonder no more!</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># systemd-analyze blame
# systemd-analyze time
</code></pre></div><p>For example on my system, I can see that my top 10 culprits for a potentially slow boot are:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># systemd-analyze blame
7.346s dracut-initqueue.service
6.787s systemd-cryptsetup@luks.service
5.378s NetworkManager-wait-online.service
2.308s abrtd.service
1.444s docker.service
1.395s plymouth-quit-wait.service
1.358s lvm2-pvscan@8:1.service
1.145s lvm2-pvscan@253:0.service
1.072s fwupd.service
609ms docker-storage-setup.service
</code></pre></div><p>I might disable the Docker service by <code>systemctl disable docker</code> and start it when I need it.</p>
<p>What else can <code>systemd-analyze</code> show me?</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
graphical.target @5.209s
└─multi-user.target @5.209s
└─abrt-journal-core.service @5.209s
└─abrtd.service @2.897s +2.308s
└─livesys.service @2.881s +13ms
└─basic.target @2.804s
└─sockets.target @2.804s
└─dbus.socket @2.804s
└─sysinit.target @2.797s
└─systemd-update-utmp.service @2.782s +14ms
└─auditd.service @2.634s +145ms
└─systemd-tmpfiles-setup.service @2.590s +41ms
└─fedora-import-state.service @2.566s +22ms
└─local-fs.target @2.562s
└─run-user-42.mount @4.614s
└─local-fs-pre.target @964ms
└─lvm2-monitor.service @321ms +442ms
└─dm-event.socket @320ms
└─-.slice
</code></pre></div><p>Let’s look at how long our network targets take to start:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-plain" data-lang="plain"># systemd-analyze critical-chain network.target
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.
network.target @2.618s
└─network.service @2.403s +214ms
└─NetworkManager-wait-online.service @1.458s +941ms
└─NetworkManager.service @1.417s +40ms
└─network-pre.target @1.415s
└─firewalld.service @976ms +438ms
└─polkit.service @891ms +83ms
└─basic.target @885ms
└─paths.target @885ms
└─brandbot.path @885ms
└─sysinit.target @881ms
└─systemd-update-utmp.service @872ms +7ms
└─auditd.service @702ms +168ms
└─systemd-tmpfiles-setup.service @676ms +24ms
└─rhel-import-state.service @653ms +22ms
└─local-fs.target @652ms
└─boot.mount @523ms +128ms
└─local-fs-pre.target @522ms
└─lvm2-monitor.service @369ms +152ms
└─lvm2-lvmetad.service @397ms
└─lvm2-lvmetad.socket @368ms
└─-.slice
</code></pre></div><p>Let’s go one step further and output the entire system hierarchy. With this one you get a nice image:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># systemd-analyze dot | dot -Tpng -o system-stuff.png</span>
</code></pre></div><h3 id="pro-tip-2-remote-commands">Pro Tip 2: Remote commands</h3>
<p>I have a server which needs a service restarted or checked constantly. Running systemctl remotely will show me or allow me to do this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># systemctl status sshd -H root@server.domain.tld</span>
</code></pre></div><p>Or:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># systemctl -H root@server.domain.tld status httpd</span>
</code></pre></div><p>I might make an alias for it. Obviously this is a pretty useless example, if you’re having to manually do this for a service/process you should fix the problem on the server. However for edge cases it can be quite handy. Use your imagination: We could use this for monitoring which takes a local ssh user found on all machines and pass this for some returned output to a monitoring server.</p>
<h3 id="pro-tip-3-what-relies-on-my-service">Pro Tip 3: What relies on my service</h3>
<p>Figure out what targets/runlevel a target runs at:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># systemctl show httpd -p wants multi-user.target</span>
</code></pre></div><h4 id="pro-tip-4-monitoring">Pro Tip 4: Monitoring</h4>
<p>Check processes, service association, and busiest processes. You can still grep/awk the output if you wish.</p>
<p>Instead of using <code>top</code> or something like this:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash">ps xawf -eo pid,user,cgroup,args
</code></pre></div><p>use the following:</p>
<div class="highlight"><pre tabindex="0" style="background-color:#fff;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-bash" data-lang="bash"><span style="color:#888"># systemd-cgls</span>
<span style="color:#888"># systemd-cgtop</span>
</code></pre></div><h3 id="summing-up">Summing up</h3>
<p>I know this is only touching the surface of systemctl or systemd as whole but from a day-to-day context this should help you play in the systemd world. I think the biggest part that people struggle with is workable, usable examples.</p>
<p>Stay tuned for future posts on unit files, unit targets, systemctl isolate and slices, journalctl, timedatectl, and loginctl.</p>