Running Splunk without root privileges

If you’re anything like me, when you run a daemon on a Linux system,  you’d prefer that it not run as root. Security, and all that. You may have noticed that some systems make this particularly hard if they want to bind to a privileged port.

In the case of Splunk, I wanted to run as user ‘splunk’ but also have it listen on tcp:514 (syslog). This is… a bit of a pain. So let me walk through how we can achieve it. (I should note that I *am* aware that we could use iptables to accomplish something similar by redirecting 514 to an unprivileged port – but… I don’t want to).

The system:
System: Ubuntu 16.04.02 LTS (Xenial Xerus)

I have created a non-root user called ‘splunk’ with a corresponding group called ‘splunk’ and unzipped the tgz into /opt. I then chown‘d the structure to that user:

splunk-folder

Then issued the following command to register the splunk service.

/opt/splunk/bin/splunk enable boot-start -user splunk

Splunk Started and Running as non-root

started-service

So we can see the service is started. Let’s try to configure a data input on 514.

splunk-fail-bind

As is plainly visible – Splunk is unable to bind to port 514. How can we fix this? Well – I initially just thought “I’ll grant the capability by using setcap”. It was genius, I assure you. It… didn’t work though.

failed-start

Interestingly, as soon as I set the capability, the daemon won’t even start. What is going on here? The journalctl command didn’t reveal as much as I’d like – so I kicked off the /etc/init.d/splunk script and got the following results:

root@gshserver01:/etc/init.d# ./splunk start
Starting Splunk...
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
Did not find "disabled" setting of "kvstore" stanza in server bundle.

Splunk> Like an F-18, bro.

Checking prerequisites...
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
 Checking mgmt port [8089]: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
open
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
 Checking kvstore port [8191]: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
open
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
 Checking configuration... Done.
/opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.2: cannot open shared object file: No such file or directory
Validating databases (splunkd validatedb) failed with code '127'. If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

This is what we know – when setcap is set, the LD_LIBRARY_PATH is ignored (see man ld.so(8)). This is apparently screwing up Splunk’s ability to start. That’s alright, we can create a file in /etc/ld.so.conf.d/ that will add the splunk/lib directory to the linker/loader path.

First – let’s try the following:

added-ld.so.conf

This should work – yes? Yes. I started Splunk and it started perfectly. Excellent. But, here is the thing… if I reboot the computer, sshd won’t start.  What’s the problem? Let’s have a look at sshd:

sshd-failure

So – we see the libcrypto.so.1.0.0 listed, but at the top we see that sshd appears to be looking for /opt/splunk/lib/libcrypto.so.1.0.0.  Why? Well, when we cat the ld.so.conf.d folder – we see the order of libraries:

ld.so-splunk-placement

So, the libcrypto.so.1.0.0 that we want for sshd is actually listed in a directory below /opt/splunk/lib – so we have an ordering problem. So, we’ll change the splunk.conf filename to z_splunk.conf to move it to the end and rerun ldconfig to rebuild the cache.

ld.so-splunk-placement-fix

So, let’s check sshd and see if it fixed the library path:

ld.so-sshd-fixed

This is looking much better now. So – post reboot we’ve got sshd running, Splunk running (as a non-root account). Let’s check if we can bind a privileged port.

splunk-added-syslog.png

… and the splunkd log (and ps) showing what happened when I created the above data input as well showing the service is running as a non-root user (splunk).

Is this the best way to achieve the goal? Hard to say… I’ll keep this instance running for awhile and continue to test to see if anything misbehaves.

Sam

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s