The memories of three separate SSH sessions I had to connect to three different virtual machines (VMs) in order to look through their ‘/var/log/auth.log’ files are pretty clear in my head as of now. I’m saying that I’m currently looking at the logs of three different VMs in order to determine what caused the failed brute-force attack on SSH. As I was looking at the logs, I couldn’t believe how much time it took for me to jump from one VM to another and then look at that log file again. My conclusion after all that was: “I need to put all the logs into one central place; however, I do NOT want the logs to be transmitted in ‘plaintext'” (i.e. in an unsecured format). And after these thoughts crossed my mind, I sat on the sofa and finally also got around to getting rsyslog TLS centralized logging correctly set up.
As for what is being prepared for you, here is:
- A self-signed Certificate Authority to generate a certificate signing request (CSR) for the servers and clients.
- A single rsyslog server that receives encrypted log messages from remote locations (via port 6514).
- Rsyslog clients that use mutual TLS to send logs over a secure encrypted channel.
- Optionally, the use of an stunnel wrapper for legacy systems that are not configured to use TLS, to allow them to securely send logs.
- A remedy for some of the more common errors seen when performing a TLS handshake.
Understanding linux log centralization and remote syslog encryption
We’ll provide you with the complete understanding of how to configure both the Linux Log Concentrator and how to implement encryption over the network using TLS. Although the ability to log all logs to one place provides convenience, it also provides additional security in knowing that sensitive information cannot be viewed by an unauthorized user. Although we are going to implement a server/client model, at every point where logs are being generated, there will be an Rsyslog Forwarder running on that node.The forwarder connects via TCP to the central server on Port 6514 – a port specifically designated for Syslog TLS according to [RFC 5425]. The connection is encrypted using TLS and includes mutual authentication. The server supplies a certificate to the client, while the client supplies a certificate to the server that is verified against the private Certificate Authority (CA) of the Client.
The rsyslog client server tls architecture
Figure the flow of information into a rsyslog system this way: Applications send their log messages to their local rsyslog client. The logs are sent down a TLS secured connection (or pipe) to the rsyslog Server (the central logging server). Finally, upon arrival at the server, the log messages are written to disk.
The TLS layer is implemented using the GnuTLS library (the gtls stream driver in rsyslog). For the client and server to establish a encrypted and authenticated connection, they both require the CA certificate. They each hold their own cryptographic public/private key pair signed by the CA mentioned previously. The client does not require user passwords or Preshared Keys (PSK). The authentication and secure channel is validated using Public Key Infrastructure.
You can think of this in terms of three separate containers: the rsyslog Client’s certificate and CA certificate, the secure TLS pipe, and the rsyslog Server’s certificate and CA Certificate. The client is responsible for providing proof of identification to the server using Certificates, while the server is responsible for authenticating the Client’s identity using its own Certificate.
Generating the rsyslog ca certificate and cryptographic keys
For our purposes, we will build a local CA and use that to issue Client and Server Certificates from the local CA. This way we will be fully self-contained. Internally, we do not need to have an external public Certificate Authority. OpenSSL is my preferred method to create and manage private keys and certificates, as OpenSSL is found on every Linux distribution available.
Setting Up a Local Certificate Authority
To create the CA, create a local Directory (directory named just like your CA) and generate a Private Key for your CA. Keep this key in a safe place; if someone gets this key, they will be able to impersonate all of your nodes.
mkdir ~/rsyslog-ca && cd ~/rsyslog-ca
openssl genrsa -out ca-key.pem 4096
openssl req -new -x509 -days 3650 -key ca-key.pem -out ca.pem -subj "/CN=MyLoggingCA"
We will generate a Private Key of 4096-bits using the RSA algorithm, along with a self-signed CA Certificate that will be valid for a maximum of ten years.Simply assign the common name (CN) or common name to the Subject.
Issuing Server and Client Certificates
You will need to create a Key and Certificate Signing Request (CSR) from the Server. The CN must match the hostname that Clients will connect to.
# Server
openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -out server.csr -subj "/CN=logserver-01.int"
openssl x509 -req -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out server-cert.pem -days 3650
You will repeat this process for each Client; however, you will need to assign a unique CN per Client.
# Client web-01
openssl genrsa -out web-01-key.pem 2048
openssl req -new -key web-01-key.pem -out web-01.csr -subj "/CN=web-01.int"
openssl x509 -req -in web-01.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out web-01-cert.pem -days 3650
At this point, you will have a collection of .pem files. The CA Certificate (ca.pem) must be found on every machine (including the Server and all Clients). The Server will have its own key/cert pair, and each Client will have its own key/cert pair.
How to configure rsyslog tls centralized logging
Why I ultimately chose this route
There are several other methods of logging using SSH or stunnel (pipe syslog into SSH tunnels or stunnel); however, when using rsyslog with TLS and mutual authentication you can have very fine grained control. For example, you can apply reload rules and filter Client by hostname, while not having to run another stunnel process that consumes system resources. The only possible disadvantage is that you will have to manage a few more certificates; however, this is easily accomplished by writing a small CA and script. The process is cleaner than any other way.
Now we need to create the /etc/rsyslog.conf configuration files. We will break this down into Client and Server configurations.
Enabling the TLS Listener on the Server
To create the central log Server, we need to load the gtls module and create a new input. Specifying the paths for the CA certificate, Server Certificate and Server key are critical. Set the Stream Driver to gtls and enable Mutual Authentication by setting the Stream Driver Authentication Mode to x509/name.To add support for encrypted syslog messages with rsyslog, we must include the following statement in both our /etc/rsyslog.conf file as well as a file under the /etc/rsyslog.d/ directory:
module(load="imtcp")
module(load="gtls")
input(type="imtcp" port="6514"
StreamDriver.Name="gtls"
StreamDriver.Mode="1" # TLS mode
StreamDriver.AuthMode="x509/name"
PermittedPeer=["CN=web-01.int","CN=db-01.int"]
)
We have included both the imtcp and gtls modules. With respect to imtcp, the setting StreamDriver.Mode=1 selects Transport Layer Security (TLS), while StreamDriver.Mode=0 uses unencrypted TCP. The setting StreamDriver.AuthMode=x509/name requires rsyslog to verify that the Common Name (CN) of the client certificate matches an entry in the PermittedPeer list; therefore, only those CNs that match the permitted list will be allowed by rsyslog to connect. The official rsyslog documentation on gtls details all of the options associated with the gtls driver.
In the same way as you told rsyslog where to locate the imtcp and gtls certificates (above), it’s imperative to provide rsyslog with the location of the certificates used for both action items. The preferred method is to create the following file, as sample code below:
global(
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca.pem"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/server-cert.pem"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/server-key.pem"
)
You will copy your ca.pem, server-cert.pem, and server-key.pem files to this new directory. You must then restart the rsyslog daemon using the command systemctl restart rsyslog.
Setting Up the rsyslog secure forwarder
The next step would be to create an action that sends log messages to the server from each of the clients. To accomplish this, we will use the same gtls module but will need to define the destination server and authentication.
module(load="gtls")
action(
type="omfwd"
Target="logserver-01.int"
Port="6514"
Protocol="tcp"
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.AuthMode="x509/name"
StreamDriver.PermittedPeer="CN=logserver-01.int"
)
The purpose of this action is to send all log messages generated by the client to the server. In addition to the ca file, clients need to have their own Certificate Authority (CA) file and a key/certificate file combination, as described above. Therefore, each client must have a global block that references its own certificate files, which will be constructed similarly to what was done for each of the server’s certificate files.
global(
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/certs/ca.pem"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/certs/web-01-cert.pem"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/certs/web-01-key.pem"
)
At this point, the rsyslog secure forwarder has been set up so that all messages transfer encrypted using TLS while they are in transit.
Enhancing Reliability with stunnel rsyslog Integrations
Continuing from the above section, we need to discuss configuring rsyslog with stunnel. Some older devices may still be able to send plain TCP syslogs through port 514 only, and/or they may want to offload performing the TLS encryption of its syslog messages to a separate process to allow for increased performance. In this case, we can use stunnel as a wrapper for the rsyslog daemon.
Configuring Stunnel for Legacy Syslog Clients
I’ve used stunnel with an older network switch that sends syslog, but does not support TLS. Therefore, I have stunnel configured on a small jump server to listen for incoming UDP on 514 and securely forward the logs (via TLS) to the central syslog server on 6514.
Below is my simple /etc/stunnel/stunnel.conf file.
[syslogtls]
client = yes
accept = 0.0.0.0:514
connect = logserver-01.int:6514
cert = /etc/stunnel/web-01-cert.pem
key = /etc/stunnel/web-01-key.pem
CAfile = /etc/stunnel/ca.pem
verify = 2
The configuration will accept unencrypted syslog communication on port 514 and wrap it in TLS before forwarding. As far as the TLS receiver is concerned, all connection requests are accepted as a normal TLS connection. The stunnel manual provides all of the verification options for configuring your verification level. If you cannot use rsyslog natively configured for TLS, this can serve as a substitute for receiving syslog via encrypted means.
Diagnosing “gnutls returned error” and Certificate Denials
You may encounter GnuTLS errors in your logs, which indicate that the connection was rejected. The two most common reasons for this are incorrect file permissions (such as those set by SELinux or AppArmor) or a mismatch between the host name used to establish the connection and the host name in the certificate.
Fixing AppArmor and SELinux Certificate Denials
By default, RSyslog operates in a confined environment. The rsyslog profile is likely not configured to allow Rsyslog the ability to read files stored in /etc/rsyslog.d/certs/. The first step is to check your audit logs for any denials related to the AppArmor or SELinux profile. The audit logs for typical SELinux denials will appear as follows:
type=AVC msg=audit(1678901234.567:890): avc: denied { read } for pid=1234 comm="rsyslogd" name="ca.pem" dev="dm-0" ino=56789 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0
To correct the issue, simply relabel the certificate directory to ensure that it has the correct context. The simplest way to relabel your directories is to use the following commands: semanage fcontext -a -t cert_t “/etc/rsyslog.d/certs(/.*)?” and restorecon -R -v /etc/rsyslog.d/certs. You can learn more about SELinux and Rsyslog from the SELinux project wiki.
For AppArmor, the same steps can be performed to resolve the issue of AppArmor not allowing reading of certificate files stored in /etc/rsyslog.d/certs/, for example, adding the following rule to your Rsyslog AppArmor profile: /etc/rsyslog.d/certs/** kr, and reloading the profile.
Validating Remote Port Connectivity
Before blaming certificate issues, you should confirm that the port you are trying to reach is indeed accessible via OpenSSL. Test your TLS handshake with:
openssl s_client -connect logserver-01.int:6514 -CAfile /etc/rsyslog.d/certs/ca.pem -cert /etc/rsyslog.d/certs/web-01-cert.pem -key /etc/rsyslog.d/certs/web-01-key.pem -state -debug
This will output something similar to the following: SSL handshake has read … bytes and written … bytes, followed by a successful return code for verification of your certificate. If you see the message verify error:num=18:self signed certificate, you will need to ensure that your CA is trusted. If GnuTLS returns an error while attempting to connect, double check that the CN matches exactly what you have requested on the PermittedPeer parameter.
Frequently Asked Questions
Does TLS encryption significantly impact rsyslog performance?
With regard to how much additional resource usage will occur with the addition of TLS to Rsyslog, there should be no significant issues with performance on modern hardware, as I have tested myself by successfully processing 10,000 messages per second within a dual-core Linux server without any issues. The heaviest load associated with the introduction of TLS is normally during the session setup; however, after establishing a session, symmetric cryptography processes messages extremely fast.
Can I use self-signed certificates for remote syslog encryption?
Yes, you can use self-signed certificates for securing remote syslog messages, but you will still need a CA structure since, without a CA, you will not have an easy method of establishing mutual authentication with the server. At a minimum, I always create a root CA, even when operating in an internal environment, to make it easier to manage trust.
How do I verify that logs are actually encrypted in transit?
To ensure that your logs are encrypted during transit, perform a tcpdump on the server-side interface: tcpdump -i eth0 port 6514 -X, which will display your logs in the format of a stream of random binary characters, rather than a readable format. Additionally, you can review the connection statistics reported by Rsyslog to see if an “TLS session established” message was received. Use journalctl -u rsyslog -f to show connection statistics.
So, this configuration is tried and true. Once your certificates have been configured and your configuration snippets have been added, then the process of centralizing log storage while ensuring that the logs remain encrypted is simple. Now you can reduce the number of machines that need to be maintained and secured from unauthorized access.
