I’m haunted by the memory of that frantic phone call from a small business owner who needed help. His contract with a freelance developer was coming to an end in two weeks. The freelance developer requested access to the shared staging environment via SFTP to upload files, but the client wanted to limit their access and prevent them from snooping on other projects as well as obtaining copies of database configurations. While the request was reasonable, granting users access through default SSH would have provided a full shell access to the server and allowed them to navigate through the server’s file system. The alternative of providing separate FTP daemon would require a new user database with complex firewall configurations to manage. Instead, the solution began to take shape using the native chroot capabilities built into OpenSSH. After many late nights working through “Broken pipe” errors and permission issues, I eventually built an isolated SFTP environment that has been in continuous use since then, and this post describes in plain English how I achieved this installation.
Quick Summary
- Isolate users in a SFTP environment that only provides access to their files via the OpenSSH internal-sftp subsystem.
- Create a group to which the users belong, give them a nologin shell and restrict their directory access using Match Group entries in the sshd_config file.
- We resolved the notorious “connection closed” issue by understanding that chroot directories should be owned by the root user only.
- Since we needed to provide written access to some of the subdirectories within the chroot environment, we were able to do it without breaking the isolation of the users.
- Further restricting deployment by disabling forwarding and directing SFTP logs to a different file.
Designing the SFTP Isolated Environment
Before configuring files, I first mapped out what “isolated” would mean for my situation. My project involved a need for secure file transfer using Linux, but for the users I wanted the users to be in a specific folder tree structure, not be able to see higher up in that structure, and not be able to execute shell commands, tunnels, or use X11 sessions.
Securing the Subsystem Architecture
I had tried to make the chroot work with sftp-server, but I thought it was way too fragile. When I switched to using the internal-sftp subsystem, this completely changed how I thought about securing. With the internal-sftp setting, the SSH daemon itself handles all aspects of the SFTP protocol; there’s no need to spawn a separate process to run sftp-server, which means you do not need any binary files such as /bin or any library files in your chroot environment. As a result, the user’s session is 100% contained within the directory that you defined when setting up the internal-sftp configuration.
What Did Not Work for Me: In the early days of my setup I spent an entire weekend trying to retrofit an existing FTP server called vsftpd with virtual users and TLS. Unfortunately, I ran into problems with the passive port ranges on the load balancer, and the separate database for users went out of sync with the system account databases. I scrapped the whole idea and started using the internal-sftp, which has fewer moving parts and runs on the current SSH port.
Threat Mitigation in Shared Hosting
In a scenario similar to the one I just described, a shared-hosting setup means that user A should not be able to see directory listings of user B or be able to read files from user B. A chroot jail would mean that user B cannot even “see” user A’s files, when you use a strict set of permissions on the jail directory, and set the root of the jail to the user’s specific path. If you also use a ForceCommand to restrict to just internal-sftp, it provides a tunnel for that user to access their own files. However, this does not allow the user to execute shells, scp, or port-forward.
System Prerequisites and Account Planning
Good account hygiene is key for a successful chroot jail. I have three steps to accomplish this.
Creating the Dedicated SFTP Group
This is the group that will own the chroot and command restriction logic. There will later be a Match Group block in /etc/ssh/sshd_config, which will restrict chroot and force commands only for users who are members of this group.
sudo groupadd sftp-only
This is the establishment of the sftp-only group. Users will only be placed into this group and restricted, if I explicitly do so.
Provisioning Restricted User Accounts
These users must be present on the system but not allowed to access a shell prompt. Therefore, I assign these users with a login shell of /sbin/nologin (or /bin/false if you’re using a different distro), a home directory within the chroot root, and membership in the sftp-only group.
sudo useradd -g sftp-only -s /sbin/nologin -d /upload/jack jack
sudo passwd jack
The -d flag in the command specifies the user’s home directory within the chroot. I have not created the user’s home directory yet, that will be done later. The use of the /sbin/nologin shell completely disables SSH terminal access for these users, so if a user tries to ssh jack@host it will disconnect immediately.
Generating Key-Based Credentials
In automation pipelines, I do not use a password, I instead use a SSH Key. As the user (or using sudo), I will then place the public key into /.ssh/authorized_keys, giving it 600 permissions. The key works within the jail since authentication takes place before the chroot is enforced.
How to Configure Chroot Jail SFTP Linux
The next step in configuring a chroot jail is to configure your SSH daemon and I will go through /etc/ssh/sshd_config in detail, explaining every block so that you do not just blindly copy and paste lines without understanding.
Activating the sshd subsystem sftp internal-sftp
The first step is to look for the line in the configuration that defines the SFTP subsystem. For most distributions, this will be set to an external sftp-server binary that is commented out. I replace it with “internal-sftp” instead.
# Subsystem sftp /usr/lib/openssh/sftp-server
Subsystem sftp internal-sftp
The configuration above tells the SSH daemon that it will manage the SFTP service internally, so there is no need for any additional executable files to be placed within the chroot jail. Furthermore, this configuration allows for advanced logging capabilities in the future.
Implementing the sshd_config match group Block
Right after the subsystem line or at the end of the file, I insert the Match Group block that only applies to the sftp-only group members.
Match Group sftp-only
ChrootDirectory /sftp/jail/%u
ForceCommand internal-sftp
PasswordAuthentication yes
As shown above, %u is a placeholder for the user that is connecting to the server. Therefore, when jack connects to the server, he will be placed in /sftp/jail/jack for his chroot directory. The “ForceCommand internal-sftp” directive ensures that the user will only be able to use the SFTP protocol when they connect. Because some of my clients still require PasswordAuthentication for certain tools in their pipelines, I left the PasswordAuthentication set to yes within the Match Group block.
Validating and Restarting the SSH Daemon
Before I reload the production SSH config file, I always make sure to run a syntax check against it first. A single typographical error in the Match Group block could easily lock me out of my own server.
sudo sshd -t
If there are no error messages when running the syntax check command, you can be confident that your syntax is correct. After verifying this, I will go ahead and restart the SSH daemon.
sudo systemctl restart sshd
During this time, I keep a separate terminal window open with a root shell, just in case anything goes wrong. It’s better to be safe than sorry.
Troubleshooting: The Root Ownership Chroot Rule
This particular issue has caused me a lot of headaches over the years. I would connect to the server, and see a message that said “Connected to server …” but shortly after that the client would immediately disconnect and either show a “Broken pipe” error message or “Connection closed”. After troubleshooting this issue for a while, it finally became apparent that there was an issue with the owner permissions of the chroot directory.
Enforcing Strict chrootdirectory permissions
OpenSSH is very rigid regarding the ChrootDirectory setting. Every directory in the defined ChrootDirectory path must have owner root and the entire path must not be group- or other-writable. If any element of the path to the ChrootDirectory contains permissions that are too permissive, OpenSSH will not allow the daemon to create the environment needed to serve the connection and will terminate it.
Correct permissions on directories are detailed in the sshd_config manual under the definition of ChrootDirectory. All directory components should be owned by root and have either the mode of 755 or more restrictive. When I first read the manual, I didn’t review this aspect and spent more than an hour trying to diagnose the connection damage in the log files.
Correcting Jail Root Ownership
I created my user’s home directory with the following: chown jack:jack. This was an error; the jail directory must always be owned root:root. I fixed this by doing the following:
sudo mkdir -p /sftp/jail/jack
sudo chown root:root /sftp/jail/jack
sudo chmod 755 /sftp/jail/jack
Yes, the jail directory is actually owned by the user jack but this is done to prevent the user from being able to rename or delete it, which would result in diminished security. Although the SFTP server runs as root with higher privileges, it still has visibility of the user’s home directory.
Creating Writable Subdirectories for Users
Having a read-only jail doesn’t allow users to upload any files. Therefore, I created a dedicated subdirectory for uploads within each jail where I designated ownership to the user:
sudo mkdir /sftp/jail/jack/uploads
sudo chown jack:sftp-only /sftp/jail/jack/uploads
sudo chmod 755 /sftp/jail/jack/uploads
Now, jack can read, write and delete his files in the uploads directory while the jail itself remains secure from his access. Creating a location where users can upload files is the most common method of controlling sftp directory access while still allowing the user to conduct business activity.
Advanced Auditing and Access Control
By locking the directory, I still have only done half of the work; I still need to monitor activity and close any non-standard access.
Disabling TCP Forwarding and X11
In the Match block of my configuration, I specifically disable the ability to create a tunnel with SSH. Even when using ForceCommand internal-sftp, which prohibits shell commands from being executed, there are still ways to publicly exchange TCP connection via SSH before the execution of that command occurs. So I disable this capability in advance.
Match Group sftp-only
ChrootDirectory /sftp/jail/%u
ForceCommand internal-sftp
AllowTcpForwarding no
X11Forwarding no
This will prevent the connection from being abused (via “poor man’s proxy”) and will prevent the creation of an X11 forwarding socket.
Configuring Custom Logging for internal-sftp
Normally, SFTP transfers are written into the same log as all other forms of authentication. I want to maintain an independent file that I can send directly to my SIEM. I use internal-sftp’s -l option to set the log level and then use a socket work-around to pipe logging into syslog.
Initially, I modified the subsystem definition line to include -l INFO and use /dev/log inside the chrooted environment, but the environment does not have the normally created /dev device entries. Rather than making a bind mount, however, I have developed a simpler solution to modify the definition so that logging is sent out via a facility-other than that normally used to write to syslog. The cleanest method is to utilise the internal-sftp’s AUTH facility and send the output to a separate file using rsyslog so that you do not need to make a bind mount to /dev/log.Instead, I left the subsystem at internal-sftp and added a configuration snippet in /etc/rsyslog.d/sftp.conf to filter by program name.
if $programname == 'internal-sftp' then /var/log/sftp.log
& stop
With rsyslog restarted, /var/log/sftp.log now contains all internal-sftp session detail records (opens/closes/renames of files), the format of which I confirmed against the OpenSSH sftp-server man page. Having had this ability to produce a single log file saved my butt when my auditor asked for a complete timeline of file access.
Frequently Asked Questions
Why can’t my SFTP user list the contents of their root directory?
99% of the time it’s a permissions problem with the jail’s root. For example, don’t forget that the ChrootDirectory must be set with ownership root:root and mode 755. If you inadvertently changed the ownership of the ChrootDirectory to the user, OpenSSH will reject the session prior to the directory actually being read. Check the permissions/owner/group of /sftp/jail/username by running ls -ld /sftp/jail/username to confirm it is root:root.
Does a chroot jail prevent users from executing shell commands over SSH?
Yes, because I’ve set the default shell for the user to /sbin/nologin and also set the ForceCommand option to be internal-sftp. So if anyone tries ssh user@host 'rm -rf /' during the SSH authentication process, that command will get ignored and only the SFTP functionality will run. The SSH authentication will still be successful, but the user will only have limited access.
Can I use dynamic variables like %h or %u in the ChrootDirectory path?
Of course! I often use %u (user name) because it allows me to keep the jails organized. You could also use %h (the user’s home directory as defined in /etc/passwd), but this will become more complicated if the user has multiple home directories across the disks. A better option is to define your path template as /sftp/jail/%u, which will allow for predictable organization and also allow for greater automation of configuring using CM tools.
This is the configuration I’ve been running in production for my thin-client upload portals as well as providing developers access to files for contractual purposes. I don’t have to run multiple daemons or configure FTP for each user’s needs; OpenSSH will handle all of the heavy lifting for users. After you resolve the user ownership quirk for the root directory, a properly configured chroot jail for SFTP-Linux remains extremely stable.
