AWS Beanstalk Sensitive Info Exposed

The best way to learn something is to create projects from scratch.  Once I have free time I always try to create different types of projects.  Right now I decided to create an event-driven process to make data transformation/processing/real-time searching (Talking about the Big Data). I have standard ETL architecture using AWS S3, AWS SQS, AWS Lambdas. I also use AWS Lambdas to create data partitions on S3 and  Elastic beanstalk for data processing.  

I've just found that some sensitive information was exposed. I found the following header in the HTTP response:

Server: Apache/2/4/39 (Amazon) OpenSSL 1.0.2l-fips

The solution is pretty straightforward. We need to open the Apache documentation and read it. Note that if you need to completely remove the header you have to install the mod_security module. On the AWS EBS side, everything is simple. We need to create a new config for the beanstalk cluster. 

Location: Amazon S3/$bucket name/build/$app/.ebextensions/httpd/conf.d/

Next time I may share the entire project architecture - how I created real-time processing/searching on AWS env, why I use AWS Lambdas, why I create parquet files for the metadata, how I run distributed processing at AWS Beanstalk with Apache Ignite etc.

Have a good holiday!

Fork Join pool - sometimes the solution is really simple

Let's discuss how people try to use the Fork Join pool in Java 7. Imagine that you have a class that extends RecursiveTask.

@Override
protected BigInteger compute() {
    if ((n - start) >= THRESHOLD) {
        Collection<FactorialTask> tasks = ForkJoinTask
                             .invokeAll(getSubTasks());
        BigInteger result = BigInteger.ONE;
        for (FactorialTask t : tasks) {
            result =  t.join().multiply(result);
        }
        return result;
    } else {
        return calculate(start, n);
    }

}

Ok, and do you know how some people try to transfer this code into the java 8 style? Yep, that's the code that you may find when you surf the web:

  @Override
   protected BigInteger compute() {
       if ((n - start) >= THRESHOLD) {
           return ForkJoinTask
             .invokeAll(createSubtasks())
             .stream()
             .map(ForkJoinTask::join)
             .reduce(BigInteger.ONE, BigInteger::multiply);
       } else {
           return calculate(start, n);
       }
   }

Off-course they still have additional methods:

    private BigInteger calculate(int start, int finish) {
        return IntStream.rangeClosed(start, finish)
                .mapToObj(BigInteger::valueOf)
                .reduce(BigInteger.ONE, BigInteger::multiply);

    }
    private Collection<FactorialTask> getSubTasks() {
        List<FactorialTask> tasks = new ArrayList<>();
        int mid = (start + n) / 2;
        tasks.add(new FactorialTask(start, mid));
        tasks.add(new FactorialTask(mid + 1, n));
        return tasks;
    }

In order to invoke:

public BigInteger factorial(Integer number){
    ForkJoinPool pool = ForkJoinPool.commonPool();
    return pool.invoke(new FactorialTask(number));
}

BUT wait! In Java 8+ we have a parallel stream, why don't rewrite this code in one line? Everything is  just simple:

IntStream.rangeClosed(1, number)
         .parallel()
         .mapToObj(BigInteger::valueOf)         
         .reduce(BigInteger::multiply);

Sometimes the solution is quite simple.

RDP linux to windows

In order to connect from Linux to windows via the RDP protocol please do the following steps.

• install rdesktop

You can build source code from github or you can just find the built one in your OS repository.Source code:
For OpenSuse:

• Connection:

HikariCP pooling

If you are going to crate connections pools you have many options:

• Apache DBCP
• C3P0
• HikariCP
• Vibur

Why shoud I HikariCP? It's just very fast and stable!

Let's create configuration with Spring boot:

And our database configuration property

Our Persisntence Configuration

Ah I know  packagesToScan is ugly. We have Lambdas in Java. Generally this method is craeted to scan components in different modules.

public List <String> packagesToScan() { 
   return scanPackages.stream()
            .map(ScanPersistencePackage::asStrings)
            .flatMap(Collection::stream)
            .collect(Collectors.toList());
}

And concrete persistence configuration for our module:

JPA + Oracle procedures

Due the Oracle's license agreement there is no public repository with the Oracle Driver JAR. So first off all download  oracle  jdbc driver and install to local maven repository (or publish to your artifactory server).


Let's create simple procedures:


Create repository and call procedure using entity manager:

How to make your servers properly secure - Why agent forwarding is dangerous

In this article, I'll show you how to hack server if is bastion host is compromised.  let's talk for about bastion host...

What is the bastion host?
On Wikipedia, you may read that  "bastion host is a special purpose computer on a network specifically designed and configured to withstand attacks."  but it's more important to make correct server configuration otherwise bastion host can be a weapon for an attacker.

How to configure bastion server?
Let's configure Linux servers and then we can start hacking.  Imagine that we have N number of Linux instances and one bastion server. We will use private key ssh authentification for our Linux instances. For testing purpose, we will use user/password authentification for the bastion server. Let's start the configuration.

• Create a user for the server

adduser bastion
passwd bastion

•  open sshd_config and edit configuration. Change PermitRootLogin yes to PermitRootLogin no

vim /etc/ssh/sshd_config

• Restart sshd and check the status

systemctl restart ssh
systemctl status sshd

You should see something like that:

[vq@localhost~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; 

 enabled; vendor preset: enabled)
   Active: active (running) 
 
since Wed 2018-07-04 10:02:06 UTC; 32s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 11595 (sshd)
   CGroup: /system.slice/sshd.service
           └─11595 /usr/sbin/sshd -D
Jul 04 10:02:06 systemd[1]:Starting OpenSSH server daemon...
Jul 04 10:02:06 sshd[11595]: Server listening on 0.0.0.0 port 22
Jul 04 10:02:06 sshd[11595]: Server listening on :: port 22
Jul 04 10:02:06 *** systemd[1]: 
  Started OpenSSH server daemon.

Now you can't log in via the root user. You can see if this configuration checks out.

vq@localhost:~> ssh root@X.X.X.X
root@X.X.X.X's password:
Permission denied, please try again.

•  Let's block user if he/she tries to log in with incorrect user/password. For example we can give him/her three chances to log in - otherwise, we can block permanently or temporarily. So we can use  fail2ban.

sudo yum install epel-release
sudo yum install fail2ban

• Duplicate fail2ban configure to create override config file:

cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
vim jail.local

•  If you use CentOS you need to change backend variable in the configuration. Just open  jail.local file and change auto to systemd.

 [DEFAULT] 
 backend = systemd  

• Now enable SSH daemon jail in the jail.local file (anyway it's necessary for CentOS 7)

[sshd]
enabled = true

• Set bantime, findtime, and maxretry values. For example:

bantime  = 1000
findtime = 1000
maxretry = 3

• Enable fail2ban and restart service

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Now if you try authentification with incorrect user/password you will be blocked.
 



How to configure ssh private key authentification on Linux?

• Crate user in the server

adduser usr
passwd usr

• Generate ssh keys

     ssh-keygen

• Copy public key to the server

    ssh-copy-id -i ~/.ssh/name.pub usr@IP

• Login using private key:

 
    ssh -i name usr@IP

• Disable root login and password authentification to the following file:

 
    vim /etc/ssh/sshd_config
    PasswordAuthentication no
    PermitRootLogin no

• hmm... why not to change SSH port? Let's tell SELinux that we need to change it. Let's use just random number 2597

   
    yum -y install policycoreutils-python
    semanage port -a -t ssh_port_t -p tcp 2597

• Now change port in sshd_config.  

vim /etc/ssh/sshd_config

    Check configuration file

    sshd -t

• Restart services

 
    sudo systemctl restart sshd

• Check what port is LITESENING

 
    sudo netstat -lntp | grep ssh

• Allow ssh login only from bastion. So open 2597 port only for bastion. If you use firewalld run the following command:

 
 firewall-cmd  
 --permanent   
 --zone=public  
 --add-rich-rule='rule family="ipv4" 
 source address="BASTION_IP/32"   
 port protocol="tcp" 
 port="2597" accept'

If this is not your server just check your default zone.

 
firewall-cmd --get-default-zone
 

And check your rules:

  
firewall-cmd --list-all

But maybe you prefer to use iptables instead of firewald:

iptables -I INPUT -p tcp -s BASTION_IP  --dport 2597 -j ACCEPT
iptables -I INPUT -p tcp -s 0.0.0.0/0 --dport 2597 -j DROP

if this is not your own server just check iptable rules and remove some of them if necessary:

iptables -L --line-numbers
sudo iptables -D INPUT {line}

How to access my server via the Bastion? Generally, there are many ways but some of them are not secure. Let's talk for about them.

 
How to use AgentForwarding

• Create config in ~/.ssh/config

    Host bast
        Hostname BASTION_IP
        ForwardAgent yes
        User bastion

• Add your authentification key to ssh agent

ssh-add ~/.ssh/name_rsa

• Connect to bastion host using username/password

    ssh bast

• Connect application server

    ssh app@IP -p PORT

As you see you should connect your remote servers via the bastion server.  The same scenario is if you want to upload/download files:

• Download file from remote server to BASTION. Run the command in Bastion server

    scp -P PORT app@IP:/src_path  /dest_path

• Download file from BASTION to LOCA machine. Run the command in LOCAL machine:

    scp  -P PORT bastion@ip:/src_path /dest_path 

or

   scp  -P PORT bast:/src_path /dest_path

• Upload file to BASTION

scp   /src_path bastion:/dest_path 

You may, well, ask me the question: Is my server secure?  And the answer is quite simple:
- Le't me tell you: NO!
 - Why?
 - Because you are using SSH Agent forwarding?
 - And where is the problem?
 - Because Agent forwarding is dangerous and it's consider considered harmful.

Let's explain everything inside out:  When you connect bastion host your glorious ssh agent is forwarded. It means that the socket will be set up so that someone may use this socket data to access your servers. Imagine that your bastion server is compromised, If someone has sufficient permissions on your Linux server he/she will just use your socket info. As a result, all your server can be accessed. I know window of compromise is very small because it depends on how much time you are connected to the bastion host. But do you really want to the risk when you have other options like ProxyCommand? Hence, just use ProxyCommand!

How to hack servers if you compromised bastion host?

In /tmp directory you may see something like that:

[root@localhost tmp]# ll
total 12
drwx------  2 bastion bastion 4096 Sep  7 17:35 ssh-mKX88v0Vlo

let's open the temporary file

[root@localhost tmp]# cd ssh-mKX88v0Vlo/
[root@localhost ssh-mKX88v0Vlo]# ll
total 0
srwxr-xr-x 1 bastion bastion 0 Sep  7 17:35 agent.10507

Let's see connections to this process id.

netstat -nxp | grep  10507

result:

unix  [ ]   STREAM     CONNECTED     501384   10507/sshd: bastion

and who is connected?

lsof -i -a -p 10507

result:

COMMAND  PID   USER  FD  TYPE DEVICE SIZE/OFF NODE NAME
sshd    10507 bastion  3u  IPv4 501301  0t0  TCP *IP*:ssh->*IP*:8279 (ESTABLISHED)

We can also see socket files:

cd /proc/10507/fd/
ls

result:

lrwx------ 1 root root 64 Sep  7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep  7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep  7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep  7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep  7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep  7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep  7 17:46 9 -> socket:[502080]

And what happens when client will be connected to remote server? let's see:

lrwx------ 1 root root 64 Sep  7 17:46 0 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 1 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 10 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:48 11 -> socket:[502267]
lrwx------ 1 root root 64 Sep  7 17:46 14 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 15 -> /dev/ptmx
lrwx------ 1 root root 64 Sep  7 17:46 2 -> /dev/null
lrwx------ 1 root root 64 Sep  7 17:46 3 -> socket:[501994]
lrwx------ 1 root root 64 Sep  7 17:46 4 -> socket:[502069]
lrwx------ 1 root root 64 Sep  7 17:46 5 -> socket:[502072]
l-wx------ 1 root root 64 Sep  7 17:46 6 -> /run/systemd/sessions/1836.ref
lr-x------ 1 root root 64 Sep  7 17:46 7 -> pipe:[502079]
l-wx------ 1 root root 64 Sep  7 17:46 8 -> pipe:[502079]
lrwx------ 1 root root 64 Sep  7 17:46 9 -> socket:[502080]

We can even see if socket file is used using netstat:

unix  3 [ ]  STREAM  CONNECTED  502267  10561/sshd: 
                     bastion  /tmp/ssh-oVoMXC6vb8/agent.10561
unix  3  [ ] STREAM     CONNECTED     502072   10561/sshd:  bastion 

but wait we need to steal the socket information while the session of bastion host is open. Oh, we also need destination server IP, so just use netstat It's funny but you might find something like that on the internet.:

netstat -tn 2>/dev/null 
grep :22  | awk '{print $5}' | cut -d: -f1 | 
sort | uniq -c | sort -nr | head

Sometimes people try to demonstrate their knowledge  but just make everything simple.

netstat -tn

The final step to use the forwarded socket file

eval "$(ssh-agent -s)"
SSH_AUTH_SOCK=/tmp/ssh-EAKxOdL4fl/agent.10507

Check if the key is loaded.

ssh-add -l

result should be something like that:

2048 SHA256:2Psdl..B5KQ /home/usr/.ssh/name_rsa (RSA)

Ok. We have seen that Agent Forwarding is a problem. But where is the solution? Fortunately, we can just use a proxy command. So let's rewrite our configuration:

Host app
    Hostname *.*.*.*
    IdentityFile ~/.ssh/your_rsa
    User *******
    Port ****
    ProxyCommand ssh -W %h:%p bast

Host bast
     Hostname *.*.*.*
     ForwardAgent no
     User ******

Now  it's very simple to connect your server

ssh app

Do you want to download something from bastion?

scp bast:/home/bastion/sh ~/

Maybe you want to download something from the remote server via the bastion tunnel:

scp -P **** app:/src_path  /dest_path

Maybe you want to ypload something to remote server via the bastion tunel:

scp -P **** app:/src_path  /dest_path

Conclusion

• If you use bastion host, don't use AgentForwarding but use ProxyCommand
• Always use non root user for authentification
• Use firewall and block all unnecessary connections.
• Use SELinux
• Block the IP address who  tries to login  several time with incorrect credentials
• If it's not necessery don't give sudo permission to the user
• Monitor your server
• Update your server for security patches

How to create your own system.d service

If you explore latest Red Hat linux, you will see that the  traditional init scripts are gone. It's much easier to use rather then SysVinit, it reduces system startup time because the processes are running in parallel. It has also very nice logging mechanism. Let's write simple examples for CentOS7

Create a bash file:

 touch /usr/local/sbin/{launcher_name}.sh

It's a simple bash file that launches java application.

#!/bin/bash
java -jar /root/{name}.jar

Add execution permission

chmod +x {launcher_name}.sh

Create service:

touch /etc/systemd/system/{service_name}.service

Configuration:

[Unit]
Description=Application launcher service

[Service]
Type=simple
ExecStart=/usr/local/sbin/{launcher_name}.sh
TimeoutStartSec=0

[Install]
WantedBy=default.target

Set permissions:

chmod 664  {service_name}.service

Reload daemon

 systemctl daemon-reload 

Usage:

systemctl enable {service_name}
systemctl start {service_name}

We can create the same using init instead of sytstem.d. We shiuld just create configuration to the following locataion: /etc/init.d

#! /bin/sh
# /etc/init.d/{service_name}
### BEGIN INIT INFO
# Provides:           {service_name}
# Default-Start:      2 3 4 5
# Default-Stop:       0 1 6
# Short-Description:  Example of init service.
### END INIT INFO
# Actions provided to make it LSB-compliant
case "$1" in
  start)
    export DISPLAY=:0
    sh {launcher_name}.sh
    ;;
  stop)
    echo "Stopping"
    ...
    ;;
  restart)
    echo "Restarting"
    ;;
  force-reload)
    echo "Reloading"
    ...
    ;;
  status)
    echo "Status"
    ...
    ;;
  *)
    echo "Usage: /etc/init.d/{service_name} 
        {start|stop|restart|force-reload|status}"
    exit 1
    ;;
esac
exit 0