Get best of the week

VPN killer. Proper remote access to battle servers

The opinion expressed in this article is the personal opinion of the author. He emphasizes that it may not coincide with the opinion of his employer, superiors and the security department.

One of the coolest things in our company in terms of infrastructure is how we implement remote access. This is just super-mega-protected magic. I spoke with many of my fellow security guards and auditors, and it seems we accidentally invented a whole new story by mixing commercial solutions and our software. And so, I thought, it would be interesting to dive deep into the technical details of how the industry works with outdated remote access solutions and how we implement it.

Good person VPN will not advise


I'll start with a strong statement: all VPNs are trash .

VPN, like other digital technologies, can be configured so that if it is hacked, then the world will not end. However, no one does this ... but theoretically they can!

In 99.95% of cases, the VPN is configured to:

  1. Creating a network bridge between the device (laptop or other server)
  2. ... and a larger network of servers (for example, in the cloud or for on-prem )
  3. ... and the Internet, protected by an additional layer of encryption



This is not a good idea. Imagine if a virus has settled on your laptop and you are connecting via VPN to a network of battle servers? Ta Dam! The virus now has LAN access to your production. What's on the exhaust? Sadness. A lot of sadness.

Okay, let's say the virus case is far-fetched. But what about the fact that a hacker can infiltrate the VPN itself, for example, using a vulnerability inside the device or software on which the VPN works? So, having made your way straight into the target network? This is a serious matter, and far from theorizing. You can read more in the article on how Heartbleed was used to capture VPN using the attack vector, which I warned about directly in my blog .

Not so long ago, a wave of VPN vulnerabilities swept around the world, which were immediately used by cybercriminals to gain access to target networks. But this is not surprising, right? These systems "look" directly into the Network, without any other protective mechanisms in front of them. Fixes and patches are usually not applied automatically, and involve proprietary mechanisms controlled by proprietary software running on proprietary OS. Well, good luck there.

Is it hard to find these VPN devices? Before writing this article, I never tried, so I didn’t know for sure. About half an hour in Shodan.io and please - a few major results from the output:

  • Thomson Reuters is a $ 41 billion company with 26,000 employees, half of which comes from financial services.
  • SAP Concur is a travel management service, breaking it would give a bunch of personal and payment data of all kinds.
  • Progressive Insurance - personal data and personal medical information, coupled with some payment data.
  • Chevron Phillips Chemical - the name speaks for itself.

Well, this is probably not very good. If these things are so easy to find, it doesn't seem like the perfect solution to put them online. Do we have any other choice?

Zero trust


Zero Trust [literally: “zero trust”] - in essence, it means that you authorize each connection, and do not assume that some of them are trustworthy because they are already inside your network. If you want to better understand this term and rethink your approach to business, read this article on Network World (sorry for another shameless self-PR).

We bought the Okta Advanced Server Access (OASA) solution to introduce Zero-Trust authorization on battle servers.
OASA is cool for three reasons:

1. This is just a wrapper around the configuration of OpenSSH on steroids


Under the hood, the OASA platform is a well-managed deployment of OpenSSH (the same ssh command in your console). And OpenSSH is an extremely well-tested and secure remote administration solution. By the way, since 2003, OpenSSH has not had a single vulnerability that could lead to unauthorized remote access in the standard configuration.
The entry points in this scheme are regular Amazon Linux 2 EC2 instances. Thus, the attack area is extremely small. Remember: one of the main problems with using a VPN is the proprietary software / OS configurations that prevent automatic patches. The ability to update our network entry points along with the rest of the infrastructure is a big win.

2. There are no network bridges


As I mentioned above, most VPNs are configured to create a network bridge between a network device, such as a laptop, and a network of servers on the Internet. One of my most painful spots in VPNs is that they intercept all your network traffic. You can, of course, reconfigure to a different behavior, but our clients and NIST 800-53 SC-7 (7) security protocols usually require just that.

This is a good example of how some security measures are far behind industry development. In the old-school world, VPN could be the only layer for encrypting traffic. Auditors at times believe that without VPN protection you will send your data through unencrypted channels.

Fortunately, there is a better way. In the OASA model, a connection is established individually between you and the server. For example, in response to the request “I want to log on to EC2 instance i-028d62efa6f0b36b5”, your system connects to the input point of the network and then to the target server. OASA also protects these connections by issuing ten-minute client certificates after the first verification of your data through SSO , and then verifying that you are on the intended (and confirmed) trusted device.

There is little freedom to roam the net. An administrator can enter a network entry point and forward ports to another destination, but this feature is disabled by default and must be explicitly requested when establishing a connection. And the best part is that no one requires me to carry all the traffic through the Virtual Private Cloud, because this is not called a “VPN”.

3. Dedicated network access and random IP


These network entry points are deployed separately for each Virtual Private Cloud (for example, one for prod, one for staging, one for dev, etc.). In addition, each of them is closely monitored by our security solution, which logs all activity and filters traffic. If the cracker is inside one of the entry points, something will not work out either. In any situation, our security scheme does not allow access to protected resources simply because you are already inside the Virtual Private Cloud.

One of my favorite defense mechanisms showed up by accident. When we first started configuring the entry points, each was configured for a static IP address from AWS. Quite quickly, we found that these IP addresses were sometimes not tied to EC2 instances at the right time, which could lead to incorrect OASA self-configuration. Having experienced the delights of a dozen different fixes in production, I could not stand it and simply deleted the whole story about static IPs - and it worked.



OASA just needs an Internet-facing IP; it does not have to be known in advance. When the client is ready to establish a connection, a unique GUID is requested, with the help of which the IP is determined:

  • User: I want to log in to connect to vpc-99f2acff
  • OASA Client Application: I have defined vpc-99f2acff as a known server with GUID 25af5d4f-e657-4583-b0bd-beb1ca4f0c1f
  • OASA server: K 25af5d4f-e657-4583-b0bd-beb1ca4f0c1f can be accessed at 3.22.198.24, here are the certificates
  • OASA client application: Certificates installed, connecting to 3.22.198.24 via SSH ...

This means that each instance of our entry point infrastructure (a separate post about it to your attention) comes with a completely new set of IP addresses. Thus, random hackers will have to look for one IP among tens of millions, and their number is growing every day. Unfortunately for them, this search is completely in vain, thanks to ...

Enterprise port knocking


Port knocking [literally: knocking on ports] is a thing that, in principle, no one uses in the real world, but which is very fun to configure. In short, port knocking is a sequence of “bumps” to various closed network ports, and if the sequence is repeated correctly, the “real” port opens to your IP. Elegant, but impractical in real large projects.
I was inspired by the idea and thought how to improve the idea. So the solution that I call Enterprise Port Knocking was born .

I wanted to create a mechanism that would guarantee the closeness of our entry points to the Internet, until someone was able to access them. This mechanism was supposed to be easy to use, reliable, and authenticated in an existing way.
I sketched the architecture of the mechanism and went with her to our incredibly talented team of engineers. A few weeks later we started.

The service is quite simple and deploys as an AWS Lambda function, with access to it through the AWS API Gateway (the joy of serverless architecture!) For easy and reliable use. The mechanism works like this:

  1. The user is successfully authenticated through SSO.
  2. The application bypasses configured AWS accounts in search of a specially marked Security Group (AWS concept for firewall rules)
  3. The application updates the Security Group, allowing access to the IP address of the requestor. A rule in the Security Group also stores the add timestamp.
  4. The cron task clears the list of allowed IPs every specified period of time.

Thanks to this situation, we are now boasting a remote access solution that is completely closed to the Internet and requires two-factor authentication through our user database before opening the port in the firewall .

And it's easy!


I have not yet touched on how simple these mechanisms are in operation. I know that there are many components, but together they create a very simple authorization:

  1. Log in through SSO.
  2. Click on the launch of Enterprise Port Knocking in the SSO portal.
  3. In the terminal, using the SSH command, specify the destination as the ID of the target EC2 instance. The OASA system is smart enough to figure out which entry point to use and everything else happens automatically!



This whole system is a great victory for our infrastructure, for our compliance with all safety agreements and for the safety of our customers. Users really like how easy it is to connect to our servers, because you do not need to go through additional authentication or remember which VPN to use. And I really like my calm sleep. Everyone won from the new model!

Well, except for the hackers, of course.

More articles:


All Articles