October 31, 2018

Planning Ahead

Managing SSH keys and server access is one of the most important first steps an organization can take to improve its security. Here at Foxpass help you centrally manage server access (and much more) through our networked identity service that connects to your directory to your cloud infrastructure. As is true for any infrastructure setup, the decision to use a cloud identity service like Foxpass comes with benefits and drawbacks.

When it’s working well, a networked identity service allows you to streamline access control improving both ease of use and security. The directory acts as a single source of truth eliminating any gaps in your authentication mechanisms. However, the biggest drawback to this solution is downtime. When your directory goes down, so does the access to all the systems you’ve integrated it with. In that case, having a single source of truth becomes a liability as you’re forced to choose between keeping your system secure and waiting for service to be restored or keeping it useable and switching to a less secure method of authentication. Fortunately, there are ways to mitigate the damage of downtime and maintain security and usability simultaneously.

Here’s some steps you can take to maintain access to your infrastructure in any scenario:


A good generalized failsafe to maintaining Linux host access in the event of an outage is to have a local sudo user on all your hosts. Use a configuration management tool (i.e. Puppet, Chef, Ansible, etc.) to manage the admins on the hosts. Then, store the password protected SSH key for that admin in a vault (i.e. KMS, 1Password, etc.). Ideally, have some sort of audit controls on that key access so you can see who retrieved it and when.

Additionally, Foxpass offers a local cache that you can run on a separate server. The cache syncs with our main database periodically. In the event of any downtime, your servers will use the local cache to maintain uninterrupted service.


In the event that you can’t contact our RADIUS endpoint, have an SSID configured with WPA2 (shared password) ready to enable. If you’re using a Mobile Device Management (MDM) solution where you can remotely configure the machines, you can store the network password automatically without any end-user involvement.

We’re working on adding RADIUS support to our local cache, as well. Contact us at help@foxpass.com to learn more.


Right now the only solution to keep a VPN functioning in the event of a service disruption to the directory is to have a backup directory or other system running as a second authentication method. As your VPN is one of your most important security tools, it’s worth considering how much protection you’re willing to sacrifice in order to make it more useable.

Putting it all together:
An overlooked aspect of these backup measures is testing. If you’re not prepared for an outage, then you could be delaying your system’s recovery substantially. You should set up a recurring task every 3-4 months to make sure that your backup systems are still functioning correctly.

If you’re using the Foxpass cache, you can check the ‘Cache’ page on the console to see the last time sync ran and if it succeeded. You can also point a host directly to your cache (bypassing the main Foxpass endpoints) to double check the authentication mechanism is working.

At the end of the day, there will always be a tradeoff between usability and security. While a networked directory can increase make your systems easier to access and more secure, it exposes your systems to an extra layer of downtime. It’s important to have contingency plans to keep your infrastructure ready for any event.

Stay safe!
-The Foxpass Team

Old post:

[Our uptime is impressive, but occasionally we do have issues, as do our service providers (or your service providers). Etc etc]

[Using Foxpass (or any other network identity service) to manage your users SSH keys and access to machines is a huge value proposition that enables administrators to streamline and better control access to various resources in your environment. The problem comes in when there is some external scenario that causes your servers to not be able to communicate with Foxpass (or other network identity service) or vice-versa and access authentication is no longer available. Here’s a guide to making sure all your systems maintain access in the event of an issue with your authentication server:]

[Our minimum recommendation is to create a local user on all systems that has full sudo access and is accessed via an SSH key that is password protected. And that the private key and password for this are stored “out of band” such that it is only available in emergency situations. A common way to implement this is to use a configuration management tool such as Puppet/Chef/Ansible/CloudFormation/Config/etc to create the user locally on each instance with the public side of the “out-of-band” SSH-key. Then store the private key (and password for it) in some kind of key vault, KMS, 1Password, or a safety deposit box at the bank, to manage access. Ideally there would be audit controls on accessing the private key in addition to logging of usage of it to make sure that it was only used in emergency or testing situations.]

[Have another SSID ready-to-go with just WPA2 (shared password) authentication that can be enabled during an incident, and disabled afterwards. If you use an MDM, this password can be stored in each machine without the users having knowledge of what the password is.]

[Which brings us to testing. Having this system is all well and good, but if you never test it the failure opportunity when needed is rather high. Once a quarter you should extract the key from its vault and spot check various hosts in your infrastructure to ensure that you are able to access your resources when the unexpected happens.]

  • LinkedIn
  • Tumblr
  • Reddit
  • Google+
  • Pinterest
  • Pocket
Comments powered by Disqus