Managing Secrets with Hashicorp Vault

It is hard to overvalue the importance of secret management in various organizations and tech companies. The complexity of managing company operations grows along with the company itself. These can be different: from managing people to managing machines. Regardless of the use case, you need a solution to share credentials between the trusted parties in a secure way. Thus, scaling the company includes scaling the secrets management solution.

top-secret

Nowadays modern applications rely on the microservices pattern and cloud technologies. Building a complicated scalable service is quite easier today than it was 10 years ago. But security risks also grow along with application complexity.

According to Gartner, worldwide spending on information security products and services is more than $114 billion in 2018. This is a 12.4% increase from last year. And the market forecast for 2019 is to grow about 8.7% to $124 billion.

Defining goals

Choosing the right pattern for the secrets management system would reduce the risk and complexity as well as the total cost. As we mentioned above, there are two main challenges to solve:

  • Sharing credentials between people;
  • Sharing credentials between machines;

These involve the third challenge:

  • Flexible identity and access management both for people and machines;

So, the resulting solution must be able to be comfortably used by non-tech people, developers, and applications. And all of these requirements are greatly fulfilled in a Hashicorp product - Vault.

Flexible identity management integrations

One of the most powerful features that Vault comes with is a rich set of authentication methods, e.g.:

  • AWS
  • GCE
  • Azure
  • Kubernetes
  • Github
  • Token
  • Username & Password
  • etc

The full list of auth methods can be found here. This differs Vault from any proprietary solution. You can leverage your existing environment to authenticate in Vault and obtain secrets. It is necessary to note, that you can extend Vault by developing your custom method.

Moreover, you're not restricted to use only one authentication method. For example, you can combine AWS and LDAP methods in a hybrid cloud. Or you even can use both GCE and Azure authentication methods to access the same data in a multi-cloud environment.

Sharing secrets across applications

Cloud-Compatible

Let's consider a quick example. We'll show how the process would look like if we integrate Vault AWS auth method with the Rails application. The app is deployed on AWS EC2 instance. The challenge is to pass the PostgreSQL credentials into our Rails application securely. For this example, we'll show a high-level description of how to do it:

vault-aws-ec2-auth-method

  1. The initial bash script is evaluated before the Rails application. It fetches the instance's metadata and retrieves a specific instance signature.
  2. The script makes an API request to the Vault server with the actual signature from the metadata.
  3. Vault takes that signature and verifies it using the AWS API.
  4. If verification succeeded - Vault will return an API token to the bash script.
  5. The script can now start the Rails application with the Vault token set in the environment. Upon start, Rails app can use this token to query Vault for Postgres credentials.

This was a brief overview of the EC2 auth method. The AWS auth method also supports IAM integration and we'll cover this in the next articles.

Cloud-Agnostic

And what about bare metal? Or any virtualized or containerized infrastructure? To authenticate the application in such case you can also use the AppRole method. The above diagram will be modified to do the following:

vault-approle-chef

  1. The admin user creates a new VM using a specific image. Admin passes the Role ID into the VM environment.
  2. Upon start, the VM triggers chef-client run. The chef-client runs the recipe and pulls a special token from Chef's Data Bag. 
  3. Chef-client uses the pulled token from the previous step to retrieve the Secret ID from Vault. After the Secret ID is retrieved, the chef-client performs Vault authentication using the Role ID and Secret ID. The app token will be returned if authentication succeeds.
  4. After the authentication is succeeded the chef-client passes the app token into the Rails app.
  5. Upon start, the Rails app queries Vault to get the needed secrets.

Sharing secrets between people

Vault is a secrets management software that is quite popular by developers. Despite its popularity, it is a highly undervalued solution. This looks like conflicting statements from the developer's point of view. But the undervalued side of Hashicorp Vault is hidden in its potential collaborative usage by non-tech people.
With the release of Vault 0.10, the UI was introduced. Now you don't need to be a developer to utilize this tool.

vault-ui

Some examples for non-tech collaboration:

  • Storing secrets for various web services.
  • Sharing secrets between employees.

Also, Vault has a rich API and you can build various applications on top of it. In collaboration with developers, you can also create a custom solution based on your needs.

Secured Infrastructure Access

As we mentioned above, you can build a custom solution using the Vault API. One of the most common cases - Vault one-time SSH passwords. You can configure your target server to check SSH passwords using the Vault API. To achieve this, you'd probably need to build a specific PAM module. This is not a trivial task and would require some distinct knowledge. But fortunately, there is already an official integration from Hashicorp called vault-ssh-helper. Here is a high-level overview of the infrastructure:

vault-ssh-one-time-password

  1. Vault SSH one-time password loginAn authenticated user makes a request to get a one-time password.
  2. If everything is ok and the user has an appropriate policy, Vault returns the one-time SSH password.
  3. A user establishes an SSH connection to the target server providing the one-time password.
  4. The SSH client queries vault-ssh-helper executable with the provided one-time password.
  5. The vault-ssh-helper queries Vault server to check the provided one-time password.
  6. Vault validates the password. After this operation the password becomes invalid.
  7. After the successful validation from the Vault server, the vault-ssh-helper permits the user to perform a login on the target server.

The vault-ssh-helper tool is a great example of how you can make your existing system even more secure using the Hashicorp Vault API.

Auditing

Nowadays all production-ready applications produce some logs about themselves. Just imagine how hard the work would be if you had no access to the application logs. This becomes even more critical when we're considering the centralized secret management system. 
Vault produces detailed logs about all events that are happened with it. You can analyze how Vault was queried at any given period of time. The information is stored in the JSON format, and you can use any log processing tool to work with it.

Vault produces detailed logs about all events that are happened with it. You can analyze how Vault was queried at any given period of time. The information is stored in the JSON format, and you can use any log processing tool to work with it.

The hidden potential

Hashicorp has made an amazing and hard work for the last 3 years by developing Vault. As a result, they created not only a unique secret management solution. They created a highly extensible core that can be used to develop new applications or enhance the security of existing tools. Vault's open source community is growing fast, and we will see a bunch of new Vault based applications in the near future.

Rockos Vault solution

It takes minutes to start using Vault in development mode. But it can take days to implement a production-ready Vault solution. At Rockos we help teams to reduce the operational overhead, minimize costs and save time by providing a battle-tested, secure and fully-managed Vault service.