🔒Security Overview

Philosophy

Limit the blast radius of a compromise:
  • A compromised customer account should not allow access to CloudTruth or other customer's data
  • A compromised CloudTruth account should not allow access to customer secrets
  • No CloudTruth service shall ever have direct public network access. Access to any service must be via load balancer.
  • Secrets must be encrypted at rest and in transit
  • Limit the amount of code that has access to cleartext secrets. This includes third-party libraries.
  • CloudTruth employees should not have access to customer secrets in plaintext without customer approval. All secret access must be audited.

Infrastructure

Every organization within CloudTruth gets their own dedicated, isolated Hashicorp Vault instance. Because of this isolation, no other organizations will have access to this vault or to the secrets stored in the vault. It also allows us to significantly reduce the fault domain to specific customers.
The rest of our infrastructure is multitenant. Our infrastructure runs in AWS, spread across 3 availability zones in the us-east-2 region. Our service is split into 3 ECS clusters, with strict security group configurations on each of them. The 3 clusters:
  1. 1.
    The main app: api, docs, and web service
  2. 2.
    Internal vault: CloudTruth's main vault, used for CloudTruth owned secrets
  3. 3.
    Organization vaults: the vault instances for each organization
All ECS clusters are in a private VPC with the only public access being from the Application Load Balancers to the API, docs, and Web App endpoints. We also have a bastion host for admin access.

User Data

Non-secret data

Non-secret parameters and other non-secret data is stored in a multitenant postgres database hosted by AWS RDS. All queries are scoped by the organization id of the user performing the operation. The scoping is performed on the backend and is not dependent on anything provided by the user. The organization id is fetched from the user's JWT, which is created and signed by Auth0.

Secrets

Secrets are stored in the organization vault dedicated to the user's organization. Vault will immediately reject any request if the organization id in the JWT does not match the organization id of vault. Our secrets management will be covered in more detail in a later post.

Authentication

We use Auth0 as our Authentication provider. They allow users to authenticate with a username/password or with various social identity providers, such as Google, Microsoft, GitHub, etc. It also means we never have access to your user passwords - that is only ever known by Auth0 (when using username/password auth) or your identity provider (Google, Microsoft, GitHub, etc).
When a user logs into our system (via Auth0), we get a JSON Web Token (JWT) in return with the user's id and their organization membership. All operations in the system require the JWT and are restricted to the organization contained in the JWT, including organization vaults. The only exceptions are access to documentation (no auth required) and API-key authentication.

API keys

API Keys provide a convenient mechanism for authenticating automation systems, such as scripts and CI/CD pipelines. An API key can be created by any user with owner or admin permissions. The user cannot create an API key with permissions greater than their own permissions, but they can reduce the access of the API key. Behind the scenes, the system will exchange the API key for a JWT from Auth0. The JWT is then used in all other operations. We'll cover more about how this exchange happens in another post.

Authorization

We have 2 different authorization systems:
  1. 1.
    Org membership, determined by the JWT
  2. 2.
    Role-based access controls

Org membership

The org membership authorization is performed early in the lifecycle of the request. You'll notice that our APIs don't ask for an organization id in any calls - that's because the organization id is part of the JWT. This makes it impossible to query for things outside of your currently active org. This carries all the way down to organization vaults. Org vaults use the same JWT as the rest of the system for authentication and will immediately reject a call containing a JWT not bound to the org the vault belongs to.

Role-based Access Controls

Our current RBAC implementation is relatively simple. The simplistic nature of the design allowed us to focus on the the component that handles permissions checks. We have implemented 4 roles as described below:
  1. 1.
    Owner - the org owner(s). Full permissions in the account.
  2. 2.
    Admin - Full permissions to the account except for account management (subscription changes, org deletion, etc).
  3. 3.
    Contributor - Full permissions on projects, environments, parameters, secrets. Cannot perform system operations like account management, user management, add/remove integrations, or view audit logs.
  4. 4.
    Viewer - read-only access to parameters.

Audit

All operations are logged with the user that performed the operation and the time it occurred. Some operations have more detail logged, such as push integration will log if it was successful or not. This log is immutable and records all operations regardless of source (UI, CLI, or API). The audit log is available to owners and admins for viewing.

Encryption

We take advantage of encryption services provided by AWS on all products that support it. For example, all EBS volumes are encrypted and all S3 buckets are encrypted. In addition, we also maintain a chain of certs and keys used for TLS encryption. The certs and keys are rotated frequently by our internal Vault service and rolled out automatically.

Network encryption

All network communications are encrypted with TLS v1.2 or later. All certificates are verified. This includes all internal service-to-service communications. There are no exceptions.

Data at rest

All customer data is encrypted at rest. This includes our main postgres database, all assets stored in S3, and all node-local storage to EBS volumes. Org vault data is encrypted by Vault and encrypted again by the S3 bucket it is stored in.

Availability

All of our multitenant infrastructure runs in the us-east-2 region, is redundant across 3 availability zones, and is configured for autoscaling via Spot by NetApp. Org vaults are the only non-redundant services we run. Each org vault is a single instance supervised by ECS. Should a vault go down for any reason it will be brought up on a new node in approximately 90 seconds. We plan on offering highly-available org vault instances for additional cost in the future.

Backups

We take regular, automated snapshots of our database and additional snapshots preceding any admin operation. All org vaults store their data in a versioned S3 bucket.

Code security

Our repositories require SSO to our corporate identity provider in order to get any access to our internal repositories. All of our code is scanned for viruses and vulnerabilities on every build - it is baked into our build automation. We also use multiple static code analysis tools to ensure we are adhering to best practices. We measure code coverage on all components and maintain a high level of coverage. We use dependabot to keep all of our dependencies up to date.

Admin access

We have no "God" accounts, but we do have admins with privileged access to various systems. Access is achieved via a bastion host which is limited to a whitelist of IP addresses of our admins. We also require MFA for admins. This includes running any out-of-cycle deploy code.
Last modified 2mo ago