AppSec Role for Secrets Management

86. Designing an architecture that requires a three-party collusion to access secrets in Secrets Manager

This is a continuation of my series of posts on Automating Cybersecurity Metrics.

In the last post we created an AppSec role.

Creating an AppSec Group to Administer Secrets Manager Secrets

I started refactoring the code to set up users with their own secrets. Here’s where I hit a bit of a snag.

Let’s say an IAM administrator wants to create a new user with an SSH key stored in a user-specific secret.

• We can’t create the SSH key until we have a secret to put it in if someone else is creating the secret.
• The team responsible for managing the secrets resource policy needs the user name before they can create the user-specific secret.
• The KMS user, role and group has to be created before they can deploy a KMS key.
• The Secrets Management user, group and role (what I am calling AppSec here) has to be created before they can create a secret.
• The KMS key needs and user needs to be created before the secret can be created with an appropriate policy.
• We do not want to put a value in the secret until it has a policy (a problem with our prior scripts).

The current deploy script has the creation of users all in one script. We either need to separate out KMS users from the deploy.sh script or separate out the creation of SSH keys into its own script, run after all the users and KMS script is created, and after the secret is deployed where we can store the SSH key. The later seems easier to manage.

Separate deploy script for SSH Keys

In my IAM directory, I’m going to create a separate deploy_ssh_keys.sh script and I’m directly going to call the create_ssh_key function directly from that script for the Developer user.

I’m going to remove the y/n parameter that I didn’t really like in the first place from the user creation code in my primary deploy.sh script. So this:

Becomes this:

We’ll remove the logic from user creation to optionally create the SSH key:

At this point I gave my user deploy.sh script a quick test to make sure I didn’t have any typos (and I did so I fixed it).

I also test my ssh key creation script to makes sure I’m starting with known-good code and it works.

Create an AppSec subfolder and secrets management code

Next we need to create the AppSec directory in our code base matching the other subfolder hierarchies. We’ll create our standard deploy.sh file and an appsec_functions.sh file just like in other earlier posts for reusable functions.

We can add a create_secret() function to create a new secret in secrets manager. Since a secret must have a value by default we’ll set a temp value that can be overridden when a secret value is added. As noted we don’t want to pass in sensitive values to CloudFormation parameters anyway as they would be visible in the AWS console.

We will pull over the CloudFormation templates from the IAM directory to create the secret and the secret policy into the cfn folder in the AppSec directory so the paths in the above code are pointing to the correct files.

I deleted all the secrets related stacks to ensure my code would not simply skip a deployment because the template hasn’t changed. I also don’t want a stack to get created with a new name and leave an old stack hanging around somehow.

Make sure you have set up a CLI profile named “AppSec” using the AppSec role and user credentials described in the last post.

It is at this point that I realized I forgot to add permission to use CloudFormation to the AppSec role in the last post so I added and deployed that change.

Ah, yes, and here is where we get tripped up once again.

Remember how we were trying to ensure that the person that creates the secret policy doesn’t have access to delete the secret? Well, we can’t create a secret without a value, so we have to encrypt the value to add the secret.

Seems as though our plans haven been foiled. Until AWS lets us create a secret and a policy we cannot do what we wanted to do with this secret and policy to create segregation of duties and non-repudiation.

Let’s say we create a secret unencrypted. Then we wouldn’t need the KMS key right? Then could we update the key to add the KMS ID later? But who would enforce and ensure that the secret was updated with a KMS ID? It’s all getting very messy at this point.

The best solution would be for AWS to allow creation of a secret with a KMS key ID but not requirement to add or encrypt a value.

The other part of this solution is for AWS to fix KMS key policies and IAM Permissions so a user can be provided permissions to EITHER encrypt or decrypt but not both.

Well, that was fun. We just spent a lot of time for nothing as we haven’t really achieved our objective of non-repudiation right? The AppSec user needs permission to encrypt and decrypt the value and they need permission to update the secret policy. So what’s stopping them from updating the policy so they can get the value and decrypt it?

Remember that we denied the “get-secret-value” permission in the AppSec IAM policy. The AppSec user knows the initial value they set (a dummy value) but cannot get the subsequent updated value due to IAM restrictions.

So a more accurate diagram of our segregation of duties from our earlier post would look more like this to be accurate if we consider who has permission to change a policy to give themselves access to the encrypted secret, with the developers being granted access by three different administrators for them to get access to their own credentials (ssh key):

In other words, it would take a three-party collusion for someone to get access to a secret they shouldn’t access if we ensure IAM administrators can’t give themselves permissions they should not have. As mentioned before there is always a set of super admin credentials that can change all of this that should be used for initial setup and then locked away.

So let’s go ahead and give our AppSec user the permission to encrypt and decrypt the same way we did for our IAM user in both the KMS policy and their IAM role policy.

Updating the role policy is simple. Copy the permissions from the IAM Admin Role:

Redeploy the roles to update the policy.

Passing in multiple ARNs to encrypt and decrypt with our KMS key

But wait — how do we add two ARNs to allow encryption and decryption? We can pass in a list of ARNs into the key policy for the IAM and AppSec roles the same way we passed in a list of users in the Developer group.

Looks like we already are accepting a comma separated list as a parameter:

Get our stack name and output name to use with our common function to get the output value from a CloudFormation stack:

Get the ARN for the AppSec role, create a comma-separated list, and pass that in to create our key policy.

Redeploy the keys to update the DeveloperResources key policy.

Well, you may or may not have noticed but I misspelled encryptarn2 above so I had to fix that. The code in GitHub should work. :-)

Check the key policy to make sure that the ARN for both roles has the necessary permissions.

Now we can try to deploy the secret again because our AppSec user should have access to deploy the secret.

Success.

IAM Admin updates the secret with the SSH key

Now that the secret exists, return to our ssh key creation script and run it from the IAM directory.

./deploy_ssh_keys.sh

I had to fix a few typos:

Success:

Now test that the developer can still get their secret value (see prior posts):

aws secretsmanager get-secret-value --secret-id Developer --profile developeruser

Again — success.

A few remaining issues

There are a few remaining issues to consider with our code above. Do you know what they are? What if our key creation fails in the middle of a deployment? A new SSH key might be created by the secret still has the old SSH key in it. The key in the secret might be created but the user policy fails to update and points to no secret or an incorrect secret if one was deleted and updated.

This may or may not create a security problem because for the most part, pointing to something that got deleted and the fact that you cannot create a specific secret with a name you control means someone can indirectly point someone to the wrong secret. There may be some tricky way to do it if there’s an existing useful secret at some point but I’m not going to thing through that further right now. In my case the risk seems extremely low.

But more airtight code would ensure that if a set of operations that are supposed to complete together fails, that the incomplete operations are rolled back or some other operation occurs to eliminate potentially invalid pointers in policies. For example, if anything fails you could delete the secret, secret policy, and user policy. We won’t take those steps here but consider the threat model in your own environment and if want to take those additional steps.

I actually plan to completely change deployments in the future if my plan works. I’m going to ditch all the bash for the most part and use the CloudFormation templates, so I’m not going to address this right now. Bash is not an ideal programming language in any sense so this is all proof of concept code at the moment.

Now I want to use this key for an EC2 instance…stay tuned…

Teri Radichel

If you liked this story please clap and follow:

Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research

© 2nd Sight Lab 2022

All the posts in this series:

Automating Cybersecurity Metrics (ACM)

____________________________________________

Author:

Cybersecurity for Executives in the Age of Cloud on Amazon

Need Cloud Security Training? 2nd Sight Lab Cloud Security Training

Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.

Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.

Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts

---

AppSec Role for Secrets Management was originally published in Cloud Security on Medium, where people are continuing the conversation by highlighting and responding to this story.