Secrets Management with Puppet, Hiera and AWS KMS

Using Hiera to separate organisation configuration data from Puppet code is a well known best practice that allows you to write reusable modules. Usually configuration data involves sensitive materials such as passwords, private keys or personal details. Fortunately, the Hiera ecosystem includes projects that integrates AWS KMS. This post highlights how we used this powerful combination to manage secrets using Puppet.

This post assumes the Puppet infrastructure and a Puppet Master is operational. Note that in all the Ruby gem install commands, beware if you need to use http proxy settings to access the internet.

hiera-eyaml and hiera-eyaml-kms

The hiera-eyaml backend to Hiera is a flexible Hiera data store allowing for per-value encryption of these secrets within yaml files. The great thing about the open source project is its plugin architecture allowing you to select your encryption scheme.

Along comes the hiera-eyaml-kms open source project. This Ruby gem is a plugin to hiera-eyaml that integrates with AWS KMS. The benefits are that you no longer need to manage the private keys for encrypting all your secrets thereby minimising unauthorised exposure of keys. The master keys are now stored in AWS KMS.

AWS KMS and Envelope Encryption

A limitation of the plugin is that envelope encryption is not used to encrypt data. This issue confirms the point

Envelope encryption means that the master key is used to generate unique data keys, which are then used to encrypt your data. Besides missing out on another security defense layer, it means the plugin can only encrypt and decrypt data less than 4096 bytes. This is a limitiation of AWS KMS to encourage use of envelope encryption to protect application secrets of abitrary size.

It's probably safe to guess that this was a concenscious design decision by the project authors. Envelope encryption requires extra functionality to perform the crpto operations to encrypt and decrypt the data keys.

Also, envelope encryption requires the encrypted data key to be stored next the data ciphertext within the 'envelope'. This increases the size of the encrypted value that is stored.

A workaround may be to add extra functionality to enable only storing the encrypted data key as another Hiera key-value in the eyaml file. All secret data ciphertext are encrypted with that data key then stored without the encrypted data key. This scheme works only if the same data key is used to encrypt all the secrets in that single file. However different encrypted yaml files may use different data keys.

Authoring secrets at design time

To be able to add, remove or update the secrets, you need to add install these Ruby gems on a secure administration workstaton.

Install the hiera-eyaml and hiera-eyaml-kms gems into the user/system space using gem install

gem install hiera-eyaml hiera-eyaml-kms

Accessing secrets at run time when operating masterless

Usually in a Puppet Master environment, the Puppet agent on nodes need not be able to decrypt secrets. However in a masterless Puppet environment, your agents will need this ability.

Install the hiera-eyaml and hiera-eyaml-kms gems into the puppet agent space

/opt/puppetlabs/puppet/bin/gem install

Accessing secrets at run time on a Puppet Master

Install the hiera-eyaml and hiera-eyaml-kms gems into the Puppet Server space to allow Puppet Server to access secrets at run time.

puppetserver gem install hiera-eyaml hiera-eyaml-kms -p http://my_proxy:80

The Puppet Server's gem install command does not pick up the https_proxy and http_proxy environment variables so you'll need to add your proxy settings to the command.

Encrypted values in eyaml are decrypted first before the manifests are compiled and the resultant catalogs sent to the agents.

AWS KMS configuration

Here is the portion of the Puppet Master CloudFormation template that configures the KMS service. It also creates an alias for easier consumption by our code.

The IAM Role identied by PuppetAdminRoleArn (supplied as a parameter to the template) are given wider privileges to manage the created master key.

The SplunkPuppetIamRole.Arn is only given limited privileges to submit decryption requests

    Type: "AWS::KMS::Key"
      Description: 'KMS key for Puppet Master to use to manage secrets in Hiera'
      Enabled: True
      EnableKeyRotation: False
        - Key: Name
          Value: Puppet Server key
        - Key: CostCentre
          Value: 123123
        - Key: Application
          Value: Splunk
        - Key: Environment
          Value: Production
        Version: "2012-10-17"
            Sid: "Allow administration of the key"
            Effect: "Allow"
              AWS: !Ref PuppetAdminRoleArn
              - "kms:Create*"
              - "kms:Describe*"
              - "kms:Enable*"
              - "kms:List*"
              - "kms:Put*"
              - "kms:Update*"
              - "kms:Revoke*"
              - "kms:Disable*"
              - "kms:Get*"
              - "kms:Delete*"
              - "kms:ScheduleKeyDeletion"
              - "kms:CancelKeyDeletion"
            Resource: "*"
            Sid: "Allow use of the key"
            Effect: "Allow"
              AWS: !GetAtt SplunkPuppetIamRole.Arn
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:DescribeKey"
            Resource: "*"
    Type: AWS::KMS::Alias
      AliasName: alias/PuppetMasterKey
      TargetKeyId: !Ref PuppetMasterKmsCmk

Hiera-eyaml configuration

In our Hiera configuration file, provide the required values for the hiera-eyaml backend and KMS plugin.

By default, decrypted values not stored in the master's cache for security hardening. This means that the master calls the KMS API for each agent that requires the value lookup.

# hiera version 3

  - eyaml
  - yaml

  - "roles/%{::trusted.extensions.pp_role}"
  - "nodes/%{::trusted.certname}"
  - common

# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.

  :cache_decrypted: false
  :encrypt_method: 'KMS'
  :kms_key_id: 'alias/PuppetMasterKey'
  :kms_aws_region: 'ap-southeast-2'

Authoring secrets

Generating the ciphertext

On your secure admin workstation, generate the ciphertext of the secret materials.

$ cat .eyaml/config.yaml 
kms_key_id: 'alias/PuppetMasterKey'
kms_aws_region: 'ap-southeast-2'

$ eyaml encrypt -v -s 'secret_string' -n KMS
[hiera-eyaml-core] Loaded config from /home/ec2-user/.eyaml/config.yaml
string: ENC[KMS,AQICAHgUzZndqbZKNNEc/2PwwHo2h1hUPGj7ulA8WSNoNTtjFgG5N8P/cFemtqPfsg/gqVDTAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMTB4JgsgVY1Gaz81TAgEQgB8SMmxA15pqwix0cjBd54w22LMTs+s3OQ/pzVQ1w4Nj]


block: >

Updating Hiera data store

You can now mix in your secret values with plain text values, and the yaml files continue to be meaningful under version control, for example you can still easily identify changes between commits. Our production/common.eyaml contain the configurations that are common to all our production Splunk instances.

In the example below, we are setting the local admin user name and password to the same values on all the Splunk instances.

There is also a common.yaml equivalent containing only plain text values, for example setting the management port to 8089 on all Splunk instances.

# production/common.eyaml
splunk_admin_user: admin