Using Hiera to separate organisation configuration data from Puppet code is a well known best practice that allows you to write reusable modules. Usually configuration data involves sensitive materials such as passwords, private keys or personal details. Fortunately, the Hiera ecosystem includes projects that integrates AWS KMS. This post highlights how we used this powerful combination to manage secrets using Puppet.
This post assumes the Puppet infrastructure and a Puppet Master is operational. Note that in all the Ruby gem install commands, beware if you need to use http proxy settings to access the internet.
The hiera-eyaml backend to Hiera is a flexible Hiera data store allowing for per-value encryption of these secrets within yaml files. The great thing about the open source project is its plugin architecture allowing you to select your encryption scheme.
Along comes the hiera-eyaml-kms open source project. This Ruby gem is a plugin to hiera-eyaml that integrates with AWS KMS. The benefits are that you no longer need to manage the private keys for encrypting all your secrets thereby minimising unauthorised exposure of keys. The master keys are now stored in AWS KMS.
A limitation of the plugin is that envelope encryption is not used to encrypt data. This issue confirms the point
Envelope encryption means that the master key is used to generate unique data keys, which are then used to encrypt your data. Besides missing out on another security defense layer, it means the plugin can only encrypt and decrypt data less than 4096 bytes. This is a limitiation of AWS KMS to encourage use of envelope encryption to protect application secrets of abitrary size.
It's probably safe to guess that this was a concenscious design decision by the project authors. Envelope encryption requires extra functionality to perform the crpto operations to encrypt and decrypt the data keys.
Also, envelope encryption requires the encrypted data key to be stored next the data ciphertext within the 'envelope'. This increases the size of the encrypted value that is stored.
A workaround may be to add extra functionality to enable only storing the encrypted data key as another Hiera key-value in the eyaml file. All secret data ciphertext are encrypted with that data key then stored without the encrypted data key. This scheme works only if the same data key is used to encrypt all the secrets in that single file. However different encrypted yaml files may use different data keys.
To be able to add, remove or update the secrets, you need to add install these Ruby gems on a secure administration workstaton.
Install the hiera-eyaml and hiera-eyaml-kms gems into the user/system space using gem install
gem install hiera-eyaml hiera-eyaml-kmsUsually in a Puppet Master environment, the Puppet agent on nodes need not be able to decrypt secrets. However in a masterless Puppet environment, your agents will need this ability.
Install the hiera-eyaml and hiera-eyaml-kms gems into the puppet agent space
/opt/puppetlabs/puppet/bin/gem installInstall the hiera-eyaml and hiera-eyaml-kms gems into the Puppet Server space to allow Puppet Server to access secrets at run time.
puppetserver gem install hiera-eyaml hiera-eyaml-kms -p http://my_proxy:80The Puppet Server's gem install command does not pick up the https_proxy and http_proxy environment variables so you'll need to add your proxy settings to the command.
Encrypted values in eyaml are decrypted first before the manifests are compiled and the resultant catalogs sent to the agents.
Here is the portion of the Puppet Master CloudFormation template that configures the KMS service. It also creates an alias for easier consumption by our code.
The IAM Role identied by PuppetAdminRoleArn (supplied as a parameter to the template) are given wider privileges to manage the created master key.
The SplunkPuppetIamRole.Arn is only given limited privileges to submit decryption requests
Resources:
PuppetMasterKmsCmk:
Type: "AWS::KMS::Key"
Properties:
Description: 'KMS key for Puppet Master to use to manage secrets in Hiera'
Enabled: True
EnableKeyRotation: False
Tags:
- Key: Name
Value: Puppet Server key
- Key: CostCentre
Value: 123123
- Key: Application
Value: Splunk
- Key: Environment
Value: Production
KeyPolicy:
Version: "2012-10-17"
Statement:
-
Sid: "Allow administration of the key"
Effect: "Allow"
Principal:
AWS: !Ref PuppetAdminRoleArn
Action:
- "kms:Create*"
- "kms:Describe*"
- "kms:Enable*"
- "kms:List*"
- "kms:Put*"
- "kms:Update*"
- "kms:Revoke*"
- "kms:Disable*"
- "kms:Get*"
- "kms:Delete*"
- "kms:ScheduleKeyDeletion"
- "kms:CancelKeyDeletion"
Resource: "*"
-
Sid: "Allow use of the key"
Effect: "Allow"
Principal:
AWS: !GetAtt SplunkPuppetIamRole.Arn
Action:
- "kms:Encrypt"
- "kms:Decrypt"
- "kms:ReEncrypt*"
- "kms:GenerateDataKey*"
- "kms:DescribeKey"
Resource: "*"
PuppetMasterKmsAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: alias/PuppetMasterKey
TargetKeyId: !Ref PuppetMasterKmsCmkIn our Hiera configuration file, provide the required values for the hiera-eyaml backend and KMS plugin.
By default, decrypted values not stored in the master's cache for security hardening. This means that the master calls the KMS API for each agent that requires the value lookup.
---
# hiera version 3
:backends:
- eyaml
- yaml
:hierarchy:
- "roles/%{::trusted.extensions.pp_role}"
- "nodes/%{::trusted.certname}"
- common
:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
:datadir:
:eyaml:
:cache_decrypted: false
:encrypt_method: 'KMS'
:kms_key_id: 'alias/PuppetMasterKey'
:kms_aws_region: 'ap-southeast-2'On your secure admin workstation, generate the ciphertext of the secret materials.
$ cat .eyaml/config.yaml
---
kms_key_id: 'alias/PuppetMasterKey'
kms_aws_region: 'ap-southeast-2'
$ eyaml encrypt -v -s 'secret_string' -n KMS
[hiera-eyaml-core] Loaded config from /home/ec2-user/.eyaml/config.yaml
string: ENC[KMS,AQICAHgUzZndqbZKNNEc/2PwwHo2h1hUPGj7ulA8WSNoNTtjFgG5N8P/cFemtqPfsg/gqVDTAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMTB4JgsgVY1Gaz81TAgEQgB8SMmxA15pqwix0cjBd54w22LMTs+s3OQ/pzVQ1w4Nj]
OR
block: >
ENC[KMS,AQICAHgUzZndqbZKNNEc/2PwwHo2h1hUPGj7ulA8WSNoNTtjFgG5N8P/cFem
tqPfsg/gqVDTAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEH
ATAeBglghkgBZQMEAS4wEQQMTB4JgsgVY1Gaz81TAgEQgB8SMmxA15pqwix0
cjBd54w22LMTs+s3OQ/pzVQ1w4Nj]You can now mix in your secret values with plain text values, and the yaml files continue to be meaningful under version control, for example you can still easily identify changes between commits.
Our production/common.eyaml contain the configurations that are common to all our production Splunk instances.
In the example below, we are setting the local admin user name and password to the same values on all the Splunk instances.
There is also a common.yaml equivalent containing only plain text values, for example setting the management port to 8089 on all Splunk instances.
# production/common.eyaml
splunk_admin_user: admin
splunk_admin_password:>
ENC[KMS,AQICAHgUzZndqbZKNNEc/2PwwHo2h1hUPGj7ulA8WSNoNTtjFgG5N8P/cFem
tqPfsg/gqVDTAAAAYjBgBgkqhkiG9w0BBwagUzBRAgEAMEwGCSqGSIb3DQEH
ATAeBglghkgBZQMEAS4wEQQMTB4JgsgVY1Gaz81TAgEQgB8SMmxA15pqwix0
cjBd54w22LMTs+s3OQ/pzVQ1w4Nj]