Writing Cloud Custodian Compliance Policies

Kion includes the open-source Cloud Custodian (c7n) rules engine, so you can easily write and run YAML policies against your cloud resources, like EC2 instances, VPCs, root users, etc. This article explains the basics of writing your own Cloud Custodian compliance policy and how webhook actions work in Kion. We suggest also reading the Cloud Custodian documentation for more in depth information and for example policies.

These policies are implemented using Kion compliance checks. For information about adding policies to compliance checks, see Add a Compliance Check.

Policy Development Environment Setup

It is best practice to develop policies in a standalone environment where immediate feedback can be received. Once policies have been tested and verified, they can be deployed to production by Kion

Requirements

To configure the development environment for policies, ensure you have access to:

• Python 3.
• A cloud account with control over applicable resources for policy testing
  This account should have an associated Cloud Access Role in Kion that allows for the generation of short-term access keys for AWS. Credentials will need to be created natively for all other cloud providers.
• (Highly recommended) An editor supporting YAML syntax highlighting and validation.

Setting Up the Policy Development Environment

Unless otherwise noted within the steps themselves, all of these steps should be executed on your development system.

1. Determine the version of c7n used in your version of Kion. Connect to a Kion node and execute one of the following commands as root (for older installations, the container name may be cloudtamer\_compliance\_1):

   • For AWS-focused development: docker exec kion_compliance_1 pip3 show c7n | grep '^Version:'
   • For Azure-focuseed development: docker exec kion_compliance_1 pip3 show c7n-azure | grep '^Version:'
   • For GCP-focused development: docker exec kion_compliance_1 pip3 show c7n-gcp | grep '^Version:'

2. Back on your own system for development, create a new folder for testing. This will contain both the policy documents and the required outputs that c7n should generate.

3. Change directories to the new folder in the command line.

   Note: If using Visual Studio Code, execute the command Install 'code' in PATH from the Command Palette for a shortcut for opening files in the editor.

4. Generate a new virtual environment for c7n development: python3 -m venv python-custodian

5. Activate the virtual environment.

   • For bash/zsh: source python-custodian/bin/activate
   • For other environments, see this article.

6. Install c7n:

   • For AWS development: pip3 install c7n==[VERSION]
   • For Azure development: pip3 install c7n-azure==[VERSION]
   • For GCP development: pip3 install c7n-gcp==[VERSION]
   • Replace [VERSION] above with the version found in Step 1.

7. c7n is now installed in the virtual environment and ready for testing.

Policy Development Cycle

Follow this cycle each time you work on a policy:

1. Change directories to the new folder from the command line.

2. Activate the virtual environment using the same instructions outlined above in "Setting Up the Policy Environment".

3. Configure your environment:

   AWS

   1. Navigate to the account you plan to use for testing in Kion

   2. Select the cloud account icon, choose your role, and select Short-term Access Keys.

   3. Copy the block from the dialog box.

   4. Paste the block into the same terminal being used for testing.

   5. Set the appropriate AWS region for the shell using the variables: AWS_DEFAULT_REGION and AWS_REGION.

      • For bash/zsh: export AWS_DEFAULT_REGION=[region] && export AWS_REGION=[region]

   Azure

   Configure the environment using these instructions.

   Google Cloud

   Configure your environment appropriately using these instructions. You must also specify the GOOGLE_CLOUD_PROJECT environment variable as your GCP Project ID.

4. Create or edit your policy file.

   Note: If you are using Visual Studio Code and have installed the code tool from the above tip, you can enter code [filename] in the built-in terminal to open the files in editor.

5. Validate your policy: custodian validate [policyfile]. You should see a line validating the configuration.

6. Execute your policy: custodian run --cache-period 0 -s [output dir] [policyfile]

   • For example, you can store our output in the directory out and execute the policy test-tags.yml with this command: custodian run --cache-period 0 -s out test-tags.yml

7. Check the output to ensure that the appropriate resources are listed: custodian report -s [output dir] [policyfile]

   • Following the previous example, you can run this command: custodian report -s out test-tags.yml

8. Test each filter top confirm that they work as intended before developing actions. Be sure to test each portion of filters and actions you enact.

Compliance Policy Basics

The remainder of this guide provides tips on developing policies.

A Cloud Custodian policy should define, at a minimum:

• name. A unique name for the policy.
• resource. The type of resource the policy runs against.
• filters. Filters narrow down the resources that the policy runs against. Filters can include tag, type, key, value, and logic statements (and, or, not).
• actions. Actions to take on the resources. This must include an action that posts the findings to Kion via a webhook. Optionally, it can also include actions to automatically remediate the finding.

The name, resource, and filters sections are native c7n YAML. You can copy/paste pre-written policies from other sources (such as GitHub repos), and add the webhook to post to Kion.

The actions section contains a webhook for posting to Kion and automatic remediation actions. The actions section typically includes the following variables:

• {{CT::CheckId}}. The check ID that will be scanned.
• {{CT::Authorization}}. The authorization token for ingesting findings via the public scan API.
• {{CT::CallbackURL}}. The callback URL cloud custodian will POST to for ingesting findings via the public scan API.

The compliance check frequency and regions are set when creating a compliance check and do not need to be included in the policy code.

When adding or editing a compliance check, the Validate Policy button ensures your policy code is valid.

AWS

Policy creation example for AWS

Basic Policy

This policy stops all EC2 instances that are tagged in the AWS Console as Test. This is the kind of policy you might find online.

policies:
  - name: stop-ec2-test-instances
    resource: aws.ec2   
filters:   
      - "tag:Test": present   
actions:
      - type: stop

As it is, the policy would work, but it would not post its findings to Kion.

Webhook for posting to Kion

To ensure findings are posted to Kion, you need to include a webhook action.

actions:  
      - type: webhook  
        url: '{{CT::CallbackURL}}'  
        method: POST  
        batch: true  
        headers:  
          Authorization: '`{{CT::Authorization}}`'  
        body: |-  
          {  
              "compliance_check_id": `{{CT::CheckId}}`,  
              "account_number": account_id,  
              "region": region,  
              "scan_started_at": execution_start,  
              "findings": resources[].{resource_name: InstanceId, resource_type: `ec2`}
          }

Make sure to change the resource_name and resource_type in the webhook action to the appropriate values. For more information, see Required Compliance Policy Fields.

Final Policy

When put together, the basic policy we found plus the POST webhook looks like this.

policies:   
  - name: stop-ec2-test-instances   
    resource: aws.ec2   
    filters:   
      - "tag:Test": present   
    actions:   
      - type: stop  
      - type: webhook  
        url: '{{CT::CallbackURL}}'  
        method: POST  
        batch: true  
        headers:  
          Authorization: '`{{CT::Authorization}}`'  
        body: |-  
          {  
              "compliance_check_id": `{{CT::CheckId}}`,  
              "account_number": account_id,  
              "region": region,  
              "scan_started_at": execution_start,  
              "findings": resources[].{resource_name: InstanceId, resource_type: `ec2`}  
          }

AWS Policy References

• Resource types. You can find a list of resources and their common actions in Cloud Custodian's article: AWS Reference.
• Filters. You can find a list of common filters in Cloud Custodian's article: AWS Common Filters.
• Actions. Remember, the actions section contains a webhook for posting to Kion and automatic remediation actions. In most cases, you can use the webhook in the example above to post to Kion. You can find a list of common remediation actions in Cloud Custodian's article: AWS Common Actions.

Azure

Policy creation example for Azure

Basic Policy

This policy adds the tag Project:NA to all virtual machines with the name test. This is the kind of policy you might find online.

policies:
  - name: tag-test-vms-project-na
    resource: azure.vm   
filters:   
      - type: value  
      - key: name  
      - value: test  
actions:
      - type: tag  
      - tag: Project  
      - value: NA

Webhook for posting to Kion

To ensure findings are posted to Kion, you need to include a webhook action.

actions:  
      - type: webhook  
        url: '{{CT::CallbackURL}}'  
        method: POST  
        batch: true  
        headers:  
          Authorization: '`{{CT::Authorization}}`'  
        body: |-  
          {  
              "compliance_check_id": `{{CT::CheckId}}`,  
              "account_number": account_id,  
              "scan_started_at": execution_start,  
              "findings": resources[].{resource_name: VirtualMachine, resource_type: `vm`, region: location}
          }

Make sure to change the resource_name and resource_type in the webhook action to the appropriate values. For more information, see Required Compliance Policy Fields.

Final Policy

When put together, the basic policy we found plus the POST webhook looks like this.

policies:   
  - name: tag-test-vms-project-na   
    resource: azure.vm   
    filters:   
      - type: value  
      - key: name  
      - value: test  
    actions:     
      - type: tag  
        tag: Project  
        value: NA  
      - type: webhook  
        url: '{{CT::CallbackURL}}'  
        method: POST  
        batch: true  
        headers:  
          Authorization: '`{{CT::Authorization}}`'  
        body: |-  
          {  
              "compliance_check_id": `{{CT::CheckId}}`,  
              "account_number": account_id,  
              "scan_started_at": execution_start,  
              "findings": resources[].{resource_name: VirtualMachine, resource_type: `vm`, region: location}  
          }

Azure Policy References

• Resource type. You can find a list of resources and their common actions and filters in Cloud Custodian's article: Azure Reference.
• Filters. You can find a list of common filters in Cloud Custodian's article: Azure Common Filters.
• Actions. Remember, the actions section contains a webhook for posting to Kion and automatic remediation actions. In most cases, you can use the webhook in the example above to post to Kion. You can find a list of common remediation actions in Cloud Custodian's article: Azure Common Actions.

Including Metadata

Metadata is information that is not typically captured by Kion. You can add metadata fields to any policy to capture additional information like IP addresses, instance state, launch time, and more.

Policy with metadata example

policies:
 - name: ec2-instance-previous-generation
   resource: aws.ec2
   description: |
	Identify previous generation EC2 Instances
   actions:
	- type: webhook
	  url: '{{CT::CallbackURL}}'
	  method: POST
	  batch: true
	  headers:
		Authorization: '`{{CT::Authorization}}`'
	  body: |-
		{
		"compliance_check_id": `{{CT::CheckId}}`,
		"account_number": account_id,
		"region": region,
		"scan_started_at": execution_start,
		"findings": resources[].{resource_name: InstanceId, resource_type: `ec2`, data_json: {instance_state: State.Name, tags: Tags, instance_type: InstanceType, ebs_optimized: EbsOptimized, launch_time: LaunchTime, private_ip: PrivateIpAddress, public_ip: PublicIpAddress, vpc_id: VpcId, image: ImageId, private_dns: PrivateDnsName, ssh_key: KeyName, subnet: SubnetId, device_type: RootDeviceType}}
		}

• AWS Reference
• Azure Reference

Advanced Logic for Compliance Policies

Cloud Custodian supports various types of advanced and conditional logic.

The following policy is an example using or and not logic. This checks if the default security group has a value for either IpPermissions OR IpPermissionsEgress. Use this as a guideline to incorporate this logic into your own policies. The or could be replaced with and to only find cases where the default security group has a value for BOTH IpPermissions AND IpPermissionsEgress.

policies:  
  - name: cis_sg-default-allowing-traffic  
    resource: aws.security-group
    description: |
      CIS 1.2.0 - 4.3 Ensure the default security group for every VPC restricts all traffic
    filters:
      - GroupName: default
      - or:
          - not:
              - IpPermissions: empty
          - not:
              - IpPermissionsEgress: empty
    actions:
      - type: webhook
        url: '{{CT::CallbackURL}}'
        method: POST
        batch: true
        headers:
          Authorization: '`{{CT::Authorization}}`'
        body: |-
          {
              "compliance_check_id": `{{CT::CheckId}}`,
              "account_number": account_id,
              "region": region,
              "scan_started_at": execution_start,
              "findings": resources[].{resource_name: GroupId, resource_type: `security-group`, data_json: {ingress_rules: IpPermissions, egress_rules: IpPermissionsEgress}}
          }

Advanced Logic References

• AWS. See Cloud Custodian's article: Advanced Usage.
• Azure. See Cloud Custodian's article: Azure > Advanced Usage.

Downloads

• transparent.gif