Azure DevOps Integration

The InsightCloudSec Azure DevOps integration helps your security and development teams integrate infrastructure-as-code (IaC) security and compliance scans (serviced using the CLI IaC Scanning Tool mimics) with Azure Pipelines and Boards. Setting up this integration is as simple as copying a YAML file to your code repository!

Prerequisites

  • InsightCloudSec Domain Admin permissions

HTML viewer extension recommended

Though it's not required, InsightCloudSec highly recommends including a third-party HTML viewer extension as part of your Azure DevOps organization as it can improve the IaC output and pipeline artifacts viewing experience.

Integrate with Azure Pipelines

To integrate the InsightCloudSec CLI IaC Scanning Tool (mimics) with Azure Pipelines, you'll need to incorporate one of the example YAML files included in the following sections into your code repository. The file you use depends on how you are currently using mimics. Currently, InsightCloudSec supports running mimics as an executable or using Docker. In each example YAML file, we have included the following Azure Pipeline environment variables to keep sensitive data secure and interact with InsightCloudSec and AWS Elastic Container Registry (ECR) if applicable:

  • BASE_URL - The web address of your InsightCloudSec environment, which can be found in System Settings.
  • ICS_API_KEY - An API Key used to access InsightCloudSec data without traditional username-password authentication
    • We recommend this value remains a secret within Azure DevOps
  • IAC_TEMPLATE_DIR_PATH - The file path to the directory that contains your IaC templates (relative to the project root). For example, /path/to/cloudformation.
  • IAC_CONFIG_NAME - The name of the IaC Configuration to run the scan against
    • If your configuration name contains spaces, you will need to wrap the environment variable value in quotation marks. For example, "Test Configuration".

For more information on Azure Pipelines variables, see Define Variables.

Planning on storing sensitive information within Azure DevOps environment variables?

Azure DevOps environment variables can be stored as secrets to not expose any sensitive information, such as API or access keys. When configuring environment variables for your Azure Pipeline, ensure that sensitive data is not exposed by selecting the Keep this a secret checkbox when inputting variables.

Run as an executable

The CLI IaC Scanning Tool can be run as an executable using Terraform or Amazon Web Services' (AWS) CloudFormation.

Using Terraform
yaml
1
trigger: 
2
  branches:
3
    include: 
4
    - releases/*
5


6
pool:
7
  vmImage: "ubuntu-latest" # Define the virtual machine image to use for the pipeline
8


9
variables:
10
  terraformVersion: "1.9.2" # Define the Terraform version to ensure consistency across runs
11
  terraformDownloadUrl: "https://releases.hashicorp.com/terraform/$(terraformVersion)/terraform_$(terraformVersion)_linux_amd64.zip" # URL for Terraform download
12
  mimicsDownloadUrl: "https://artifacts.rapid7.com/cloudsec/mimics/latest/mimics_latest_linux_amd64" # URL for Mimics download
13


14
jobs:
15
  - job: Rapid7_IaC_Terraform_Scanner
16
    displayName: "Rapid7 IaC scanner"
17


18
    steps:
19
      # Check if Terraform is already installed, and download it if necessary
20
      - script: |
21
          if [ ! -f /usr/local/bin/terraform ]; then
22
              wget $(terraformDownloadUrl)
23
          else
24
              mv /usr/local/bin/terraform /usr/local/bin/terraform.old
25
              wget $(terraformDownloadUrl)
26
          fi
27


28
          # Unzip and install Terraform
29
          unzip "terraform_$(terraformVersion)_linux_amd64.zip" -d terraform_temp
30
          chmod +x terraform_temp/terraform
31
          mv terraform_temp/terraform /usr/local/bin/
32
        displayName: "Install Terraform"
33


34
      # Initialize Terraform, create a plan, and convert it to JSON
35
      - script: |
36
          terraform init
37
          terraform plan -out=tf.plan
38
          terraform show -json tf.plan > plan.json
39
        displayName: "Initialize Terraform & Generate Terraform plan"
40
        workingDirectory: $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH)
41


42
      # Format plan.json to provide position output of violating resources
43
      - script: |
44
          jq "." ./plan.json > formatted_plan.json
45
          mv formatted_plan.json plan.json
46
        displayName: "Format Terraform Plan to provide accurate output"
47
        workingDirectory: $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH)
48


49
       # Download the IaC Scanning Tool Mimics executable
50
      - script: |
51
          wget $(mimicsDownloadUrl) -O $(Agent.ToolsDirectory)/mimics
52
          chmod +x $(Agent.ToolsDirectory)/mimics
53
        displayName: "Install Mimics-binary" # Downloads and installs the Mimics tool (binary executable) for infrastructure as code (IaC) scanning
54


55
      # Use the Mimics executable to scan the plan.json
56
      - script: |
57
          mkdir -p $(System.DefaultWorkingDirectory)/mimics-reports
58
          $(Agent.ToolsDirectory)/mimics scan \
59
            $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH)/plan.json \
60
            --api-key $(ICS_API_KEY) \
61
            --base-url $(BASE_URL) \
62
            --no-verify \
63
            --ics-config $(IAC_CONFIG_NAME) \
64
            --log-format json \
65
            --report-formats all \
66
            --report-name results-rapid7_iac \
67
            --report-path "$(System.DefaultWorkingDirectory)/mimics-reports" \
68
            --save-report \
69
            --no-fail \
70
            --verbose
71
        displayName: "Scan IaC files with Mimics"
72


73
      # Publish additional files (results-rapid7_iac artifacts)
74
      - task: PublishBuildArtifacts@1
75
        displayName: "Publish Scan Artifacts"
76
        inputs:
77
          pathtoPublish: "$(System.DefaultWorkingDirectory)/mimics-reports"
78
          artifactName: "results-rapid7_iac" # Publishes the scan reports as build artifacts for later access
79


80
      # Publish the HTML report using the PublishHtmlReport task
81
      - task: PublishHtmlReport@1
82
        condition: succeededOrFailed()
83
        inputs:
84
          reportDir: $(System.DefaultWorkingDirectory)/mimics-reports/results-rapid7_iac.html
85
          tabName: "R7 IaC Scan Results" # Publishes the HTML scan report to the Azure DevOps build summary for easy access
86
Using CloudFormation
yaml
1
trigger: 
2
  branches:
3
    include: 
4
    - releases/*
5


6
pool:
7
  vmImage: "ubuntu-latest" # Define the virtual machine image to use for the pipeline
8


9
variables:
10
  mimicsDownloadUrl: "https://artifacts.rapid7.com/cloudsec/mimics/latest/mimics_latest_linux_amd64" # URL for Mimics download
11


12
jobs:
13
  - job: Rapid7_IaC_CloudFormation_Scanner
14
    displayName: "Rapid7 IaC scanner"
15


16
    steps:
17
       # Download the IaC Scanning Tool Mimics executable
18
      - script: |
19
          wget $(mimicsDownloadUrl) -O $(Agent.ToolsDirectory)/mimics
20
          chmod +x $(Agent.ToolsDirectory)/mimics
21
        displayName: "Install Mimics-binary" # Downloads and installs the Mimics tool (binary executable) for infrastructure as code (IaC) scanning
22


23
      # Use the Mimics executable to scan the plan.json
24
      - script: |
25
          mkdir -p $(System.DefaultWorkingDirectory)/mimics-reports
26
          $(Agent.ToolsDirectory)/mimics scan \
27
            $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH) \
28
            --api-key $(ICS_API_KEY) \
29
            --base-url $(BASE_URL) \
30
            --no-verify \
31
            --ics-config $(IAC_CONFIG_NAME) \
32
            --log-format json \
33
            --report-formats all \
34
            --report-name results-rapid7_iac \
35
            --report-path "$(System.DefaultWorkingDirectory)/mimics-reports" \
36
            --save-report \
37
            --no-fail \
38
            --verbose
39
        displayName: "Scan IaC files with Mimics"
40


41
      # Publish additional files (results-rapid7_iac artifacts)
42
      - task: PublishBuildArtifacts@1
43
        displayName: "Publish Scan Artifacts"
44
        inputs:
45
          pathtoPublish: "$(System.DefaultWorkingDirectory)/mimics-reports"
46
          artifactName: "results-rapid7_iac" # Publishes the scan reports as build artifacts for later access
47


48
      # Publish the HTML report using the PublishHtmlReport task
49
      - task: PublishHtmlReport@1
50
        condition: succeededOrFailed()
51
        inputs:
52
          reportDir: $(System.DefaultWorkingDirectory)/mimics-reports/results-rapid7_iac.html
53
          tabName: "R7 IaC Scan Results" # Publishes the HTML scan report to the Azure DevOps build summary for easy access
54


55

Run using Docker

The CLI IaC Scanning Tool can also be run as a Docker container using Terraform or AWS CloudFormation, but requires additional configuration. To run mimics in a Docker container, you'll need to provide the following variables to authenticate for the AWS CLI so InsightCloudSec can retrieve the mimics container image from AWS Elastic Container Registry (ECR):

  • AWS_ACCESS_KEY_ID - The ID of the Access Key associated with an IAM User to enable programmatic requests
  • AWS_SECRET_ACCESS_KEY - The secret value of the Access Key

To view all publicly-available mimics container images, visit the Rapid7 ECR Gallery. For more information on AWS Access Keys and long-term credentials, see Managing access keys for IAM users.

AWS authentication should be stored as a secret

Both variables should be stored as secrets within your Azure DevOps Pipeline.

Using Terraform
yaml
1
trigger: 
2
  branches:
3
    include: 
4
    - releases/*
5


6
pool:
7
  vmImage: "ubuntu-latest" # Define the virtual machine image to use for the pipeline
8


9


10
variables:
11
  terraformVersion: "1.9.2" # Define the Terraform version to ensure consistency across runs
12
  terraformDownloadUrl: "https://releases.hashicorp.com/terraform/$(terraformVersion)/terraform_$(terraformVersion)_linux_amd64.zip" # URL for Terraform download
13


14
jobs:
15
- job: Rapid7_Docker_IaC_CloudFormation_Scanner
16
  displayName: "Rapid7 Docker IaC scanner"
17
  steps:
18


19
  # Configure AWS CLI with access key, secret key, and region
20
  - script: |
21
      aws configure set aws_access_key_id $(AWS_ACCESS_KEY_ID)
22
      aws configure set aws_secret_access_key $(AWS_SECRET_ACCESS_KEY)
23
    displayName: "Configuring AWS CLI Credentials"
24
    
25
  # Check if Terraform is already installed, and download it if necessary
26
  - script: |      
27
      if [ ! -f /usr/local/bin/terraform ]; then
28
          wget $(terraformDownloadUrl)
29
      else
30
          mv /usr/local/bin/terraform /usr/local/bin/terraform.old
31
          wget $(terraformDownloadUrl)
32
      fi
33
      
34
      # Unzip and install Terraform
35
      unzip terraform_$(terraformVersion)_linux_amd64.zip -d terraform_temp
36
      chmod +x terraform_temp/terraform
37
      mv terraform_temp/terraform /usr/local/bin/
38
    displayName: "Install Terraform"
39


40
  # Initialize Terraform, create a plan, and convert it to JSON
41
  - script: |
42
      terraform init
43
      terraform plan -out tf.plan
44
      terraform show -json tf.plan > plan.json
45
    displayName: "Initialize Terraform & Generate Terraform plan"
46
    workingDirectory: $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH)
47


48
  # Format plan.json to provide position output of violating resources
49
  - script: |
50
      jq '.' ./plan.json > formatted_plan.json
51
      mv formatted_plan.json plan.json
52
    displayName: "Format Terraform Plan to provide accurate output"
53
    workingDirectory: $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH)
54


55
  # Scan the provided IaC Template with the IaC mimics image
56
  - script: |
57
      pull(){
58
        mkdir -p $(System.DefaultWorkingDirectory)/data/mimics-reports
59
        docker run \
60
        -v $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH):/data \
61
        -e MIMICS_BASE_URL=$(BASE_URL) \
62
        -e MIMICS_API_KEY=$(ICS_API_KEY) \
63
        --name iac-mimics-container \
64
        public.ecr.aws/rapid7-insightcloudsec/ics/mimics:latest scan /data/plan.json --no-verify --ics-config $(IAC_CONFIG_NAME) --log-format json --report-formats all --report-name results-rapid7_iac --report-path "/data/mimics-reports" --save-report --no-fail --verbose
65
      }
66


67
      echo '1 2 3 5 8 13' | tr ' ' '\n' | while read t; do
68
        pull && break
69
        echo >&2 "docker pull failed; sleeping $t seconds"
70
        sleep $t
71
      done  
72
    displayName: "Scan IaC files with Mimics"
73


74
  # Copy mimics reports from docker container to host
75
  - script: |
76
      docker cp iac-mimics-container:data/mimics-reports $(System.DefaultWorkingDirectory)/data
77
    displayName: Copy Mimics Reports from Docker Container to Host
78


79
  # Publish additional files (results-rapid7_iac artifacts)
80
  - task: PublishBuildArtifacts@1
81
    displayName: "Publish Scan Artifacts"
82
    inputs:
83
      pathtoPublish: "$(System.DefaultWorkingDirectory)/data/mimics-reports"
84
      artifactName: "results-rapid7_iac"
85


86
  # Publish the HTML report using the PublishHtmlReport task
87
  - task: PublishHtmlReport@1
88
    displayName: "Publish HTML Report"
89
    condition: succeededOrFailed()
90
    inputs:
91
      reportDir: $(System.DefaultWorkingDirectory)/data/mimics-reports/results-rapid7_iac.html
92
      tabName: "R7 IaC Scan Results"
Using CloudFormation
yaml
1
trigger: 
2
  branches:
3
    include: 
4
    - releases/*
5


6
pool:
7
  vmImage: "ubuntu-latest" # Define the virtual machine image to use for the pipeline
8


9
jobs:
10
- job: Rapid7_Docker_IaC_Terraform_Scanner
11
  displayName: "Rapid7 Docker IaC scanner"
12
  steps:
13


14
  # Configure AWS CLI with access key, secret key, and region
15
  - script: |
16
      aws configure set aws_access_key_id $(AWS_ACCESS_KEY_ID)
17
      aws configure set aws_secret_access_key $(AWS_SECRET_ACCESS_KEY)
18
    displayName: "Configuring AWS CLI Credentials"
19


20
  # Scan the provided IaC Template with the IaC mimics image
21
  - script: |
22
      pull(){
23
        mkdir -p $(System.DefaultWorkingDirectory)/data/mimics-reports
24
        # Use Docker to run the Mimics tool for scanning
25
        docker run \
26
        -v $(System.DefaultWorkingDirectory)$(IAC_TEMPLATE_DIR_PATH):$(IAC_TEMPLATE_DIR_PATH) \
27
        -e MIMICS_BASE_URL=$(BASE_URL) \
28
        -e MIMICS_API_KEY=$(ICS_API_KEY) \
29
        --name iac-mimics-container \
30
        public.ecr.aws/rapid7-insightcloudsec/ics/mimics:latest scan $(IAC_TEMPLATE_DIR_PATH) --no-verify --ics-config $(IAC_CONFIG_NAME) --log-format json --report-formats all --report-name results-rapid7_iac --report-path "/data/mimics-reports" --save-report --no-fail --verbose
31
      }
32


33
      echo '1 2 3 5 8 13' | tr ' ' '\n' | while read t; do
34
        pull && break
35
        echo >&2 "docker pull failed; sleeping $t seconds"
36
        sleep $t
37
      done  
38
    displayName: "Scan IaC files with Mimics"
39


40
  # Copy mimics reports from docker container to host
41
  - script: |
42
      docker cp iac-mimics-container:data/mimics-reports $(System.DefaultWorkingDirectory)/data
43
    displayName: Copy Mimics Reports from Docker Container to Host
44


45
  # Publish additional files (results-rapid7_iac artifacts)
46
  - task: PublishBuildArtifacts@1
47
    displayName: "Publish Scan Artifacts"
48
    inputs:
49
      pathtoPublish: "$(System.DefaultWorkingDirectory)/data/mimics-reports"
50
      artifactName: "results-rapid7_iac"
51


52
  # Publish the HTML report using the PublishHtmlReport task
53
  - task: PublishHtmlReport@1
54
    displayName: "Publish HTML Report"
55
    condition: succeededOrFailed()
56
    inputs:
57
      reportDir: $(System.DefaultWorkingDirectory)/data/mimics-reports/results-rapid7_iac.html
58
      tabName: "R7 IaC Scan Results"

Integrate with Azure Boards (optional)

To integrate the InsightCloudSec CLI IaC Scanning Tool (mimics) with Azure Boards, you'll need to follow the instructions for integrating Azure Pipelines first and then incorporate the example in this section with the appropriate YAML file for your setup. In the following example YAML section, we have included the following environment variables to assist your new InsightCloudSec Azure Pipelines integration in interacting with Azure Boards:

  • PAT - A Personal Access Token (PAT) for your Azure DevOps user, which helps pipelines create work items for each finding produced by the IaC analysis scans.
    • PATs should be stored as a secret within your Azure DevOps environment variables. For more information on PATs, visit Use Personal Access Tokens.
  • AZURE_BOARDS_ORGANIZATION - The Azure DevOps Organization where the Azure Board work items should be placed.
  • AZURE_BOARDS_PROJECT - The Azure DevOps Project where Azure Board work items should be placed.
Example Azure Board script for Azure Pipeline integration

The following example section needs to be placed in the appropriate section of the example Azure Pipelines YAML file. The location is determined by how you are running mimics:

  • If you are running mimics using an executable, copy and paste it after the Scan IaC files with Mimics step
  • If you are running mimics using a Docker container, copy and paste it after the Copy Mimics Reports from Docker Container to Host step
yaml
1
# Create Azure Devops work items for each finding produced by the IaC Mimics scan
2
- script: |
3
    # Construct the URL for API requests, choosing the correct work item type and API version.
4
    url="https://dev.azure.com/$(AZURE_BOARDS_ORGANIZATION)/$(AZURE_BOARDS_PROJECT)/_apis/wit/workitems/\$Issue?api-version=6.0"
5


6
    # Specify the path to the SARIF file generated by the previous steps
7
    sarifFile="$(System.DefaultWorkingDirectory)/mimics-reports/results-rapid7_iac.sarif"
8


9
    # Check for the existence of the SARIF file and process it
10
    if [ -f "$sarifFile" ]; then
11
      echo "Found SARIF file: $sarifFile"
12


13
      # Iterate through each result in the SARIF file, extracting relevant details
14
      jq -c '.runs[].results[]' "$sarifFile" | while read -r result; do
15
        ruleId=$(jq -r ".ruleId" <<< $result)
16
        message=$(jq -r ".message.text" <<< $result)
17
        locations=$(jq -r '.locations[]' <<< $result)
18
        title=$(jq -c --arg ruleId "$ruleId" '.runs[].tool.driver.rules[] | select(.id == $ruleId).name' "$sarifFile")
19


20
        # Prepare a JSON payload for creating a new work item via the Azure DevOps REST API
21
        json=$(jq --arg ruleId "$ruleId" --arg message "$message" --arg title "$title" --arg project "$(AZURE_BOARDS_PROJECT)" '
22
          [
23
            {"op": "add", "path": "/fields/System.Title", "value": $title},
24
            {"op": "add", "path": "/fields/System.Description", "value": "<div>\($message)</div><pre style=\"white-space: pre-line\">\(.properties.changes)</pre>"},
25
            {"op": "add", "path": "/fields/System.State", "value": "To Do"},
26
            {"op": "add", "path": "/fields/System.AreaPath", "value": $project}
27
          ]
28
        ' <<< $locations)
29


30
        # Make an API call to create a new work item with the prepared payload
31
        curl -X POST -H "Content-Type: application/json-patch+json" -H "Authorization: Basic $(echo -n ":$PAT" | base64)" -d "$json" "$url"
32
      done
33
    else
34
      echo "SARIF file not found: $sarifFile"
35
    fi
36
  displayName: "Process SARIF and Create Azure Board Work Items"
37
  env:
38
    PAT: $(PAT) # Pass the Personal Access Token (PAT) securely as an environment variable

Create a Azure Pipeline

After you have selected the appropriate example YAML file for your project and copied it to the code repository, you are ready to set up the integration in Azure Pipelines.

Something not quite right?

For the most up-to-date information, refer to Microsoft Azure’s documentation on creating an Azure Pipeline.

  1. Log in to the Azure Console.
  2. Navigate to Azure Pipelines.
  3. Click New Pipeline.
  4. In the Connect section, follow the steps to connect and select the host of your code repository.
  5. In the Configure section, click Starter Pipeline.
  6. Copy and paste the example YAML file you selected in the Integrate with Azure Pipelines over the existing YAML on this page.
  7. Click Variables and provide the applicable environment variables.
  8. Validate the YAML and update as necessary. For example, setting pipeline triggers and a valid Terraform version.
  9. Click Run > Save.