Previous, I published a blog article on Publishing TFSec Terraform Quality Checks to Azure DevOps Pipelines. Continuing on the topic of working with DevOps, and performing quality checks on our Infrastructure-as-Code (IaC), this article will be similar, but focused on using Checkov.
Checkov is a static code analysis tool for infrastructure-as-code, published and maintained by BridgeCrew. It detects security and compliance misconfigurations in various templating languages including Terraform, Azure Resource Manager (ARM), and CloudFormation, among others.
Quality Checks for Terraform
Similar to the previously mentioned article, I have been focusing lately on DevOps and Infrastructure-as-Code (IaC), and in particular, HashiCorp Terraform. One of the tools that I am using to perform quality checks against my Terraform templates, is BridgeCrew’s Checkov.
At the time of this writing, Checkov has:
- 181 x AWS checks
- 106 x Azure checks
- 67 x GCP checks
- 142 x Kubernetes checks
When you look at the documentation for Checkov, you will see that, aside from installing it locally, you can also run Checkov in a Docker container. Here is the command you can use to do so:
docker run --tty --volume /directory-to-terraform-files:/tf bridgecrew/checkov --directory /tf
To run this in an Azure DevOps pipeline, this is what the Job looks like…
When you run this in the Azure pipeline, this is the type of output you would see. Notice that the execution exits with a non-zero exit code if a potential problem is detected. This enables us to be able to use it in a CI/CD pipeline and exit/error-out, as we would expect to.
That’s all great, but our goal is to use this in a DevOps CI/CD pipeline. And we want to be able to consume the results in the pipeline (not just from the terminal).
Publish the Results
In Azure Pipelines, the Publish Test Results task is used to publish test results. But how do we get the results out of the container, and into one of the supported results formats?
Here’s what worked for me…
docker run --tty --volume $(System.DefaultWorkingDirectory):/tf bridgecrew/checkov --directory /tf --output junitxml > $(System.DefaultWorkingDirectory)/Checkov-Report.xml
Notice a few things in this revised docker run command. First, notice the –output JUnitXML parameter. The Checkov Results documentation shows that we can request different output types by using this –output parameter. The supported formats are: CLI, JSON, JUnit XML. Knowing that the Azure DevOps Publish Test Results task supports JUnit as a test result format, that is what we are targeting.
Next, you’ll notice that, in order to retrieve the output from the docker container into pipeline, we are piping the StdOut to a file within the working directory of the pipeline (the ‘>‘ part).
After we get the output from the docker container, we can now use the Publish Test Results task, and publish the results to the pipeline.
A Little More Work Required
While that approach may seem simple and straightforward, you’re about to discover a few gotchas.
First, if Checkov has any issues with resolving anything in your Terraform code (like, say, a module reference), the terminal will show these as Warnings (as depicted below).
The problem is, these Warnings will also appear in the converted/produced XML file.
So… when the Azure DevOps pipeline runs the Publish Test Results task, it throws this error:
##[warning]Failed to read /home/vsts/work/1/s/junit.xml. Error : Data at the root level is invalid. Line 1, position 1..
Here’s what the error looked like in the pipeline output:
So, let’s assume you’ve addressed these Warnings, and re-run the scan. In my case, the above mentioned error no longer occurred! But wait, there’s more!
This time, there was a new error. One about an invalid hexadecimal character, and on a different line.
##[warning]Failed to read /home/vsts/work/1/s/junit.xml. Error : '', hexadecimal value 0x1B, is an invalid character. Line 9, position 1..
Here’s what the error looked like in the pipeline output:
When you download and look at the resulting XML file, you will notice that at the end of the file, there are 2 additional lines:
This causes issues with the Publish Test Results task from publishing the results in the pipeline.
If you dig a little deeper into the Checkov code, you will find in the Python script how they generate the XML file output. Now, I don’t know Python myself, but I did encounter the same hexadecimal value error with another Terraform scanning tool that I’m using as well (namely TFSec). But when I used the same method that I am using for Checkov (the same StdOut redirect), the XML file (from TFSec) is rendered correctly and published successfully.
So, that made me think that the issue is with how Checkov is generating the XML file, and may be related to the parser or formatter. For reference, the TFSec product is using the following JUnit Schema.
The Good News
So the good news is, I was able to get in touch with the BridgeCrew Checkov team, and share with them what I was trying to accomplish, my challenges, and my findings. At the time of writing this article, there is still an issue with the XML output. But wait…
The Even Better News
While working through a similar Publish Test Results issue for another tool (the GitHub Super-Linter, which I will post about in a separate article), I was able to come up with a work-around!
Do you remember what the issue was, after resolving the ‘Warnings’ in the XML file? It was the last 2 lines in the file that were causing an issue. So, after some research, I realized that, all I have to do (at least, until the output is officially fixed), is to remove those last 2 lines. The rest of the XML file looked fine.
And so, I added a script after capturing the converted output into JUnitXML, and ran sed -i ‘$d’ to remove the 2 error-throwing lines. Now, I am not a Linux person, so let me break it down for anyone else in the same situation.
I Googled “sed command delete line” and came across this article: Unix Sed Command to Delete Lines in File – 15 Examples. In it, it explains (Example 2) that the command sed ‘$d’ file is used to remove the footer line in a file. The $ indicates the last line of a file. OK, now we’re getting somewhere.
At the end of Example 15 in the article, it stated:
Note: In all the above examples, the sed command prints the contents of the file on the unix or linux terminal by removing the lines. However the sed command does not remove the lines from the source file. To Remove the lines from the source file itself, use the -i option with sed command.
And so, by adding the -i along with the ‘$d’ in the command, I was able to successfully remove the last line in the XML file. But, since there were 2 lines that are causing the issue, I had to run it twice.
Pulling It All Together
All that was left is to execute the Publish Test Results task.
And now, since the XML file is formatted correctly, Azure Pipelines can read the results properly, and we get the following results:
Well, even though the general approach is similar to what I had to do for TFSec (as detailed in my Publishing TFSec Terraform Quality Checks to Azure DevOps Pipelines article), as you can see, there were a few additional bumps in the road.
I can also happily state that I am working with the BridgeCrew team, as we collectively look for a more permanent solution. But for now, at least the work-around is an option. With this work-around we can successfully publish our Checkov test results to the Azure Pipeline.
Bonus: Stay tuned for another similar article about the GitHub Super-Linter!
Update (now with less ‘-t’)
While writing this blog post, and working with the BridgeCrew team, they were able to discover that the command to run Checkov in a Docker container (the
docker run -t part) was causing the two additional lines in the XML output.
So what this means is, you don’t have to run the SED commands after receiving the converted XML output from Checkov, and the new/revised Docker command is this:
docker run --volume $(System.DefaultWorkingDirectory):/tf bridgecrew/checkov --directory /tf --output junitxml > $(System.DefaultWorkingDirectory)/Checkov-Report.xml
From there, you can publish the results by using the Publish Test Results task, as was previously mentioned!
Kudos to the great work by the BridgeCrew team, their transparency, and their efforts to help and contribute to the community!