Well, this has been a bit of a rabbit hole. I published Part 1 here and thought I would quickly smash out the rest of the posts around it. Alas I got carried away and managed to get an MVP that is a bit rough round the edges but seems to function well. So without further ado here is part 2 - where I plan to detail using the persistent top level flags in conjunction with viper and how I approached the scanning of Terraform files.
Flags Precedence
I wanted to be able to be very flexible with flags in my application and have multiple ways to define them to fit the various use cases. The flow should be:
- Load from config
- Overridden by flag
- Default value used
An example of this would be using a verbose or -v flag. I want to be able to populate this to all commands so set this at the top level. I might then ignore it in my local runs of the command but when I run the cli in an automated fashion I might want to set this value to true so that as much detail is available as possible to the pipeline run.
The code currently looks like this:
package cmd
import (
"fmt"
"os"
"calfinn.io/p45bot/pkg/opts"
"github.com/spf13/cobra"
"github.com/spf13/viper"
)
//Define flag variables here
var cfgFile string
var DryRun bool
var Manifest string
var Scanpath string
var Output string
// rootCmd represents the base command when called without any subcommands
var rootCmd = &cobra.Command{
Use: "p45bot",
}
func Execute() {
err := rootCmd.Execute()
if err != nil {
os.Exit(1)
}
}
func init() {
cobra.OnInitialize(initConfig)
rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.p45bot.json)")
rootCmd.PersistentFlags().BoolVarP(&Verbose, "verbose", "v", false, "verbose output")
rootCmd.PersistentFlags().BoolVarP(&DryRun, "dryrun", "d", false, "dryrun only")
rootCmd.PersistentFlags().StringVarP(&Manifest, "manifest", "m", "./manifests/example.json", "Path to manifest file")
rootCmd.PersistentFlags().StringVarP(&Scanpath, "scanpath", "s", "./", "Path to scan")
rootCmd.PersistentFlags().StringVarP(&Output, "output", "o", "", "Output type for commands")
viper.BindPFlag("verbose", rootCmd.PersistentFlags().Lookup("verbose"))
viper.BindPFlag("manifest", rootCmd.PersistentFlags().Lookup("manifest"))
viper.BindPFlag("dryrun", rootCmd.PersistentFlags().Lookup("dryrun"))
viper.BindPFlag("scanpath", rootCmd.PersistentFlags().Lookup("scanpath"))
viper.BindPFlag("output", rootCmd.PersistentFlags().Lookup("output"))
}
func initConfig() {
if cfgFile != "" {
// Use config file from the flag.
viper.SetConfigFile(cfgFile)
} else {
// Find home directory.
home, err := os.Getwd()
cobra.CheckErr(err)
// Search config in home directory with name ".p45bot" (without extension).
viper.AddConfigPath(home)
viper.SetConfigType("json")
viper.SetConfigName(".p45bot")
}
viper.AutomaticEnv() // read in environment variables that
viper.BindEnv("directoryconfig.clientsecret", "AZ_SP_SECRET")
// If a config file is found, read it in.
if err := viper.ReadInConfig(); err == nil {
fmt.Fprintln(os.Stderr, "Using config file:", viper.ConfigFileUsed())
}
fmt.Println("Using config file:", viper.ConfigFileUsed())
}
So most of this is fairly standard out of the box config. I have my variables defined at the top of my root.go
//Define flag variables here
var cfgFile string
var DryRun bool
var Manifest string
var Scanpath string
var Output string
var Verbose bool
I can then add the persistent flags in to the init function like so:
rootCmd.PersistentFlags().BoolVarP(&Verbose, "verbose", "v", false, "verbose output")
This line give me the functionality on the command line to run the command like so:
.\p45bot.exe validate
Using config file: C:\Users\tim.DESKTOP\git\p45bot\p45Bot\.p45bot
validate called
Verbose Value is: false
Or call it with the verbose flag (either using the shorthand -v or longhand --verbose):
.\p45bot.exe validate -v
Using config file: C:\Users\tim.DESKTOP\git\p45bot\p45Bot\.p45bot
validate called
Verbose Value is: true
All working well. The magic comes though with the following line in the init function that binds the flag and viper together:
viper.BindPFlag("verbose", rootCmd.PersistentFlags().Lookup("verbose"))
I can then set the value in a different config file and call the command:
*/ CONFIG FILE @C:\Users\tim.DESKTOP\git\p45bot\p45Bot\p45bot2.json
{
"verbose": true,
"manifest": "./manifests/budgetOwner.json",
"scanpath": "C:\\Users\\tim.DESKTOP\\git\\p45bot\\DummyOrg",
"directorysource": "azuread",
"directoryconfig": {
"tenantid": "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"clientid": "xxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx",
"clientsecret": ""
}
}
/*
.\p45bot.exe validate --config "C:\Users\tim.DESKTOP\git\p45bot\p45Bot\p45bot2.json"
Using config file: C:\Users\tim.DESKTOP-NE75G19\git\p45bot\p45Bot\p45bot2.json
validate called
Verbose Value is: true
Overall I was happy with this layout until I started writing the various application logic functions. I was finding a lot of extra code having to be presented to handle passing the flag from the root in to the various sub-functions. I then realised I could remove the variable from the root.go file and define it in its own package.
THIS:
rootCmd.PersistentFlags().BoolVarP(&Verbose, "verbose", "v", false, "verbose output")
BECAME:
rootCmd.PersistentFlags().BoolVarP(&opts.Verbose, "verbose", "v", false, "verbose output")
With the contents of opts.Verbose being:
package opts
var Verbose bool
// GetVerbose returns the value of the Verbose flag
func GetVerbose() bool {
return Verbose
}
I could now access the flag setting within any function importing the opts package like so:
if opts.GetVerbose() {
fmt.Printf("Filename is excluded - %v (input) and %v (excluded)\n", fName, ex)
}
And I now longer had to handle passing it in to the calling function.
Other flags in the PersistentFlags set could then follow the same pattern and allow different use cases to be handled for the cli and more functions can be added to opts to define checks on DryRun settings for example.
Scanning Terraform Files
Having the flags in place and the ability to import config files meant I could now start tackling how to scan the files I would be shovelling through the CLI. The aim was to import a manifest file that defines what file types to look for, what strings within the config to find and what files to exclude. This should hopefully allow the tool to be used in different ways later on. viper allows for a new config file to be imported really easily by just standing up a new instance of viper:
y := viper.New()
y.SetConfigFile(viper.GetString("manifest"))
Here I have stood up a new viper and set the config file to be that I defined on the cli/config. The manifest would have all the information I need now accessible as a viper object:
{
//File types to find in the initial run through the directory
"fileType": ".tf",
//filenames to exclude. unless you are a masochist then you most likely have you TF files split out from main.tf. Here I am excluding variables.tf and terraform-configuration.tf which is an internal setup of provider and core config.
"fileNameExclusions": [
"terraform-configuration",
"variables"
],
//The strings to search for. These are essentially terraform configuration keys.
"searchStrings": [
"budgetAlerts",
"projectOwner"
]
}
The CLI then passes over to the searcher package which contains the logic for finding files and strings etc.
r := searcher.SearchForFiles(
viper.GetString("scanpath"),
y.GetString("fileType"),
y.GetStringSlice("fileNameExclusions"))
This finds all the files on the current filesystem in the scan directory, filters them to remove the exclusions and then returns some stats and the files that might contain content to string search.
The search for file types is essentially an fs.WalkDir. I settled on this over filepath.WalkDir as I was able to pass in a filesystem to fs which made setting up a test filesystem with "testing/fstest" that much easier. The ffilterFile function is then just an iteration over the returned list of files to filter out those that match the fileNameExclusions:
func filterFiles(f, ext string, excl []string) int {
//return value and where we keep track of hits against the exclusions
i := 0
//loop over the exclusions we have form the manifest file
for _, e := range excl {
//munging the exclusion with the file type extension
ex := e + ext
//stripping the full file path back to just the last piece - i.e. the filename with extension so we can compare
fName := filepath.Base(f)
if strings.EqualFold(fName, ex) {
if opts.GetVerbose() {
fmt.Printf("Filename is excluded - %v (input) and %v (excluded)\n", fName, ex)
}
i++
} else {
if opts.GetVerbose() {
fmt.Printf("Filename not in exclusion list - %v (input) and %v (excluded) \n", fName, ex)
}
}
}
return i
}
The idea here is that if a matching exclusion is found it adds 1. The expectation is that 0 is returned for a valid filename that is not excluded.
Once we have our list of file names we then begin a search for strings within them - essentially the identifiers that HCL uses within a block. I wanted to limit the thrashing of files so I opted to take an array of search strings and then construct regex for each of them. Looping over each file and scanning it we could thrown multiple regex terms in and get the results out without having to open and close multiple times.
func SearchString(root string, files, search []string) (SearchStringResults, error) {
r := SearchStringResults{}
for _, file := range files {
targetFile := filepath.Join(root, file)
fmt.Println("Processing", targetFile)
openFile, err := os.Open(targetFile)
if err != nil {
fmt.Println("Error opening file:", err)
return nil, err
}
defer openFile.Close()
patterns := make(map[string]*regexp.Regexp)
for _, target := range search {
pattern := regexp.MustCompile(regexp.QuoteMeta(target) + `\s*=\s*\[(.*?)\]`)
patterns[target] = pattern
}
scanner := bufio.NewScanner(openFile)
lineNumber := 1
replacer := strings.NewReplacer(`"`, "", ` `, "")
for scanner.Scan() {
line := scanner.Text()
// Check each pattern and find matches
for target, pattern := range patterns {
matches := pattern.FindStringSubmatch(line)
if len(matches) > 1 {
textBetweenBrackets := matches[1]
for _, f := range strings.Split(textBetweenBrackets, ",") {
var t SearchStringResult
t.FileName = targetFile
t.SearchString = string(target)
t.Upn = replacer.Replace(f)
t.LineNumber = lineNumber
r = append(r, t)
}
if opts.GetVerbose() {
fmt.Printf("For '%s', text between brackets: %s\n", target, textBetweenBrackets)
}
} else if len(matches) == 0 {
if strings.Contains(line, target) {
lineSplit := strings.Split(line, "=")
upn := replacer.Replace(lineSplit[1])
if !strings.Contains(upn, "[") {
var t SearchStringResult
t.FileName = targetFile
t.SearchString = string(target)
t.Upn = upn
t.LineNumber = lineNumber
r = append(r, t)
}
}
}
}
lineNumber++
}
if err := scanner.Err(); err != nil {
fmt.Println("Error reading file:", err)
}
}
return r, nil
}
Within a resource or data block we can have a myriad of expressions defined but these boil down to a single type (integer or string), an array of those types or a link to another piece of terraform configuration (a variable being passed in or remote state lookup). So the above function anchors itself to the key (search string) and then uses a regular expression - \s*=\s*\[(.*?)\] - to find all the configuration between the [] but after the = sign:
module "my_awesome_module" {
source = "../../../../tf_module_subscription"
date = var.date
ado_project = "CalFinn"
subscription_name = "CalFinnIO"
subscription_id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
budgetAlerts = [
"Ben.Wyatt@tailspin.com",
"Andy.Dufresne@tailspin.com",
"oscar.wallace@tailspin.com"
]
projectOwner = "emperor.palpatine@calfinn.io"
available_to_other_tenants = false
}
So the above terraform module would be searched and the following returned:
"Ben.Wyatt@tailspin.com",
"Andy.Dufresne@tailspin.com",
"oscar.wallace@tailspin.com"
We then do some munging to split out the string to unique elements (in this case splitting on ,) and then format the result to remove whitespace and quotes.
Face Palm Moment
I spent far, far too long in the process of writing this trying to troubleshoot an error I had with flags. I personally blame my 2 year old who doesn't understand the need to be covered by a duvet to keep warm at night so wakes us up instead. The error I encountered was down to trying to pass in the verbose flag from the config file. I knew that viper was reading the config as I was able to pass in scan as a command and the details in the config file were read in correctly and meaningful data was output. Alas, try as I might I could not get a different config file, where I had set "verbose": true to function - all I got was the default flag values output.
I went round the houses trying to pull apart why it did not work before finally realising the error:
func initConfig() {
if cfgFile != "" {
// Use config file from the flag.
viper.SetConfigFile(cfgFile)
} else {
// Find home directory.
home, err := os.Getwd()
cobra.CheckErr(err)
// Search config in home directory with name ".p45bot" (without extension).
viper.AddConfigPath(home)
viper.SetConfigType("json")
viper.SetConfigName(".p45bot")
}
With no --config flag set the iniConfig() function drops though the first if statement and starts pulling config from a default location. The key bit for me was viper.SetConfigType("json"). In my rush to generate a second config file I had merely copied the existing config .p45bot to .p45bot2 and though nothing of it. Of course viper then had no concept of what type of config file this was. It would complete, process the file and even print out which file it was using but the calls to viper.GetBool("verbose") never returned the value in the config file.
The fix was merely a case of adding .json to the end of my new config file. Super frustrating to have wasted so much time on it but tired brains definitely do not process difficult situations very well. I hope someone eventually reads this and saves themselves a trawl through SO or github issues.
Next Steps
Am very aware that this has become quite a lengthy piece so if you are reading this - congrats and thank you. Once we have our list of files and the elements within those that we want to validate we can start to work on looking them up in AzureAD.