The widely adopted SCM tools we use today, Github and Gitlab, are built on the dated architecture and design of git. I recently happened upon this thread on HackerNews that talked about a lot of the pains we’ve spoken about in our article On Git and Cognitive Load. These aren’t the only issues with Git, many of them are just really simple missing features that are no longer aligned to our current engineering practices, take this post on Folder (Non)Tracking, and today I want to talk about common workarounds (AKA hacks) for secure storage with today’s SCMs.
When we build applications, from mobile to desktop, across industries we often have many configurations in order to connect our applications to external services. Some of these services are required at runtime (e.g. ad networks, configuration servers, analytics), and others at build time (e.g. code signing). We’ll dive into some of the challenges third-party services and imports introduce into our development processes.
Common Challenges with Securing Credentials in SCMs
With security taking center stage over the past few years, from the code to the build processes - shift left to DevSecOps, many best practices have been formalized for handling the credentials to our third-party services and supply chain. For example, for good security reasons, credentials to the services described above are usually not stored inside the SCM. What is usually done is that each developer has their own local ‘.env’ files that store their environment variables for quick access in development processes. These files are set to be ignored from commit processes by the SCM so that they do not propagate to your repository and remain there for all of posterity.
However, in order for these services to run properly, these credentials do need to be accessible to the running service in production or at build time, and so the way this works is that the CI/CD platform of choice then has its own copy of these secure variables. These are then either injected into the code at build time or used at build time to sign the app itself.
While this is important for defending our systems, and ensuring our secrets and environment variables are not exposed to would-be hackers, this does add a lot of complexity to the management and maintenance of our environments and systems, and can be tedious to keep up with.
This to us is yet another area that git and similar SCMs have not kept up pace with modern engineering and the attack landscape. With the amount of data and services running in the cloud, that require remote access, and are interdependent as chained services - the communication between these services is critical, while security cannot be compromised either.
There are products who have tried to address this known challenge, such as Fastlane’s Match, that stores code signing certificates for iOS in a dedicated git repo. This makes it possible to use just one setup for both your local development environment and for the CI/CD to access the same credentials. The drawback is the git repo itself. Over time, the files it contains grow so big, that the cloning starts to take so long that it becomes the most significant part of the build time.
This isn’t scalable, and ultimately is a huge time sink on our most valuable asset - our developers’ time. Recently Match started supporting alternative storage - Amazon S3 and Google Cloud, which is great progress. No point in keeping revoked certificates in version control, taking up space and increasing clone time, right? However, it just replaces one problem with another of the same type. Now you need to store access credentials to S3 / GC. The documentation reads:
When running fastlane match init the first time, the setup process will give you the option to create your gc_keys.json file. This file contains the authentication credentials needed to access your Google Cloud storage bucket. Make sure to keep that file secret and never add it to version control. We recommend adding gc_keys.json to your .gitignore
Back to square one, then. Each developer and the CI/CD need their own credentials file, so additional setup needs to be done between cloning the repo and being able to run or distribute the code. The same thing applies to using a git repo to store the certificates, because it’s a different repo than the one that contains the codebase, and usually requires another SSH key for access. Oh, and I forgot to mention you also need a passphrase to decrypt the Match repo, which is another secret to worry about.
This is a good resource to reference on code signing, and the complexity involved. I’ll also add that currently Match is the only solution trying to overcome this challenge without compromising your credentials to the Apple Developer Portal - so it’s a good effort on their part, and a useful utility, despite some of its drawbacks.
Scaling Secret Management - Truly as Code
For this to truly be scalable, these important variables need to be stored inside the native workflow and context, and be easily maintained. What if you could just store all this data inside your regular repo, but somehow mark the relevant files as sensitive and/or volatile? What’s more, what if we could have them secured and replaced with different versions at will (per environment or per user)?
A post about security and DevOps wouldn’t be complete without some reference to GitOps. So let’s talk about roles and policies, and how all this could come together. We’ve established that it would be really great to have a “secrets” folder inside our regular ol’ repo. Once that happens, we would be able to leverage this repo, manage and apply policies like we would any other repository in our organization. We would be able to define who has access to these folders, and make that possible only for certain roles, and in our favorite way–as code (e.g. with separation for dev and prod).
From a cost optimization perspective (as we head into a bear market), this would not only save us the setup and troubleshooting time, but also the added cost of additional tools in our stack just for secret management.
From Git to Developer Experience - It is Possible
When it comes to git, it’s clear that it wasn’t built with developer experience in mind. However, this could also be a byproduct of the times. Perhaps that’s what developer experience looked like historically with little external services, no CI/CD, less security in mind, and the features were sufficient then.
With the constant evolution in engineering processes - from the speed and velocity that code needs to be delivered to production, the many and multiple services that each application is broken down into, the evolving attack landscape, and even our processes and culture with global, distributed teams—the way we work together and collaborate on code is becoming a core piece of our developer experience, without compromising security.
At Diversion, as a team that dogfoods its own product, and wants to improve our very own developer experience––we’re trying to find the many points of friction and ruthlessly remove these unnecessary obstacles from our development processes. What else is bugging you about git, or other tools you use for development? Let us know in the comments.
