When Code Guarantees Are Checked

Golang

As with many programmers, as my career has progressed I’ve come to appreciate the utility of “compile time checking”. I don’t ever want to go back to a language where it is hard to verify constraints are checked at compile time.

Followers of this blog may know that I do a lot of programming in Go, and find it hard to square that attitude with the use of Go, which has relatively weak compile-time guarantees.

Part of that is that compile-time checking is in my opinion a very 80/20 situation. The vast bulk of the value of a compile time check is accomplished by the simplest of manifest typing, assuring that the thing coming in to a function is guaranteed to be of a certain type. Combined with all the things you can do with even the simplest manifest typing, this is generally the vast bulk of what you need, and additional checks tend to have diminishing returns, both in terms of becoming increasingly expensive to implement and the utility of those increasingly expensive implementations diminishing. There does come a point where the expense of putting in a particular guardrail exceeds the expected benefit that it will ever provide to you.

That is not to say Go is necessarily at the optimum point. I do often wish for a bit more. However I do think a lot of programmers demand as a bare minimum a feature set that is actually quite a bit past the optimum, somewhat overestimating the benefits and badly underestimating the costs. (At least, pre-AI. Evaluation of this post-AI is in progress and I expect to take some years yet.)

But another aspect of my comfort with not having the most compile-time guarantees is that there is an even richer model for thinking about when guarantees are checked. It isn’t just “runtime versus compile time”. There is a spectrum. I see at least:

  1. No checks at all: The degenerate case, where presumably some bug will happen if anything at all gets through. We don’t want to be here, but it’s useful to list here as one end of the spectrum.
  2. Edit-time checked: Things that your IDE or editor are checking as you type, without even having to compile the code.
  3. Run-time checked: Something has gone wrong, but at least the code at runtime noticed and did something sensible with it, even if the only sensible option is “scream and die”. This is better than nothing.
  4. Compile-time checking: What we all know and love, except for those who don’t yet.
  5. Commit-time checking: What may pass the compiler but can fail out a commit due to a pre-commit hook failing. This one is underutilized, I think. Every programming repo should have a pre-commit hook of some type. And I draw a very strong distinction between this and:
  6. Continuous-Integration-time checking: What is done after some bit of code is pushed. While CI should incorporate all of the compile-time and commit-time checks, CI also generally has a larger time and computation budget than commit-time checking. Commit-time should really run in seconds. CI should run in minutes, but if necessary it can run into hours. This is good for extensive integration testing that may be impossible on a developer’s machine or environment, or that simply takes too long, like extensive browser-based testing.
  7. Traditional QA deployment: A full deployment of the code into a QA environment, possibly integrated with other QA environments within the organization. Humans may run manual or creative tests on this.
  8. Customer-time checking: The other extreme, and the other worst-case scenario, when a customer finds the problem rather than any of the rest of these checks. Sometimes this may also take the form of a staged/feature-flagged rollout in the hope that maybe not all the customers will find it.

Obviously, I have not come up with any of these ideas for the first time. What I think may be novel for many readers is that you should consciously consider this as a continuum, and even more importantly, consciously consider what checks belong where rather than simply knee-jerking about some of these being Platonically “good” or “bad”.

I also make no claims that this is the final list. In the real world, the closer you look, the more gradations you will find. Certain languages or environments may have steps not included above; for instance a language with macros may have a distinction between “before” and “after” macro application you could wedge in there, and certainly there are QA deployments in the world with more staged releases than I describe above precisely because some local distinction has some utility, such as perhaps rolling out “backend” and “frontend” updates to QA separately.

As in all things engineering, you must always be thinking costs and benefits. There can be things that are very expensive, even infinitely expensive (i.e., impossible) to put into compile time checking that may be very easy and cheap to add to a pre-commit hook. Having a rich and conscious understanding of all of these options gives you a richer menu of cost/benefit tradeoffs to choose from than being focused on a much smaller set of options. The important thing is finding the problem as cheaply as possible, preferably before the customer finds it, not that it happen at a particular stage.

I encounter people online that I think have gotten that taste of the power of compile-time checking after years of living without it for some reason (dynamic languages or underutilization of the features), and with some quite valid justification want to do more at compile-time. No problem with that.

The problem is that I think they get a monomania and end up overly focused on that one thing, trying to jam things into it that may fit better in other parts of the continuum.

In particular I think “commit-time checking” is generally undervalued, squeezed between compile-time and the CI system. There is a lot of things that can be effectively done at commit time, such as, linting, the fast part of the test suite that may not include integration testing, other style verifications like spell-checking on any new .md files commited, and so forth, that are very valuable to have as quick feedback before it even gets to the CI. Even the fastest CI system still takes some time before it finishes providing feedback on the commit and it is extremely easy for that to bloat into a multi-minute or even multi-hour process. You should still harvest the benefits of fast feedback as much as possible for commit-time checking.

One of the impacts on my thinking this way is that more of my programs are growing an official -verify command-line flag that causes the code to run through as much of the startup process as it can without actually starting the service. Maybe it does some run-time type checks of some internal code. Maybe it does some checking that can only be done at run-time with regard to my serialization mechanisms to make sure they have certain additional constraints that my serialization tool can’t enforce on its own. All sorts of interesting checks can be done at runtime that may not be possible at compile time, and that can easily be made part of a commit-time check.

Commit-time checks are almost as good as compile-time checks in terms of their cost/benefits but have a lot more available power. I suggest more people use them.

Having a distinct concept of “commit time” versus “CI time” also helps keep the commit-time stuff short and sweet. I know I’ve seen several projects that started with commit-time checks, but they bloated up and the project simply gave up commit-time entirely and moved it into CI, leaving no commit-time checks behind. That is, I believe, a natural consequence of not clearly realizing those are two separate things and should be treated as such.

I would still rather keep the fast things in a commit check and move the slow things into CI, and adjust the balance over time as needed, rather than give up on my commit-time feedback. CI-time-checks naturally expand to become slower and slower. It isn’t much at all for a CI-time-check to expand into the minutes, long past the point where a programmer has lost the context of the change and has to pay a context switch to bring it back in. Using commit-time checking intelligently, even when you have all the other checks, can still be if nothing else a more pleasing way to work.

Using the full range of options intelligently will, if nothing else, make development less annoying.