One thing I found interesting was the idea of doing code-review pre-checkin, and your reasoning makes sense. Unfortunately I think it would be a hard-sell here. We’re slowly adopting an “agile” programming style, which means constantly checking in code which is tested on an almost daily basis by people within the company. Our software lead is adamant that all developers review all code, so requiring that step be done pre-checkin would hold things up an unacceptable amount I think. The idea – at present – is to complete an agile “story”, have it checked in and tested by Q.A., and at the same time do a code review.

That being said, this is all new to us. Just like your “first attempts”, we may give it a go and find it just doesn’t work at all. But being able to see what your company have done and how you’ve adapted over time is greatly helpful! Thanks!

Cheers,
-Mark

So, moving to a pre-commit review system is a tough sell, especially as the team size grows. My current studio is pretty big and I wouldn’t even consider rolling it out to the full team. I’d propose it just on a very small scale, just within my immediate group, then grow it from there with some evangelism.

But I don’t think we’ll ever really be able to let go of the frustration with “I know this works and I want to check it in now so Bob can use it” yet having to do a review before checkin. I think this frustration is actually a good thing, though. It keeps the machine moving with little nags.

Regarding granularity – we reviewed every changelist, required before checking in. Sometimes with emergency fixes someone would check in and we’d do a post-review while the build was running or whatever but that was rare. I occasionally ran an audit to make sure of this (just a script that made sure every changelist description had a review link in it).

Over time I tried to encourage people to break up their changelists into functional, atomic packets. Instead of a giant 30-file feature change, a few smaller ones broken down into core lib changes, service changes, and feature change is better. It’s better for post-hoc diagnosis of problems down the road, and it’s better for review discussion.

Sometimes we’d still have a giant changelist but have it broken into several separate reviews just for reviewer convenience. While that checkin to P4 must be atomic, so you do not break the build, the review service doesn’t have that requirement. So I would often break out the reviews I created into “trivial crap you can probably skim”, “ok here’s the real change”, and (sometimes) “pay careful attention to this stuff”.

We could end up with reviews of any size, though they were typically in the <5 file range if I remember right. Sometimes it would be a one-line change, and sometimes it would be a new system created over a week or two. Obviously the stage in development would affect this.

As we started our next project, I experimented a bit with no-checkin reviews to get early feedback on new systems. So: just create a review with the headers and some key cpp files, and send it around to the affected parties to see what they think. So, using it as a discussion tool, with no intention of checking anything in.

But when it comes to really big monolithic checkins of “here’s my new system, all done!” that is going to happen on occasion, then that’s only going to work by starting with a sit-down review for the lay of the land, leading to offline for the more detailed review work, with followup as you go, either in IM or in person. Or by writing up questions in comments and having a dialogue in there.

How did this affect the process? Well, obviously it slowed it down, but again I think that’s a good thing. Nobody should be in a hurry to check in a massive changelist with confidence. We varied a lot in our return time on reviews, but typically it was no more than half a day, probably two days at worst if the change was huge, had a lot of comments, and the reviewers were slammed with their own work. There were problems on occasion but overall the timing worked out well enough. I also think people have a responsibility to be able to work on multiple things at once, using multiple source control clients. Engineers shouldn’t work in serial. There’s just too much we always have to do. So nobody should ever be blocked. And I don’t think I saw it happen at all at my last gig.

Performing lots of small reviews was really no more overhead than a larger batched review. Just a couple more clicks. I actually liked smaller reviews better because they more clearly separated the changes from each other. And everyone gets more of a sense of accomplishment as they check off reviews. Feels like real forward progress to check those boxes. Though obviously I wouldn’t use any of these counts for any kind of employee performance metrics.

And finally, regarding code review guidelines of time spent, we didn’t do anything like this. Our informal standard was just “do a good job”. We clearly had different review styles. I got a reputation for spending a lot more time on tiny details than others, for example. That’s why we want to spread the load around, and why I made the tool choose someone at random by default on review creation. Everyone’s going to look for and catch different things, and while an individual review won’t get total coverage, over time we will get it, due to the overlap among all the reviewers.

We do face to face check in ‘buddying’ (all changes in a CL must be described/explained to a fellow programmer prior to a P4 submission) and it’s sort of a half-way house between no / a full review. It’s effective for picking up simple mistakes/omissions/oversights but that’s often where it ends.

When sitting down and reviewing a change list, folk usually point out things like:

* “you’ve duplicated functionality that already exists in one of our other libs”

* “extract this into a sub-method and get rid of the duplicate logic”

* “Change this magic number to a well named const”

These typically come under the umbrella of code cleanliness and common conventions. If someone stacks up a lot of changes I am more inclined to read their code more thoroughly before putting my name to it, but it’s tough to buddy each and every change list with a high level of scrutiny, especially when a deep discussion can take hours and energy levels start flagging.

Some people are very good at being stringent in buddying situations (I’m always grateful when someone ‘calls bullshit’ on my stuff), whereas others just want to quickly muddle through it and get on with their work.

Personality types come into play; for some, the to and fro’ing of buddying feels more like a sparring session than a useful exercise and it can become a style/opinion stalemate. For those reason, we usually keep the buddy process pretty light — I don’t think scaling it up any more would work.

When reviewing is done separately (or ‘offline’, whatever you want to call it) with multiple reviewers, it gives a good mix of programmers (from different groups, abilities, specialities etc.) time to think (I view this as ‘catching the stuff the buddy missed’). It also helps the less vocal (or more vocal ) members of the group participate without butting heads mano-a-mano.

Formal code reviews (which ‘offline’ reviews most closely resemble) also have a lot of published studies backing them up. Numerous studies have shown that code reviews are more effective than any of manual/unit/integration/functional testing for finding defects. There is also a strong correlation between lines of code read per hour and the number of defects found. If you read too quickly, you find fewer bugs. If you read for a small amount of time, you find less bugs than if you read for a larger amount. If you read the code for too long, you reach a point of diminishing returns.

In my opinion, you cannot realistically review code on someone’s screen while simultaneously talking to them and find defects in the same way — everyone moves at a different pace and the atmosphere is totally different. In a one to one discussion, it’s easier to derail the discussion or become fixated with specific aspects.

My only concern with our review system is that it happens after the changes go in, but I’m not sure that we can move to a thorough pre-checkin review. As Scott said, once it’s in, it’s in. Prevention is better than the cure, and once code is checked in and used, it hardens much more quickly and sucks up a lot more time than the writing phase ever consumed. E.g. someone ninja’ed some rebadged legacy code into a common namespace (probably a bad merge or a misunderstanding) and, before it was picked up, it’d already hit our main branch and been used by various people! Small change, lots of knock-on effects.

Out of interest, what sort of granularity were the change lists you guys were reviewing? Are we talking daily change sets containing a handful of files with minor changes, or entire features that took a week to create?

If you had wildly varying sizes of change lists to check over, how did you find it affected the process? E.g. did folk get blocked while waiting to check in an important (large-ish) feature for a day or two while key reviewers were busy? If the changelists were small and frequent, did the overhead of performing numerous small reviews cause any hassle?

Finally, did you have any code review guidelines in terms of how long you should spend on x lines of code, or how to pay more or less attention to a change list?

Sorry for the long reply, but this post piqued my interest.

Cheers,
-Mark

Limiting to one reviewer is a problem. We frequently include more than that on reviews. Getting that to work in person is just too hard.

Cliques can’t be legislated away. People are people and they’re going to do what is most convenient. Yeah, we’re all professionals, but when you come right down to it, you can’t make rules to govern social relationships. In addition, I just can’t get past the problem of synchronizing schedules and mental spaces. Only by using a tool and permitting asynchronous, offline reviews can you run the “80% case” (mostly simple reviews) efficiently.

We of course use face-to-face reviews for the big stuff. Though we still use a review to at least track the questions that came up in order to do the face-to-face review.

As for documentation, “writing a simple summary” is just not going to work, nor will it be anywhere near complete. Engineers aren’t going to do that reliably or consistently. It will be the first thing to fall away when we get busy and never get picked up again. The documentation must be a core part of the process or it will just feel like a waste of time. Which means it will get done poorly and incompletely. Having it all in the tool to begin with guarantees that you have 100% of the comments documented.

I would think that a code review face to face, 2 engineers staring at the same screen talking to each other would be better simply because of the face to face communication going on.

If it was because cliques were forming like your wrote about in the post, then could you not just tell the engineer to make sure dont’ always grab the same person for the review?

The advantage I see in doing the code review offline is that the review is documented and can be referred back again in future. But this could be addressed by writing a simple summary for the comments in the change list before p4 check in.

We’ve practiced 2 engineers to 1 screen code review I find the face to face communication during the review works really well – bugs could be found quickly, small discussions, knowledge transfer and mentoring works well in this situation.