Which PR to review first
On building a system to surface the most important pull request
It's a question our top users ask. Can Neat tell me which pull request to review first? Large software teams face a bottleneck in code review. More developers? More PRs, plus more overhead in sorting.
A large, open-source repository like Electron can see thousands of pull requests. Where do you start?
The principles in this article extend to issues and tickets. GitHub Issues, Linear, Jira, Trello, or a whiteboard. You name it. We encourage you to experiment to find what works for your team.
Product teams have mature ways of planning features. If you search "prioritization framework", you'll find lists of the best: RICE, Kano, Cost of Delay. It goes on. Consider each PR to be a feature (including bugfixes and changes). Why don't we, as software developers and tech leads, build such frameworks?
Let's summarize the top resources to help you decide. The popular kid is Intercom's RICE framework, where priority is the product of Reach, Impact, and Confidence, divided by Effort. The ICE, Kano, and Value vs Effort models are variations on RICE. A ratio of outputs to inputs. ICE: Impact times Confidence over Effort. Kano: closeness to "must-have" over time-to-implement. Value vs Effort: well, value over effort. Think of this class of frameworks as 2D, with varying degrees of subjectivity.
The RICE framework, Value vs Effort, and Kano model, three popular tools for prioritizing features, exhibit similarity.
At Neat, we started from the RICE framework, adapting as needs shifted. We grouped Reach and Impact into a single metric: how many users does this affect? A single user, 10 users, or 1000 users?
Next, we dropped the Confidence factor. We accept that our estimates are probably far off (and yours are too).
We're left with Priority as Impact over Effort. Let's call it the PIE framework.
Finally, we recognized how many factors are specific to pull requests. Top integrators look at criticality, size, urgency, complexity, and age to handle stacked PRs. Criticality fits under Impact. More users affected? More critical. Size and complexity fit under Effort.
From Automatically Prioritizing Pull Requests, by Erik van der Veen, 2015, TU Delft.
What about Urgency? Age is especially important to pull requests. The GitHub Stale bot, which flags then closes old PRs, powers over 12k teams. All else equal, let's consider an older PR to be more important.
So Priority is Impact over Effort and Age. Let's call it the PIEA framework (pronounced like Paella, "pie-AY-uh").
Priority is higher if more users are affected or the ticket has been sitting around. Priority is lower if more effort is required.
With this simple algorithm, let's hand it off to the computer. Project software is often opinionated, so we used a Notion kanban board. Tickets have slots for number of users affected, estimated time (an hour, an afternoon, or a week), and age. We then added a Priority property of type Formula. Here's our formula:
Ticket sorting algorithm written in Notion's special language. The tenth root of age (with dateBetween) is taken to reduce its importance. We came to this value by tuning.
Now, the easiest, oldest, most wide-reaching features surface. This opens our sprint planning time for more meaningful discussions.
Tickets sorted by Priority in a Notion table view.
This system is far from perfect. It's not immune to political or loudest-voice decisions. It's very opinionated but also very extensible.
In a sentence, review the oldest PRs that affect the most users in the least time. How do you handle your merge queue? At Neat, we use Neat to automate the first layer of PR priority for you. We're building the best way to stay on top of PRs and ship more.
Try Neat for yourself and let us know what you think.
$0 to use. Log in with GitHub.