Ask a professional development lead where the reviewer lists come from and the answer is usually an email. Partners are asked which associates they worked with over the past year, and they reply from memory, often weeks after the question was sent and months after the work.
The lists that come back are wrong in consistent directions. Recent matters are over-represented. Short assignments are missing. The associate who was pulled onto a difficult matter for six weeks and did not enjoy it is sometimes left off, not always deliberately. And the partner who is slowest to reply holds up the cycle for everyone in their group.
Meanwhile the firm already holds a complete, dated record of exactly who worked with whom. It is in the time entries.
What the time records actually contain
Every billed hour carries a timekeeper, a matter, a date and a narrative. Aggregate a year of that and you have each working relationship in the firm, with the hours attached and the period they ran over. For the purposes of designing an evaluation, that is a better source than recollection, because it was recorded at the time and it was recorded for a different purpose.
That last point matters. Time entries were not written to influence a review, which makes them harder to shade than an answer to a direct question about who should assess whom.
Getting the data out
Firms expect this to be the hard part, and it usually is not. What an evaluation needs from a billing system is narrow: timekeeper, matter, the supervising or responsible attorney, hours and a date range. It does not need narratives, rates or client names. That is a smaller and less sensitive extract than most people assume when the question is first raised.
There are two ways to supply it. A periodic export is enough for a firm running one cycle a year, and it is where most firms start. Where the data needs to stay current, SRA integrates with billing systems, either directly or through middleware, so the rater pools come off the live record rather than off a file somebody remembered to send.
Either route is a known quantity rather than a discovery exercise. The judgment is in how supervision relationships map onto the matter structure, which is a question about the firm rather than about the data.
Four things derived rater pools fix
Coverage becomes visible before fielding, not after. Run the hours and you can see which reviewees have enough raters to report against your threshold and which do not. A partner supervised by three associates will not support an individual report, and knowing that in week one is worth considerably more than discovering it in week six when the responses already exist and the threshold cannot be loosened.
Omissions stop being invisible. When a partner's suggested list leaves off the associate who logged three hundred hours on their matters, that absence is now a fact you can see rather than an oversight nobody noticed. You do not have to accuse anyone of anything. You add the name.
Exposure floors replace judgment calls. An associate who logged twelve hours with a partner over a year is not well placed to assess how that partner supervises. Set the floor in hours and apply it consistently, rather than deciding case by case and defending each decision separately.
Rater load can be distributed. If the data shows one associate is eligible to rate nine partners, you can see the survey fatigue coming and cap it before it drives the response rate down.
Where the operational time goes
In most cycles the longest delay is not instrument design or analysis. It is the wait for rater lists, followed by the reconciliation of lists that disagree with each other. Two partners name the same associate differently. One group replies in a spreadsheet, another in the body of an email. Somebody rebuilds all of it by hand, then sends it back for confirmation, and the confirmations arrive over another fortnight.
Deriving the pools from the records removes that whole stage. What remains is a review step, which is faster and more useful: a group head is shown a list that was generated rather than requested, and asked what is wrong with it. People are much quicker to correct a list than to produce one.
Why this shows up in compensation discussions
A rating gets contested on two grounds before anyone argues about the substance. Who answered, and how many of them.
Those are reasonable questions, and a process built on emailed lists answers them weakly. The honest version is that the instrument went to the people the group suggested, which invites the obvious follow-up about who was not suggested.
Derived pools answer both questions in advance. Nine of the eleven associates who logged more than fifty hours on a partner's matters responded, against a threshold agreed before the cycle opened. That is a different sentence to defend, and it moves the conversation onto what the feedback says rather than onto how it was gathered. For a firm using review data in compensation and progression decisions, that shift is most of the value.
What it does not fix
Hours are not supervision. A partner can bill substantial time to a matter without supervising anyone working on it, and a senior associate can run the day-to-day while the relationship partner barely appears. Derived pools need a supervision relationship, not co-presence on a matter number, which means the rules have to encode seniority and role rather than hours alone.
Non-billable work is invisible. Pro bono, recruiting, training, committee work and secondments generate real working relationships that leave little or no trace in the billing data. Any pool derived from time entries needs a manual add step, and the people most affected by its absence are often the ones carrying the most firm citizenship work.
And the data says nothing about quality. It tells you who is positioned to answer. What they say is the instrument's job.
A reasonable first step
Take one practice group and one cycle. Derive the rater pools from the matter and time records, then ask the group head for their list the way you always have. Compare them.
The difference between the two lists is the finding. In most firms it is larger than anyone expects, and it is concentrated in exactly the relationships an evaluation most needs to capture.
SRA designs upward review and 360-degree feedback programs for US law firms, and will check reviewer coverage against a firm's own matter data before an instrument is agreed. The methodology sets out how thresholds are fixed and what a report can carry.