Key takeaways
- The number that matters is responses received per reported category, not pool size.
- A rater pool of eight at a 70% response rate leaves six responses, which becomes three and three once split by seniority.
- Low response rates make small groups riskier, not safer. Fewer responses mean fewer people to consider.
- Suppression thresholds of three appear in practice and are too low for legal practice groups.
- Four options exist when a group falls below threshold, and widening the rater pool is usually the right one.
A 360-degree review stops being anonymous when a reviewee can infer who said what from the number of people who could have said it. There is no single group size at which this happens. It depends on rater pool size, response rate and reporting granularity. Survey Research Associates (SRA) has designed these programs for US law firms since 1991.
Why is this harder in law firms than in corporates?
Because legal practice groups are small and reviewees can reconstruct the rater pool from memory.
A corporate department of forty running a 360 has a pool where attribution is genuinely hard. A twelve-person tax group does not. A four-person practice does not require effort.
Two features make it worse in a law firm:
- Reviewees know who staffed which matters, so they can rebuild the pool without access to any data
- The reviewee often controls the reviewer's work allocation, so the cost of being identified is high enough that people assume the worst
What is the actual arithmetic?
Three variables decide whether a rater group is safe to report: pool size, response rate and reporting granularity. The number that matters is responses received per reported category.
A peer group of eight sounds adequate until a 50% response rate leaves four, and a report splitting those by seniority leaves two.
Does a low response rate make a small group safer?
No. It makes it riskier, which is the opposite of what most people assume.
Fewer responses mean fewer people the reviewee needs to consider, and the identity of non-responders is often guessable too. In a five-person group with three responses, a reviewee can frequently work out both who answered and who did not.
This is why thresholds must apply to responses received, not to pool size. A group that looked safe at design becomes unsafe at fieldwork, and the only way to catch it is to test before generating reports.
What is a safe minimum number of responses?
High enough that the reviewee cannot infer authorship from the count. Thresholds of three appear in practice, reported by associates describing third-party programs at large firms. Three is below the level at which a partner in a small practice group can work out who responded.
The important property is that the number is fixed in advance and not adjusted to produce more reportable data. A threshold that moves is not a threshold.
Not sure whether your practice groups clear the threshold? Our attorney 360-degree feedback service tests group sizes against reporting thresholds at design, before fieldwork, and tells you where a 360 is the wrong instrument.
What can we do when a group is too small?
Four options, in order of how often each is the right answer.
- Widen the rater pool. Include people from adjacent practice groups who have genuinely worked with the reviewee. Usually the best answer, and it often produces a better review, since a pool confined to one small group was never a full picture.
- Collapse the categories. Report peers and associates as one group. You lose the comparison between rater types and keep the anonymity. For a small group that is the correct trade.
- Suppress and say so. Issue the report with the group marked below threshold. Visible suppression demonstrates to future respondents that the threshold is real.
- Use a different instrument. An upward review across the firm, where the reviewer pool is larger, may deliver most of the value with none of the exposure.
When is a 360 the wrong instrument entirely?
Three situations, and recognizing them early saves a wasted cycle.
How should this be handled at the design stage?
Test every rater group against your threshold before fieldwork, modeled at your expected response rate rather than at full participation. Full participation will not happen.
Then tell people the answer. Reviewees and raters told in advance which categories will and will not be reported respond more honestly, because the rule is visible rather than discovered afterward.
Frequently asked questions
Can we report a group of three with a warning? We advise against it. A warning does not stop a reviewee working out who responded, and doing it once teaches the firm that the threshold is negotiable.
Does anonymizing comments solve the small group problem? It helps and is not sufficient. A comment can identify its author through a matter reference or a turn of phrase, which is why comments need review, not just name removal.
Our firm is small overall. Can we run 360s at all? Often yes, with a wider pool and collapsed categories. The design work is heavier at smaller firms and worth doing rather than skipping.
Should we tell people which groups will be suppressed? Yes. Visible suppression builds trust. Silent suppression looks like missing data.
How is this different from an upward review? An upward review runs in one direction across the whole partnership, so the reviewer pool is larger and the threshold problem is smaller. A 360 covers every direction for one attorney.
360 feedback designed for the size of your practice groups
SRA has designed confidential 360-degree feedback for US law firms since 1991, including rater pool testing against anonymity thresholds before fieldwork begins.
Contact us · Survey Research Associates, Inc. · 30 Wall Street, 8th Floor, New York, NY 10005 · 800-523-8350
Related reading
- What Questions Go in an Attorney 360-Degree Review?
- Why Does Your Law Firm Need 360-Degree Reviews?
- What Is Structural Anonymity in a Performance Review?
.png){kind=small}