We have a longstanding tradition (since 1968!) at the Great Lakes Science Center of giving an annual award to the best scientific paper published by our staff. As the number of publications has increased over time, the task of reviewing has become more onerous. So, we are considering divvying up the reviewing work among all of our scientists, rather than just a small committee. And we chose to do this by organizational group, to encourage discussion and debate within each group assigned the same set of papers to review.
It sounded like an easy enough task, but it took me a while to wrap my head around the problem.
Let’s say there were 100 papers published in 2016, and each of those papers was associated with one of 8 organizational units.
pubs <- data.frame(paper=paste("Paper", format(1:100)),
orgunit=sort(sample(paste0("Grp", LETTERS[1:8]), 100, replace=TRUE)))
DT::datatable(pubs)How do we assign each paper to a group to review, such that no group is assigned to review one of its own papers and each group gets about the same number of papers to review?
I wrote a function to do the work, assignReviewer() in the jvamisc package.
I started with the end result in mind, a matrix of the number of papers by review groups (rows) and author groups (columns).
I figured out how many papers each group should review, and if the number of papers was not evenly divisible by the number of groups, I assigned more papers to those groups that published more.
Finally, I filled up the matrix one column (publishing group) at a time, with a random ordering of their papers getting distributed approximately evenly to all the other groups.
To ensure that the totals came out as evenly distributed as possible in the end, I had to prioritize how the matrix was filled. The trick to making it work was keeping track at every step of not only the total number of papers assigned to each review group, but also which publishing group was coming up next. For example, if publishing group E was the next column in the matrix to fill, I should fill up reviewer group E with extra papers right now, because I won’t be able to assign any papers to them in the next step.
library(jvamisc)
out <- assignReviewer(data=pubs, Unitvar="orgunit",
prefOrd=paste0("Grp", LETTERS[1:8]))
DT::datatable(out)Approximately even distribution of 100 papers published by 8 organizational units (columns), assigned to those same 8 units for review (rows).
knitr::kable(addmargins(table(out$Reviewer, out$orgunit)))| GrpA | GrpB | GrpC | GrpD | GrpE | GrpF | GrpG | GrpH | Sum | |
|---|---|---|---|---|---|---|---|---|---|
| GrpA | 0 | 1 | 2 | 1 | 2 | 2 | 2 | 2 | 12 |
| GrpB | 2 | 0 | 2 | 2 | 2 | 1 | 1 | 2 | 12 |
| GrpC | 1 | 1 | 0 | 1 | 3 | 2 | 2 | 3 | 13 |
| GrpD | 2 | 0 | 2 | 0 | 2 | 2 | 1 | 3 | 12 |
| GrpE | 1 | 1 | 2 | 2 | 0 | 2 | 2 | 3 | 13 |
| GrpF | 2 | 1 | 3 | 1 | 2 | 0 | 1 | 3 | 13 |
| GrpG | 1 | 1 | 2 | 2 | 3 | 1 | 0 | 2 | 12 |
| GrpH | 1 | 1 | 2 | 2 | 3 | 2 | 2 | 0 | 13 |
| Sum | 10 | 6 | 15 | 11 | 17 | 12 | 11 | 18 | 100 |