One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.
Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.
Coming from a natural science field we have a very different review process (but we also "only" review up to 100 papers per committee). Essentially we have topical sub-committees with ~10-15 members. The authors choose which sub-committee they submit to and there is typically a rearranging process by the program chairs and subcommittee chairs to check if there are some very obvious wrong category submissions. I should note that it's typically a disadvantage to submit to the wrong subcommittee, because if members don't really understand the paper they are much more likely to reject.
Every committee member reads all the papers (and indicates conflicts if necessary). We then have a committee meeting where all papers are discussed and accept/reject is being voted on. In these meetings it does happen that one reviewer picks up a subtle point (or finds e.g. a previous publication), that others have missed and this can lead to the reject of even highly scored papers. Having this many eyes and a discussion about he papers definitely helps IMO. The big difference here is that we don't get 10,000 submissions (more like 1,000).
I was actually very surprised that it is possible to register duplicate accounts at those CSE conferences. We get send a single invite to our work address and need to lock into the system using that email. And we are being nominated to get onto the committee.
Coming from the natural sciences, I was really surprised that reviewers get to pick what they review! Despite the deeper problems discussed above, removing this glaring bug in the system would be quite easy. If editors have to choose the reviewer assignment may be less efficient, but at least editors have to think more about who would be a good reviewer, and mostly they also know the players in their field and who may have conflicts of interest.
Yea, I think this is likely a pretty big issue with highly specialized field - there can be an issue with new participants breaking into the field (due to an entrenched old boys club) along with a difficulty getting enough sample data specific to that corner of academia.
I wonder if we could randomly assign reviewers but allow the reviewers to self-report a level of familiarity on the subject matter in general (ideally in advance) and on the paper topic in particular.
People can also cheat on this self-reported familiarity right? On specialized areas I don't see a solution, the colluders might as well be the only experts in the field so you have to enlist them no matter what. But from the description this doesn't seem like what's happening.
"The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers."
I think that by involving randomness before any active self-selection you wouldn't necessarily involve specialty-wide clusters but you'd make it a lot harder to actively seek out the documents you want to review.
It might also help to attack the ability to create duplicate accounts. Given how relatively few professors exist in the world I'd assume you could put a lot more effort into duplicate account detection than they are right now.
One critical vulnerability in the current reviewing pipeline is that the reviewer assignment algorithm places too much weight on the bids. Imagine if you bid on only your friend's paper. The assignment system, if they assign you to any paper at all, is highly likely to assign you to your friend's paper. If you register duplicate accounts or if there are enough colluders, the chance of being assigned to that paper is extremely high.
Fortunately, this is also easy to detect because your bid should reflect your expertise, and in this case it doesn't. What we showed in our paper is that you can reliably remove these abnormal bids. It's not a perfect solution, but it helps.