[EM] Amateur peer-reviewed "journal" for voting methods, criteria, and compliances?

[Jameson Quinn]

One subdomain of voting methods in which the peer-reviewed academic
literature is decidedly behind the amateur enthusiast community (that's us)
is in its coverage of different methods, criteria, and compliances. This
lag is unfortunate for several reasons. For one thing, it affects whether
known facts can be covered on wikipedia. For another, though some
academics clearly are aware of the amateur knowledge, insofar as others
aren't, it leads to needless misunderstandings and/or duplication of effort.

I believe we can fix this by creating a peer reviewed open-access journal
which combines the strengths of the academic and enthusiast communities.
The purview of such a journal should be strictly limited to what is
mathematically expressed and/or provable, so that sufficiently meticulous
amateurs can be considered as qualified peer reviewers (when accompanied by
at least one experienced reviewer).

This would take significant work to start and sustain. I'm volunteering to
do up to around half the work, but in practice that means I need at least
two or three others to step forward as volunteers before we can seriously
get started. Here's what I think we need to do:

*Managing editors*

These would be the people who would be ultimately responsible for
everything. However, their role *qua*¹ managing editors would be more to
shepherd things along; they may or may not also take the role of reviewers,
authors, etc. I think a group of 3-5 managing editors would be sufficient
to get things done without burnout. Among the managing editors would need
to be at least one with a relevant postgraduate degree (for instance
mathematics, statistics, economics, or political science).

*Site*

I think this would work best if it were a sub-branch of some credible
existing site. That way, any existing credibility would be inherited. My
first suggestion would be Adrian Tawfik's "Democracy Chronicles". I'd also
be happy to discuss it if any existing organizations (hint, hint) wanted to
lend their name and/or site.

*Hosting, software, etc*.

I suggest that the journal should be run MediaWiki software, the same
software Wikipedia runs on. However, all "main space" articles should be
protected from changes by all but a limited group of editors. This would
allow freewheeling discussion on "talk" pages, but keep actual "published"
content in a stable, citeable form.

*Purview*
*
*
I think that this should focus on only four kinds of articles: system
definitions (or equivalent re-formulations to help with proofs); criteria
definitions; inter-criteria implications and equivalencies; and
system-criteria (non)compliance proofs/counterexamples. Initially, only
single-winner systems and criteria would be considered, although that could
change later. Systems and criteria would not be considered "published"
without a certain level of "coverage" in terms of (non)compliance proofs.
Thus, each individual "article" would frequently (though not always) be
under a page in length. This short length and limited purview would
establish an important differentiator between this effort and existing
journals.

*Peer reviewers*
*
*
We'd need to have a broad, balanced group of peer reviewers. Reviewers
would NOT be required to have any specific degrees, but WOULD be required
to demonstrate a clear knowledge of the norms of mathematical proof. I'd
think that 12-20 reviewers is a reasonable goal. I would expect that around
2/3 of these reviewers would be capable "amateurs"; I hope we can get
participation from enough professional academics to constitute at least 1/3
of the reviewers. (I already have several ideas of whom I'd ask, though I'd
also expect the other managing editors to help with this.)

*Peer review and other article life-cycle issues*

There would be a clear naming scheme to distinguish the various article
types. Articles would initially be developed in the main namespace in
unlocked form. This would allow any wiki user to help or comment.
(Obviously, spammers and other troublemakers would need to be banned.)

When an article was considered ready for peer review, its main author would
tag it as such, and it would be protected from further editing (though the
associated talk page would still be open for comments).

Peer reviewers would have a chance to volunteer using one-shot pseudonymous
accounts (for which the identities would be secretly verified by any
managing editor). Thus, authors would NOT be anonymous, but reviewers
WOULD. Reviewers would be encouraged to volunteer if they have any serious
negative concerns on an article, even if they do not wish to fully review
all aspects of that article.

Each article would need at least 3 reviewers, at least 1 of whom must have
prior experience with the peer review system at this or any other journal.
If that requirement isn't met by volunteers, the managing editor would
attempt to assign reviewers until it was.

A review would consist of any number of suggestions, along with a
determination of "acceptable as is", "acceptable with minor revisions",
"potentially acceptable with major revisions", or "unacceptable". Because
of the sharply limited scope of each individual article, it is expected
that "acceptable as is" would not be as negligibly rare as it is in most
peer review. Reviewers who gave one of the top two determinations would be
expected to have carefully reviewed the entire article; those who didn't,
wouldn't.

Once an article had all its reviews, its author would be given permission
to revise if necessary. When they tag it as "revisions done", those
permissions would be revoked, and reviewers would have a chance to raise
(or lower) their determination. This process could iterate if necessary.

In order to be considered "published", an article would need to be
unanimously graded "acceptable". At that point, it would be permanently
locked, though the talk page would remain open.

....

Supposing we have 20 "core" systems and 20 "core" criteria which are to be
(dis)proven for each of the "core" systems. That's a substantial total: 400
proofs. However, perhaps half of those will already exist in the published
literature, and perhaps half of the remainder will be utterly trivial (such
as well-known counterexamples). That leaves about 100 compliances that
would need careful review. With 15 reviewers, that's about 20 "core"
compliances each for review. Including non-"core" systems and criteria, I'd
expect that to increase to around 30. If the average reviewer handled 1 a
month (plus 1 of the "trivial" cases), that workload would take around 3
years to burn through, with each month's "issue" containing around 5
serious and 5 trivial compliances. I think that that's roughly doable, if
we put our minds to it.

Also, note that once we got the ball rolling and showed we were doing a
good job, we could attempt to get an existing professional society to
"adopt" the journal. If this were successful it would massively increase
our credibility and ability to attract new peer reviewers and authors.

....

The above gives a basic idea of what it would take. Obviously, there's a
lot of minor and not-so-minor details still to work out. But I hope that
this message is enough to get the ball rolling.

So: please respond. Any comments, suggestions, or questions? Do you think
you could be a managing editor or peer-reviewer? Is there someone else you
think should be in on this conversation?

Jameson

¹ "*qua*" in this case means approximately "as, per se"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.electorama.com/pipermail/election-methods-electorama.com/attachments/20120928/d7ccd5ef/attachment-0003.htm>