<div dir="ltr"><div>Hi Rob and all,</div><div>I like the simultaneous readability, versatility and robustness of ABIF.</div><div>If the multiplier separator is made variant (asterisk / colon / both),</div><div>I think the only difference between Pivot and ABIF would be the</div><div>significance of whitespace in candidate tokens, something that the</div><div>parser could also be liberal in accepting - I see the [] bracketed form</div><div>as more readable (and thus default) but no problem in making</div><div>the non-bracketed form acceptable.</div><div>Also, if leaving out the scores for the unordered variant is allowed,</div><div>the format will be able to record approval votes in a very readable</div><div>form, which I would support.<br></div><div>If the community eventually arrives at some sort of consensus,</div><div>I'll put the ABIF read/write implementation into the backlog of</div><div>votelib (which can already read BLT and STV formats) so that</div><div>it has an interchange format for ordinal ballots as well.</div><div> The implementation would benefit from a formal parsing definition</div><div>(BNF or similar); if not, I might create it in the process.</div><div>All the best,</div><div>Jan<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 28, 2021 at 11:32 AM Rob Lanphier <<a href="mailto:roblan@gmail.com">roblan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi everyone<br>
<br>
John, thanks for putting the subject on this thread and for the<br>
pointer to Vote::Count. Carl and James, thanks for the pointers.<br>
I'll endeavor to keep all of you (and everyone on this mailing list)<br>
in the loop on the progress we're making. The reddit thread<br>
continued, which I'll repeat the important stuff that I said there.<br>
<br>
The name (and file extension) for the format that I'm gravitating<br>
toward is ABIF (".abif"), which stands for "aggregated ballot image<br>
format". I'm using the term "ballot image" because that seems to be<br>
the term of art for publishing real-world electoral results. Once<br>
upon a time, "ballot image" meant "a picture of the ballot", but now<br>
just means a crude ASCII representation in a line of text.<br>
<br>
I did some processing of the ballot images from San Francisco's 2018<br>
mayoral election, which involved some coding and some manual shell<br>
processing with grep and friends. My work was ugly the way that all<br>
manual futzing in bash is ugly, but I got a few regexps and some test<br>
data (and some experience) that I'm applying here. As I was<br>
processing the results, I had wished the results were aggregated in an<br>
easier to process manner. I would love to finish my processing work<br>
and publish it in a sane format that other programmers can use, which<br>
I'm hoping ABIF can become.<br>
<br>
I've been working for WAY TOO MANY years on text formats for<br>
aggregated results. The format that I used in my Perl script<br>
published in the Perl Journal in 1996 was a noble attempt, and was<br>
better (in many ways) than the revised JSON-based format I published<br>
in 2005 with electowidget. I think my obsession with JSON was an<br>
unfortunate detour in coming up with a good format. I've been<br>
studying text formats for structured data for a very long time, ever<br>
since I learned Perl (in 1994 or so) and the work intensified with the<br>
work on RTSP in 1996 through 1998. We briefly dabbled with making<br>
RTSP a binary protocol using ASN.1, but Dr. Henning Schulzrinne from<br>
Columbia University convinced us (by publishing his "RTSP prime"<br>
draft) that a text-based HTTP knockoff (with MIME headers) was the way<br>
to go. As we worked on RTSP (and SMIL), we saw the rise of XML, and<br>
the slow steady fading of XML as a data format with the advent of<br>
JSON, and YAML, and TOML, and many other simpler formats.<br>
<br>
That's my longwinded way of saying that the test cases that I've<br>
published on the ABIF electowiki page are (I think) a respectable<br>
start for a flexible text format that a wide variety of programmers<br>
can get their heads around:<br>
<a href="https://electowiki.org/wiki/ABIF" rel="noreferrer" target="_blank">https://electowiki.org/wiki/ABIF</a><br>
<br>
IThe thing that I love about ABIF (as it's shaping up) is that it<br>
solves several big problems which my 2005 electowidget format didn't.<br>
It goes back to the roots of the 1996 Perl script, back before I "knew<br>
what I was doing", and seems simpler to work with for a reasonably new<br>
programmer (as I was in '96). My electowidget format REQUIRED<br>
ratings, which I would normalize to rankings as appropriate. But I<br>
was often trying to express elections that only had ranked ballots<br>
available (e.g. the 2003 Debian election, and the 2009 Burlington<br>
mayoral race). Having a format that ALLOWS for ratings, but doesn't<br>
require ratings seems appropriate given that IRV/RCV is much more<br>
common in municipal use right now than rating systems. I love that<br>
the format is very similar to the ad hoc format many have used here on<br>
the EM list for expressing rankings. I'm reasonably sure that writing<br>
a parser in any language that has reasonable regular expression<br>
support will be easy, and can probably be done with a single-pass<br>
parser. I haven't really started it yet, but I know how to write a<br>
spec that many programmers can look at, nod their heads, and say<br>
"yeah, I can work with that". I think having a test suite with<br>
well-specified expected output is going to be a key part of solving<br>
the interoperability problem, and it will be helpful for others to<br>
inspect in a piecemeal fashion rather than feeling obligated to read a<br>
ponderously long specification.<br>
<br>
In the next few days, I'll take a look at the implementations that<br>
have been mentioned in this thread. For example: Pivot's format that<br>
Carl pointed to. It looks to me that the format that Pivot uses is<br>
very similar to this proposed ABIF format, with the only difference I<br>
see (at first glance) is that this ABIF:<br>
<br>
27: A > B > C > D<br>
26: B > A = C > D<br>
24: C > A = D > B<br>
23: D > C > A > B<br>
<br>
...would become this in Pivot:<br>
27 * A > B > C > D<br>
26 * B > A = C > D<br>
24 * C > A = D > B<br>
23 * D > C > A > B<br>
<br>
Changing colon (":") to asterisk ("*") is an interesting change to<br>
consider. I suspect that as we all look more closely at Pivot and<br>
other formats, there's going to be other incompatibilities and mindset<br>
differences to hash out. These all seem like easily solved problems,<br>
because I get the sense that many programmers are hungry for<br>
compatible solutions in this space, and are willing to write<br>
converters to be part of the compatibility party.<br>
<br>
At any rate, I think (if others on the mailing list don't mind) that<br>
we should just use this mailing list and electowiki as places to hash<br>
out the format. If we do this right, it will be easy enough to use<br>
that people on this mailing list (and over on reddit, and many other<br>
places) will keep ABIF compatibility in mind when they write examples<br>
of elections to consider. Hopefully having more software<br>
compatibility in our ecosystem will make it easier for us to<br>
collaborate on analysis and speed up reform efforts.<br>
<br>
Rob<br>
<br>
On Thu, May 27, 2021 at 1:34 PM John Karr <<a href="mailto:brainbuz@brainbuz.org" target="_blank">brainbuz@brainbuz.org</a>> wrote:<br>
><br>
> As the author of Vote::Count, a standardized format for ballots would be<br>
> a big plus. When I've been able to collect sample data, the first thing<br>
> I need to do is convert it to my format. Currently Vote::Count has two<br>
> formats, a text one for ranked ballots and a json/yaml format for range<br>
> ballots. The documentation on my formats is here:<br>
> <a href="https://metacpan.org/pod/Vote::Count::ReadBallots" rel="noreferrer" target="_blank">https://metacpan.org/pod/Vote::Count::ReadBallots</a><br>
><br>
> I'm not on Reddit, but I think creating a working group of people with<br>
> an interest to propose a standard would be a great idea, and I'm<br>
> interested in helping.<br>
><br>
> A standard format would allow creation of a library of data for which<br>
> electowiki would seem to be a natural home.<br>
><br>
> On 5/27/21 4:02 PM, <a href="mailto:election-methods-request@lists.electorama.com" target="_blank">election-methods-request@lists.electorama.com</a> wrote:<br>
><br>
> > Send Election-Methods mailing list submissions to<br>
> > <a href="mailto:election-methods@lists.electorama.com" target="_blank">election-methods@lists.electorama.com</a><br>
> ><br>
> > To subscribe or unsubscribe via the World Wide Web, visit<br>
> > <a href="http://lists.electorama.com/listinfo.cgi/election-methods-electorama.com" rel="noreferrer" target="_blank">http://lists.electorama.com/listinfo.cgi/election-methods-electorama.com</a><br>
> ><br>
> > or, via email, send a message with subject or body 'help' to<br>
> > <a href="mailto:election-methods-request@lists.electorama.com" target="_blank">election-methods-request@lists.electorama.com</a><br>
> ><br>
> > You can reach the person managing the list at<br>
> > <a href="mailto:election-methods-owner@lists.electorama.com" target="_blank">election-methods-owner@lists.electorama.com</a><br>
> ><br>
> > When replying, please edit your Subject line so it is more specific<br>
> > than "Re: Contents of Election-Methods digest..."<br>
> ><br>
> ><br>
> > Today's Topics:<br>
> ><br>
> > 1. (no subject) (Rob Lanphier)<br>
> ><br>
> ><br>
> > ----------------------------------------------------------------------<br>
> ><br>
> > Message: 1<br>
> > Date: Wed, 26 May 2021 23:38:14 -0700<br>
> > From: Rob Lanphier <<a href="mailto:roblan@gmail.com" target="_blank">roblan@gmail.com</a>><br>
> > To: <a href="mailto:election-methods@lists.electorama.com" target="_blank">election-methods@lists.electorama.com</a><br>
> > Subject: [EM] (no subject)<br>
> > Message-ID:<br>
> > <CAK9hOYn2T=ympC7gEd8wS_8S8yjzK==<a href="mailto:xsmEfNKWo99cBjaXDgA@mail.gmail.com" target="_blank">xsmEfNKWo99cBjaXDgA@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="UTF-8"<br>
> ><br>
> > Hi folks,<br>
> ><br>
> > There's an interesting discussion happening on reddit about ASCII<br>
> > formats for aggregated ballot images. I'll provide a deep link to my<br>
> > comment here:<br>
> ><br>
> > <<a href="https://www.reddit.com/r/EndFPTP/comments/nkm2cd/standardizing_cardinal_ballot_notation/gzls6pj/" rel="noreferrer" target="_blank">https://www.reddit.com/r/EndFPTP/comments/nkm2cd/standardizing_cardinal_ballot_notation/gzls6pj/</a>><br>
> ><br>
> > What the original reddit poster (/user/jman722) made me realize is<br>
> > that it's possible to come up with a format that works for both range<br>
> > ballots and ranked ballots. The range ballots can be on a scale of<br>
> > 0-5, where 5 is "awesome", and 0 is "awful". The ranked ballots can<br>
> > be A>B>C.<br>
> ><br>
> > I'm going to use the example that the original reddit poster made:<br>
> ><br>
> > 12: Allie/5, Billy/5, Candace/4, Dennis/3, Edith/3, Frank/2, Georgie/1, Harold/0<br>
> > 7: Allie/4, Billy/0, Candace/2, Dennis/3, Edith/1, Frank/0, Georgie/5, Harold/3<br>
> > 5: Allie/0, Billy/3, Candace/2, Dennis/3, Edith/4, Frank/5, Georgie/3, Harold/4<br>
> ><br>
> > That format is good but not great. It takes a careful eye to see that<br>
> > Allie, Billy, Frank, and Georgie are the passionate favorites (earning<br>
> > a "5" score), and another close look to see that Allie, Billy, Frank,<br>
> > and Harold are listed as completely unacceptable (earning a "0" score)<br>
> ><br>
> > My old format that I used for my 1996 Perl script that I wrote and<br>
> > published in The Perl Journal would express those ballots this way:<br>
> ><br>
> > 12: Allie=Billy>Candace>Dennis=Edith>Frank>Georgie>Harold<br>
> > 7: Georgie>Allie>Dennis=Harold>Candace>Edith>Billy=Frank<br>
> > 5: Frank>Edith=Harold>Billy=Dennis=Georgie>Candace>Allie<br>
> ><br>
> > With this format, it becomes clear that 12 voters really like Allie<br>
> > and Billy and really don't like Harold. The next 7 voters really like<br>
> > Georgie, and really don't like Billy and Frank. The remaining 5<br>
> > voters really like Frank, but really dislike Allie. One has to add up<br>
> > 12+7+5 to realize there are 24 voters in this election.<br>
> ><br>
> > The ratings are stripped from my old 1996-ish format. It only<br>
> > provides the following parse tokens:<br>
> ><br>
> > [quantity]: [cand5yay] [> or =] [cand4good] [> or =] ... [cand0boo]<br>
> ><br>
> > It seems as though it would be possible to come up with a merged<br>
> > format that would express the range ballots above like this:<br>
> ><br>
> > 12: Allie/5 =Billy/5 >Candace/4 >Dennis/3 =Edith/3 >Frank/2 >Georgie/1 >Harold/0<br>
> > 7: Georgie/5 >Allie/4 >Dennis/3 =Harold/3 >Candace/2 >Edith/1 >Billy/0 =Frank/0<br>
> > 5: Frank/5 >Edith/4 =Harold/4 >Billy/3 =Dennis/3 =Georgie/3 >Candace/2 >Allie/0<br>
> ><br>
> > The ">", "=", and "," characters could all be optional delimiters<br>
> > between the candidate/score tuples on each line (though at least one<br>
> > of those three delimiters WOULD be required). If ">" or "=" is used as<br>
> > a delimiter, then the candidates MUST be ordered by score (highest<br>
> > score first). Candidate tokens can be one or more ASCII characters<br>
> > ([A-Z] or [a-z]) OR the candidate token MUST start with a square<br>
> > bracket ([) and end with the closing square bracket (]), and the<br>
> > intervening text can be any unicode character (e.g. [Do?a Garc?a<br>
> > M?rquez] or [Ximena Pe?a] or [???]) . Whitespace can be discarded, but<br>
> > SHOULD be included for legibility.<br>
> ><br>
> > Linters could be created to deduplicate ballot lines, sort the<br>
> > candidate by score on each line, convert commas to ">" and "=" (for<br>
> > ranked ballot equivalents), and add whitespace for readability. They<br>
> > could optionally normalize the candidates to a range of ASCII letters<br>
> > (e.g. changing "Allie" to "A", "Billy" to "B", etc).<br>
> ><br>
> > The goal would be to make it useful for two people debating whether<br>
> > the Condorcet criterion or the Monotonicity criterion is more<br>
> > important. They could both easily crank out a set of ballots that<br>
> > could be fed into either a ranked-ballot counter or a rated-ballot<br>
> > counter. Having the candidate tuples sorted in each line makes it<br>
> > clearer what the preferences were of the set of voters represented by<br>
> > the given line.<br>
> ><br>
> > I think that parsers could be written for this format such that they<br>
> > follow Postel's Law (a.k.a the "robustness principle"):<br>
> > <a href="https://en.wikipedia.org/wiki/Robustness_principle" rel="noreferrer" target="_blank">https://en.wikipedia.org/wiki/Robustness_principle</a><br>
> ><br>
> > To quote that^: "be conservative in what you do, be liberal in what<br>
> > you accept from others"<br>
> ><br>
> > People trying to express ranked ballots could drop the scores, and<br>
> > ONLY include ">" and "=" as a delimiter between candidates, People<br>
> > trying to express rated ballots could use commas (",") instead of ">"<br>
> > and "=". Programmers trying to parse handcrafted scenarios could<br>
> > figure out how to fill in the blanks.<br>
> ><br>
> > I'm tempted to write a reference parser for this, but first, what do<br>
> > you all think? Let the list know! Let me know! Let reddit know!<br>
> > :-D<br>
> ><br>
> > Thanks<br>
> > Rob<br>
> ><br>
> > p.s. I'm thinking of calling my version "ABIF", standing for<br>
> > "Aggregated Ballot Image Format". I may just document it here:<br>
> > <a href="https://electowiki.org/wiki/User:RobLa/ABIF" rel="noreferrer" target="_blank">https://electowiki.org/wiki/User:RobLa/ABIF</a><br>
> ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > Subject: Digest Footer<br>
> ><br>
> > _______________________________________________<br>
> > Election-Methods mailing list<br>
> > <a href="mailto:Election-Methods@lists.electorama.com" target="_blank">Election-Methods@lists.electorama.com</a><br>
> > <a href="http://lists.electorama.com/listinfo.cgi/election-methods-electorama.com" rel="noreferrer" target="_blank">http://lists.electorama.com/listinfo.cgi/election-methods-electorama.com</a><br>
> ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > End of Election-Methods Digest, Vol 202, Issue 7<br>
> > ************************************************<br>
><br>
> ----<br>
> Election-Methods mailing list - see <a href="https://electorama.com/em" rel="noreferrer" target="_blank">https://electorama.com/em</a> for list info<br>
----<br>
Election-Methods mailing list - see <a href="https://electorama.com/em" rel="noreferrer" target="_blank">https://electorama.com/em</a> for list info<br>
</blockquote></div>