OpEdNews » Peeling the Onion: Can Voting System Testing be Reliable and Extensive?

Support OEN
With Your
Tax-Deductible Donation

Tags for This Article:

Populum Tag Cloud Control Panel
	Fine tune your search to access content

Articles	Links
Diaries	Products
Events	All

All time
Last 6 mos
Last month
Last week
Last 24 hrs

From:
Month Day Year

To:
Month Day Year

Alphabet Popularity	Count ON Count OFF
This Level	Sub-levels

Tag(s): Democracy; Electronic Voting; Government; Optical Scan Voting Machines; Software; Voting; Voting Integrity; Voting Laws Federal HAVA; Voting Laws State; Voting Machines; (more...) Voting Reform; Voting Technology (less...)
Add to My Group

April 30, 2008 at 20:02:34

Peeling the Onion: Can Voting System Testing be Reliable and Extensive?

by Ellen Theisen

www.opednews.com

SAVE FAVORITES

VIEW FAVORITES

View Ratings | Rate It

An email dialog between Lani Brown (author of "A Margin of Error, Ballots of Straw") and Ellen Theisen (founder and Co-Director of VotersUnite.Org).

LANI: "Testing voting systems should and can be reliable and extensive."

ELLEN: Should, yes, but not can.

Bear in mind that testing would have to be done, reliably and extensively, before every election on every machine that was newly programmed for a new election.

LANI (responding): When you say "newly programmed," are you talking of the ballot parameters input into the machine that define the ballot for each election? If so, the software program remains the same as that tested and certified by the Secretary of State. The state's certification process should be intensive enough to test for the most complex ballot configurations and voter behavior.

ELLEN (responding): Some emails that Jody Holder got hold of in an open records request revealed that, with ES&S optical scanners (at least), the ballot programming is merged into the basic executable code to generate brand new software for the election. It isn't just putting in some parameters.

John Washburn wrote a terrific piece about it where he used a paint analogy - which fits perfectly. He pointed out that the source code is like yellow paint and the ballot programming is blue paint. In the ES&S system, the two are inextricably mixed, and the final software is green. So it isn't the same software as the certified software at all. And if there's different ballot programming for different precincts, then it isn't even same software on all the county's machines.

LANI: Another, "Yes, but." The base line is the same. The base source program code they started out with SHOULD have been the same base program code that was certified. So in effect the only new piece is the ballot, not the base program code that counts the votes. In other words, some computer programmer somewhere is not writing new source code for every election.

ELLEN: What if ES&S does the ballot programming? - as they do all over the country? Aren't they producing new software for every election?

ELLEN (continuing): And the testing of the new program would have to be done by an election administrator who has no experience in testing any type of equipment -- let alone software -- and no understanding of what such testing involves. (Perhaps there are a couple of exceptions out there somewhere.)

LANI: Is it not the election administrator's job to know the election equipment and to understand and know his/her ballot definitions and how the ballot performs on the equipment?

ELLEN: Al Kolwitz wrote a beautiful letter to the editor in which he pointed out that the election administrators' job is to administer the election. Then they got hit with having to be IT experts, which they aren't. While it may be their jobs, they don't have the necessary qualifications. It would be daunting (actually, probably impossible) to replace all election administrators with IT personnel who could also manage all the other aspects of elections.

LANI: I do not believe election administrators should have to be IT experts. They're administrators not programmers. If they wanted to be computer programmers, they would have gone to geek school. (And even if they were IT experts, they would never know the election program code and would not initially be familiar with their election system, until they became familiar with it. Hmm.)

However, election administrators are indeed users of the system and they should know their system from a user perspective. If their system is too complex, too cumbersome to know and use effectively, it's a fault of the system.

Your statement, "While it may be their jobs, they don't have the necessary qualifications," is troubling. In a perfect world none of us would be assigned jobs for which we lack access to the tools, budget, experts necessary to be successful in that job. Easy to say, "Hire the best people." Not so easy to do when you've no budget, training, resources.

ELLEN (continuing): AND, if the jurisdiction has DREs, for the testing to be reliable and extensive, every single machine would have to have a huge number of ballots input by hand -- not the automated scripting that only tests that the software can read the other software. To test DREs correctly would be prohibitive. Especially, when you consider that an error in the results on one machine would have to be investigated, which means reviewing the video of the person inputting the hundreds of ballots by hand on the touch screen (or push button).

LANI: Back to "same hardware, same software." If the election administrator PROVES the ballots work on a sampling of like DREs, they "should" work on other like DREs. However, all machines should be functional, whatever process the election administrator uses to prove the screens are calibrated, the correct ballot is loaded, the vote count is zero, and ...

Yes, a huge number of ballots should be input by hand testing all possible voter behavior, page forward/backward, single/double click (GAO's word not mine).

ELLEN (continuing): Of course, it's more feasible to test the opscans, but even there each one would have to be tested -- not just a representative sample. You could use many of the same ballots in each deck, but some large counties have hundreds of scanners. Even to run the same deck through all these scanners and compare the results with the expected results would take quite a chunk of time. So, they don't do it.

LANI: Same as above.

ELLEN: But just because the read heads in one opscan are working fine, how do you know the read heads in all of them are? (LANI: Exactly. You don't.) Maybe they got dirty in a few and not the others. Maybe a sensor or a light is wearing out in some and not the others. Maybe the rollers or the ballot alignment thingy ..... (Actually, I don't know the critical parts, so I'm sort of making up these part names.)

IF the state certification is sound, then using John Washburn's testing guidelines to build a test deck and executing it on every opscan would be pretty close to reliable.

LANI: I like this idea.

ELLEN (continuing): But using his guidelines to test every DRE would be virtually impossible -- especially when you add the page-forward/page-back aspects you pointed out.

I think my real point is that IT NEVER IS "same hardware, same software." (I got a terrible Dell computer once, but other people love theirs.)

LANI: And components break at different rates, which is why it is so troubling to me that we don't have better "it's a bad election" processes in place. You get a tainted read head and so goes the election.

The certified software should be the same on all machines, as should the operating system.

ELLEN (continuing): Election administrators don't do the necessary testing, partly because they have no idea how to do it right, and partly because if they did know how, they would realize they didn't have the time.

Instead, they fall back on faith in federal testing and state certification -- with ABSOLUTELY NO UNDERSTANDING that their particular election makes everything new and different, that all previous testing is virtually irrelevant, and the testing has to be started and done thoroughly all over again.

LANI: IF the state certification is sound, all other things being equal only their ballot(s) changes. I don't intend to minimize the job of defining and testing the ballot(s). It's enormous. But, if you're starting on a level playing, one in which you know your machines work, then proving your ballot is less the impossible dream and more within the realm of your expectations and control.

ELLEN (continuing): This is exactly the reason why it is indisputable that every election is a beta test of the election equipment, with no reporting mechanism in place to make it a respectable beta test.

LANI: Your comment, "every election is a beta test," is indeed the core of the problem.

Two identical computers containing identical hardware and identical software will perform identically until something breaks or is altered. The Secretary of State certifies those computers (hardware and software) as working and acceptable to the task of correctly counting votes, using variable ballot input parameters. That means every time: when the ballot is intrinsically complex, when there is record turnout, when there's power fluctuation or failure, and so on. In the case of the DREs it means even when the voter changes the ballot and pages forward and backward innumerable times. It is the job of the Secretary of State to prove the machines will not botch the votes.

The election supervisor/election administrator "should" be able to be confident that rigorous, comprehensive acceptance testing has occurred. The election supervisor "should" be able to assume the machines he/she acquires perform to the quality certified by the Secretary of State.

However, the election supervisor (ELLEN: More often, the vendor) creates and inputs his/her ballot parameters for every election. The election supervisor must scrupulously test that ballot. In Sarasota 2006, 97% percent of the possible Election Day ballot data combinations were not tested. Testing absolutely every ballot data combination would not be possible. Testing the more likely combinations is reasonable and responsible and absolutely doable.

Should every machine be load tested prior to each election? If all computers are identical, I don't believe this is necessary. However, one extreme is Miami-Dade's SoE in 2000 who didn't clean the chads out of his ballot stands (one obstacle to punching the chads out cleanly). Another is Sarasota 2006 with the (un)calibrated touchscreens and the smoothing filter and the warning memo that went unheeded.

The election supervisor should expect the state to do its acceptance testing job during certification. However he/she cannot and should not expect that nothing will ever go wrong with the ballots or the machines. Each and every machine should be proved working prior to being used for voting.

ELLEN: Hmm, it feels like we're having an argument -- or a debate. But I'm not sure what it's about. We seem to agree on an awfully lot. Is our basic point of disagreement in the subject line ["Testing should and can be reliable"]? I agree it should be reliable, but not that it can be. Or do we even disagree?

LANI: Ellen, Absolutely not, no, neither. Neither argument nor debate, more like peeling the proverbial onion. There is layer upon layer of complexities and systems and processes and people responsibilities. On any given day, we might all of us say something completely different but all be absolutely correct.

I don't believe we disagree. On the "testing should and can be reliable," I believe, I know we can and should do more. I look at this from the technical perspective, knowing that in a "perfect world" all systems are quality tested to a fault before being shipped out the door. Also knowing that, depending on the corporation and circumstances (your Dell), some glitches fall through the cracks.

"Cracks" not canyons. Now comes Florida. Consider what might have been IF Florida's Secretary of State had performed a thoughtful analysis of voting machine requirements before sanctioning the touchscreens in the first place. You know, the part about counting all votes, reliability, stability, auditability, proof.

Consider what might have been IF Florida's Secretary of State had performed rigorous, comprehensive testing before certifying the touchscreens. (Remember the GAO reported they didn't even touch the touchscreens when performing load testing. Plus they used ES&S test data.)

Consider, in the event Florida's Secretary of State's scrupulous testing failed to uncover the touchscreen calibration problem, what might have been in 2006 had Sarasota's supervisor of elections performed a more thorough testing of her ballot and performed a cursory sniff-test on the touchscreens.

ELLEN: Yes, peeling the onion. I feel like I've been doing that for the past five years. Waaay too slowly.

Five years ago, I worked really, really hard to get additional co-sponsors to HR.2239 - Holt's first vvpat bill, which required a whoppin' 0.5% spot check (which the bill called an "audit"). And we got a LOT of co-sponsors -- nearly enough to call the bill out of committee for a floor vote. Hurrah for us!

But since then I've been reading, learning, stopping and thinking, and absorbing information passed on by others:

Teresa Hommel on the necessity of observability. ("Oh My Gosh, of course," say I)

Bev Harris on the real meaning of audits ("OMG, of course," say I)

Nancy Tobi on the EAC ("OMG, of course," say I)

Pokey Anderson on the incomparable incentive for stealing an election ("OMG, of course," say I)

... and the list could go on and on.

Bless the uncompromising stance of thinking people who aren't willing to compromise democracy, because they know that a compromised democracy is a contradiction in terms.

Slowly (too slowly), I've realized that election "systems" are unlike anything else at all, and so none of the rules that apply to other things apply to election "systems."

1. Secret ballots make a voting system unauditable. Period. Full stop. So, if we don't get it right on election night, it's going to stay wrong. Is there anything else that is unauditable? I can't think of any.

2. No other "system" provides as much incentive for fraud. Not even a financial system.

3. No other "system" provides as many or as varied opportunities for fraud -- most of which cannot be proven to be fraud. The "oops" defense is alive and well all over the place, not just in ES&S ballot programming.

And then we have computerized elections:

1. The very nature of software and the software development process renders software incompatible with the voting system testing and certification process.

2. The defensive self-interest of the very people who have the potential to improve the system causes them to refuse to do so. Pilots, for example, WANT to report airplane failures. They have no motivation to hide problems; quite the contrary. They don't point out that "no passenger was lost" so engine #3 blowing up wasn't actually a problem after all -- just a glitch. 'Cause the pilot doesn't want to fly it again until the "glitch" is fixed. Not so with many election officials and their beloved voting equipment. They're happy to fly it again even if the glitch isn't fixed. (Incidentally, the pilot has a basis in reality for claiming that no passenger was lost, while the EO has no such basis in reality for claiming that no vote was lost.)

ELLEN quoting LANI: Consider what might have been IF Florida's Secretary of State had performed a thoughtful analysis of voting machine requirements before sanctioning the touchscreens in the first place. You know, the part about counting all votes, reliability, stability, auditability, proof.

ELLEN (resuming): Even to consider this, we have to suspend disbelief in:

- Money changing hands
- Unwillingness to admit ignorance of both the requirements and the touchscreens
- Peer pressure and the influence of the Election Center
- Political concerns trumping concern for honest elections
- Motivation to rig elections
- Me first, the people who elected me second

... and perhaps a bunch more items.

It's become difficult for me to suspend disbelief in any of them, let alone all of them.

Government simply will NOT protect citizens from itself. And this is where we, as a nation, have gone astray for decades. We forgot what the founders knew first hand, from real-life experience -- that we must at all times be vigilant to protect ourselves from government.

LANI: To your point on the "politics" of elections systems, I was asked in an interview if I believe another political party in charge would correct the election process. My immediate answer was, "Well, yes!" Then I stuttered and sputtered, "Errr... ahhh... I hope so." Do you think it would be different? I wonder if "the win" is too great a temptation.

You may recall the primary reason I turned to fiction was my humble attempt to spread the word to a reluctant audience/public who doesn't understand or believe the seriousness of our election failures. Hoping fiction would succeed where truth does not.

I am more convinced than ever that an interested and informed public is the greatest deterrent to election fraud and failure.

Ellen Theisen is the founder and Co-Director of VotersUnite.Org.

	Contact Author
	Contact Editor
	View Other Articles by Author

Bookmark this page: (what's this?)

(More...)

2 comments

Graduate of MIT and Stevens; 50 years as systems engineer on cutting edge projects, civilian and military; Fifth Air Force, WWII; sworn defender of the Constitution
abacusGraduate of MIT and Stevens; 50 years as systems engineer on cutting edge projects, civilian and military; Fifth Air Force, WWII; sworn defender of the Constitution

Two footnotes

Excellent, Mesdames!

Footnote 1. There is no way for an election official to know if the software in his/her machine is the same as the software that was "certified." [Skipping over the incompetence of the certifying process and entities and the conflicts of interest involved.]

A digital signature identifying the software? No; any check on the signature held in the machine can /would itself be subverted by an evildoer.

[Btw, I have read that if you have enough power [eg, a botnet under the control of our evildoer] software can be altered and the digital sgnature left unaffected...]

Compare the software in the machine with a true copy of the software as certified? No, for several reasons. The "true copy" may hav been subverted. Then, if not, what machine would do the comparison? Not the machine under test. And any other machine which might be used for the comparison itself may have been subverted.

Footnote 2. A fundamental theoretical proposition

‘Why NO computer system can ever be trusted with the vote counting”

By an information theoretician - out of my depth but not to be ignored...

Begin clips
I would like to lay out the fundamental argument against the use of computers anywhere in the chain of voting to vote counting. I don't take this from a luddite stance, but from a Information Theoretic stance. In Computer Science, we study what can and cannot be computed. We have found many limits in what computers can do. Unknown to most people, but there is no such thing as "verifying" a computer program. No process can detect all possible outputs from any sufficiently large computational system. Computer Scientists know this from very clever and fundamental arguments about how computation happens.

In "An Undetectable Computer Virus" David Chess and Steven White show that you can always create a vote changing program (called virus there) that no "verification software" can ever detect. They do this by a very clever argument which you can pursue in that paper, but the important thing to realize is that their results are not in doubt. You also should know that
these arguments apply to every computer system that can ever be created. Therefore, if you use a computer anywhere in the vote counting process, you cannot be certain of the result.

I would like to point out that as much as I personally like open source software, it too falls victim to this very same argument. No computer is immune. No computer that ever will be constructed is immune. It is as impossible as traveling faster than the speed of light. Every computer system invented or YET TO BE INVENTED has this flaw. It can never work.

So, given that we can logically deduce that we can never trust computer voting, we are left with hand counting. We can use computers to double check our mathematics, but the marks should be on paper, counted by hand, tallied by hand, and only then added to a computer. It will be a chore, but it is the chore of representative rule.

End clips
Check the paper out at:

For lively discussion see
click here

by abacus (2 articles, 2 quicklinks, 4 diaries, 58 comments) on Friday, May 2, 2008 at 1:30:21 AM