Methodology Used by the Modern Library to Choose the 100 Best Non-Fiction Books of the Century
As we approach the end of the century and the end of the millennium, we are seeing many "best" lists of the century. Many of the "greatest" items of the last 100 years are only favored as the "best" by a sliver of those voting. For example, in balloting for best actor/actress, a recent poll deemed John Wayne and Katherine Hepburn our "favorites" despite each of them only receiving 10 percent and 12 percent of votes respectively.
Last year, the Modern Library’s list of the greatest 100 works of fiction created much debate and was deeply criticized for its omissions.
This year, the Modern Library produced a list of the 100 greatest non-fiction English-language works of the century, but, recognizing they needed to produce a more balanced list than last year’s fiction selections, they employed a different methodology to arrive at the final 100. The methodology this year evolved as the process continued.
Modern Library’s approach this year shows there are alternative voting systems that work for these types of lists. Their experience last year proved to them that they needed to find a better way to make their choices.
Last Year
For last year’s fiction list, the Modern Library used the following process – which was a form of "approval voting":
A small group of people – the Editorial Board – assembled a list of 400 novels. This list of 400 was compiled and sent back to them. The Editorial Board then assigned "yes" or "no" votes to each title as to whether it should be on the list of 100. The books with the most "yes" votes were put on the list, the others were not.
One problem with this approach is that they started off with a very limited number of titles to choose from, 400. Another problem was that it was a simple binary "yes" or "no" and there was no room to assign weight to each book.
This Year
For this year’s non-fiction list, the Modern Library used the following process:
They expanded both the pool of titles to choose from (to 900 titles), and the number of people submitting titles for consideration. Instead of a few people recommending titles like last year, dozens of editors as well as the 13-member Editorial Board submitted titles for the ballot.
Because they wanted to spark an interest in books for the general leisure reader, they chose two criteria, that 1) the works be literary; and 2) the books must have intellectual merit. This was not a list of the 100 most influential books, which might have been different – for instance, you will not find Dr. Spock’s baby book or The Joy of Cooking on this list.
The ballot was sent to the Editorial Board – which had been expanded to 13 members to make a more diverse committee. They added two young writers, a black male author, and two women to the committee to compensate for the overwhelmingly older, white male committee. Titles were listed alphabetically, and it was not a simple "yes" or "no" vote for each book. Rather, committee members were asked to rank the book on a scale from –10 to 10. A zero would be counted as neutral. This gave the committee members the ability to vote either for or against a book and allowed them to assign values to the books. Another rule specified that the committee could only vote on books they knew.
When the scores were tabulated, each book had an average score. This allowed the Modern Library to cut the list to 300 books. At this point, they still had to whittle down the list to 100 books, but before they could vote to cut the list by two-thirds, they realized they had a statistical problem. The problem was that since committee members did not each have the same knowledge of each book, and they were required to skip books they had no knowledge of, the books each had different numbers of people voting on them. Some books may have received a very high score, but only one person voted for the book.
At this point, the Modern Library consulted a University of Chicago statistics professor, who devised a system to fairly pare the list to 100 books.
The ballot listing the 300 books was sent out to the committee members, but this time, they could only vote on a scale of 0-10. Further, there were a limited amount of 10s that each member could assign (they could only give out 35). This was designed to make the votes more consistent with one another, since initially, where one Board member assigned half the books 10s, another Board member assigned no 10s. This way, members had to be thoughtful about giving out 10s.
The professor took the data and ranked each book according to several criteria to arrive at an accurate picture:
Voila! And there we have the 100 "best" non-fiction English-language works of the 20th Century, from The Education of Henry Adams and The Souls of Black Folk to Silent Spring and The Elements of Style.