Omnivark’s Division of Labor

As I taught a computer to recognize great writing, the division of labor between me (the teacher) and the computer changed over time. The Omnivark software started relatively stupid and ended relatively smart.

At the beginning, I was heavily influencing Omnivark’s daily picks of great writing on the Web. This was necessary to create a training set of great writing for Omnivark’s algorithms, so they could learn to find other great writing. Over time, and a lot of experimentation, Omnivark became smart enough so it could do most of the work in choosing great reads for an edition.

But to be clear, there was a big distance between Omnivark’s doing most versus all of the work. Most of the work was Omnivark’s finding and scoring hundreds—sometimes over a thousand—of candidate pieces a day. I still needed to pick the best of Omnivark’s best, factoring in issues like diversity of topics and sources.

Unless I was working on the algorithms, I would need to read only perhaps ten of the top-scoring candidates. From them, I often was able to find two of the three Web picks for an edition. I’d find the third pick by scanning further down the list of candidates for an interesting headline, or I would find it from my own normal Web browsing, or I’d occasionally get a great suggestion from an Omnivark reader.

The Omnivark algorithms were capable of rating not only an entire piece but also individual sentences. Once in a while (maybe one in ten times), I’d agree with Omnivark’s choice for the best sentence to use as a quote from the piece. The low agreement was due to Omnivark’s simply judging a sentence for artfulness, whereas I was also judging how well a sentence indicated what the piece was about. Also, I could easily see when multiple sentences, or fragments of sentences, were better than the best sentence—a far from easy task for software.

The fourth and final pick in every Omnivark edition was from a book. This turned out to be a distinct challenge because book excerpts are often in PDFs or special viewer applications such as Amazon’s “Look Inside.” Automating their extraction was different enough from everything else I was doing that I ended up picking the books manually, guided by high user reviews and official endorsements.

So, except for the book picks, the division of labor shifted nicely from human to computer. In terms of replacing a human’s hours, Omnivark got maybe 90% of the way there. However, that last 10%’s hours are a lot harder to automate than the first 90%’s. They involve creative judgment such as knowing when different picks go well together, or recognizing that a certain sentence captures the essence of a larger point. Perhaps someday a computer will do that too, but there will always be the need to model specific humans’ judgments; otherwise, it would be like having the same editor for every magazine. In that sense, humans will always be the teachers.