Thursday, July 26, 2012

Omnivark’s Time and Place

Today saw the final edition of Omnivark, my personal project to teach a computer to identify great writing. Each day, Omnivark would pick three pieces of new, nonfiction writing on the Web, plus a book. I’m proud of Omnivark’s quality during its six-month run. (Feel free to click a random day from the archive, and see what you think.)

So why stop now? Omnivark was something different and fun to do during the past months while I was also doing consulting. However, that period was an in-between time, from when I completed the Intelligent Cross-Sell integration at RichRelevance until my family’s move back to the Bay Area, which is happening in August. Soon after, I’ll be resuming my normal career—more on that in a future post.

Suffice to say, Omnivark was of a certain place and time, which are changing. I enjoyed creating it; I hope you enjoyed reading it. If you did, here are some suggestions for alternatives to keep your tank topped with great reads.

And finally, for those interested in how the technology worked:

Omnivark’s Division of Labor

As I taught a computer to recognize great writing, the division of labor between me (the teacher) and the computer changed over time. The Omnivark software started relatively stupid and ended relatively smart.

At the beginning, I was heavily influencing Omnivark’s daily picks of great writing on the Web. This was necessary to create a training set of great writing for Omnivark’s algorithms, so they could learn to find other great writing. Over time, and a lot of experimentation, Omnivark became smart enough so it could do most of the work in choosing great reads for an edition.

But to be clear, there was a big distance between Omnivark’s doing most versus all of the work. Most of the work was Omnivark’s finding and scoring hundreds—sometimes over a thousand—of candidate pieces a day. I still needed to pick the best of Omnivark’s best, factoring in issues like diversity of topics and sources.

Unless I was working on the algorithms, I would need to read only perhaps ten of the top-scoring candidates. From them, I often was able to find two of the three Web picks for an edition. I’d find the third pick by scanning further down the list of candidates for an interesting headline, or I would find it from my own normal Web browsing, or I’d occasionally get a great suggestion from an Omnivark reader.

The Omnivark algorithms were capable of rating not only an entire piece but also individual sentences. Once in a while (maybe one in ten times), I’d agree with Omnivark’s choice for the best sentence to use as a quote from the piece. The low agreement was due to Omnivark’s simply judging a sentence for artfulness, whereas I was also judging how well a sentence indicated what the piece was about. Also, I could easily see when multiple sentences, or fragments of sentences, were better than the best sentence—a far from easy task for software.

The fourth and final pick in every Omnivark edition was from a book. This turned out to be a distinct challenge because book excerpts are often in PDFs or special viewer applications such as Amazon’s “Look Inside.” Automating their extraction was different enough from everything else I was doing that I ended up picking the books manually, guided by high user reviews and official endorsements.

So, except for the book picks, the division of labor shifted nicely from human to computer. In terms of replacing a human’s hours, Omnivark got maybe 90% of the way there. However, that last 10%’s hours are a lot harder to automate than the first 90%’s. They involve creative judgment such as knowing when different picks go well together, or recognizing that a certain sentence captures the essence of a larger point. Perhaps someday a computer will do that too, but there will always be the need to model specific humans’ judgments; otherwise, it would be like having the same editor for every magazine. In that sense, humans will always be the teachers.