Ken Jennings Reflects on Watson Matches

***CaptHayfever*** · 02-18-2011, 03:54 AM

For those who missed it, this week on Jeopardy, former champions Ken Jennings (74 straight wins, $2+ mil) & Brad Rutter (undefeated, $3+ mil) played a 2-game exhibition tourney against Watson, a mainframe built by IBM specifically to be able to understand regular language well enough to play Jeopardy.
Spoiler Below

Watson won both games.
Ken placed 3rd & 2nd respectively, placing 2nd overall.
Brad placed 2nd & 3rd respectively, placing 3rd overall.

Ken then wrote this essay about the experience for Slate. (He discusses the results, so those who want to wait & see a rerun or YouTube of the games should hold off on reading this.) His Game 2 Final Jeopardy response also sums things up pretty amusingly.

The computer showed some interesting behavior during the show as well:
--It cannot see or hear (it is sent the clues as txt files as Trebek reads them), so a/v clues are out of the question. This handicap also means it can (& did) repeat an opponent's wrong answer.
--While it is excellent at parsing context as intended, it struggles with figuring out the desired answer type. On at least one clue, Watson was wrong only for not giving a complete enough response.
--The IBM programmers set it only to ring in if it reached a "buzz threshold" of certainty in its results. Watson's uncertainty actually resulted in most of the human players' right answers. On Daily Doubles & Final Jeopardy, Watson is forced to give its best guess if the buzz threshold is not reached.
--Speaking of Daily Doubles & Final Jeopardy, the computer also makes some hilariously-precise wagers. Protip: The only ones-digits you ever need for betting on Jeopardy are 0, 1, & 9.
--Watson's biggest advantage, though, was physical reaction time. You could see that Ken & Brad each knew a lion's share of the answers, often faster than Watson, but a hydraulic thumb which can consistently ring in the microsecond the clue ends is kind of unfair. Jennings is known for being a quick-draw (it's how he lasted to a 75th game), it took him until Game 2 to finally start beating Watson to the buzz.

And remember, "I'm-a Luigi, number one!"

**X-3** · 02-18-2011, 08:39 PM

Quote:

Originally Posted by Ken

Watson has lots in common with a top-ranked human Jeopardy! player: It's very smart, very fast, speaks in an uneven monotone, and has never known the touch of a woman.

Funny guy.

**Valigarmander** · 02-18-2011, 08:43 PM

How long before IBM changes its name to Skynet?

***Ace Mercury*** · 02-19-2011, 02:36 PM

I loved Trebek's reponse when Watson repeated a wrong answer from Ken; it was a smarmy "No, Ken said that

".

Yeah, Watson's wager amounts are hilarious. Also hilarious: guessing final Jeopardy catagory "U.S. Cities" with "What is Toronto?????"

***CaptHayfever*** · 02-20-2011, 01:06 AM

Yeah, I was kinda hoping for a screwup like that in Game 2 as well. I also like how it qualifies its guesses when it was forced to answer without certainty on that Final (the question marks) & on one of the Daily Doubles ("I will take a guess").
It's obvious that Watson does not include the categories in its search parameters, too, only the clue itself.

And remember, "I'm-a Luigi, number one!"

***Ace Mercury*** · 02-20-2011, 11:06 AM

Right. It was after the fact that I read up on some links off of the Metafilter thread re: this; apparently including the category screwed up the answers for some reason.

Irregardless, I think this is pretty exciting stuff, this Watson answer generator. Even if the quality stays the same, this sort of answer generator would be an awesome digital assistant. In a few years, Watson's going to shrink down from a room full of server racks into a more portable device, and we'll have a portable navi/agent-avatar. (Now that I think about it, it doesn't have to shrink; just port all of the answers over the cloud or something.) Working together with a human to weed out stupid stuff, I think it's a great advance for personal efficiency.

02-18-2011, 03:54 AM	#1
*CaptHayfever* Smash SuperMod Join Date: Jul 2002 Location: (n) - the place where I am Gender: Posts: 27,661 Thanks: 1,991 Thanked 2,486 Times in 1,513 Posts	Ken Jennings Reflects on Watson Matches For those who missed it, this week on Jeopardy, former champions Ken Jennings (74 straight wins, $2+ mil) & Brad Rutter (undefeated, $3+ mil) played a 2-game exhibition tourney against Watson, a mainframe built by IBM specifically to be able to understand regular language well enough to play Jeopardy. Spoiler Below Watson won both games. Ken placed 3rd & 2nd respectively, placing 2nd overall. Brad placed 2nd & 3rd respectively, placing 3rd overall. Ken then wrote this essay about the experience for Slate. (He discusses the results, so those who want to wait & see a rerun or YouTube of the games should hold off on reading this.) His Game 2 Final Jeopardy response also sums things up pretty amusingly. The computer showed some interesting behavior during the show as well: --It cannot see or hear (it is sent the clues as txt files as Trebek reads them), so a/v clues are out of the question. This handicap also means it can (& did) repeat an opponent's wrong answer. --While it is excellent at parsing context as intended, it struggles with figuring out the desired answer type. On at least one clue, Watson was wrong only for not giving a complete enough response. --The IBM programmers set it only to ring in if it reached a "buzz threshold" of certainty in its results. Watson's uncertainty actually resulted in most of the human players' right answers. On Daily Doubles & Final Jeopardy, Watson is forced to give its best guess if the buzz threshold is not reached. --Speaking of Daily Doubles & Final Jeopardy, the computer also makes some hilariously-precise wagers. Protip: The only ones-digits you ever need for betting on Jeopardy are 0, 1, & 9. --Watson's biggest advantage, though, was physical reaction time. You could see that Ken & Brad each knew a lion's share of the answers, often faster than Watson, but a hydraulic thumb which can consistently ring in the microsecond the clue ends is kind of unfair. Jennings is known for being a quick-draw (it's how he lasted to a 75th game), it took him until Game 2 to finally start beating Watson to the buzz. And remember, "I'm-a Luigi, number one!"

02-18-2011, 08:43 PM	#3
Valigarmander The Bee's Knees Join Date: Jun 2006 Location: The land of rain and trees (Oregon) Gender: Posts: 29,755 Thanks: 1,649 Thanked 5,701 Times in 2,581 Posts Blog Entries: 20	How long before IBM changes its name to Skynet?

02-19-2011, 02:36 PM	#4
*Ace Mercury* Marshmallow Knight ☆ Supermod Join Date: Jun 2000 Location: Southern Ontario Gender: Posts: 23,275 Thanks: 568 Thanked 3,297 Times in 1,582 Posts Blog Entries: 1	I loved Trebek's reponse when Watson repeated a wrong answer from Ken; it was a smarmy "No, Ken said that ". Yeah, Watson's wager amounts are hilarious. Also hilarious: guessing final Jeopardy catagory "U.S. Cities" with "What is Toronto?????"

02-20-2011, 01:06 AM	#5
*CaptHayfever* Smash SuperMod Join Date: Jul 2002 Location: (n) - the place where I am Gender: Posts: 27,661 Thanks: 1,991 Thanked 2,486 Times in 1,513 Posts	Yeah, I was kinda hoping for a screwup like that in Game 2 as well. I also like how it qualifies its guesses when it was forced to answer without certainty on that Final (the question marks) & on one of the Daily Doubles ("I will take a guess"). It's obvious that Watson does not include the categories in its search parameters, too, only the clue itself. And remember, "I'm-a Luigi, number one!"

02-20-2011, 11:06 AM	#6
*Ace Mercury* Marshmallow Knight ☆ Supermod Join Date: Jun 2000 Location: Southern Ontario Gender: Posts: 23,275 Thanks: 568 Thanked 3,297 Times in 1,582 Posts Blog Entries: 1	Right. It was after the fact that I read up on some links off of the Metafilter thread re: this; apparently including the category screwed up the answers for some reason. Irregardless, I think this is pretty exciting stuff, this Watson answer generator. Even if the quality stays the same, this sort of answer generator would be an awesome digital assistant. In a few years, Watson's going to shrink down from a room full of server racks into a more portable device, and we'll have a portable navi/agent-avatar. (Now that I think about it, it doesn't have to shrink; just port all of the answers over the cloud or something.) Working together with a human to weed out stupid stuff, I think it's a great advance for personal efficiency.