Let's Keep It Real! The "Imitation Game" AI
Developing a variety of tests to replace Alan Turing's 67 year old "imitation game" is a challanging task that was revered as the ultimate test of machine intelligence, so let's take a look at and evaluate how bots are filling in the void.
What Are Bots? "Robots, automated processes"
What Is A Rubicon? A measure of whether machines have truly arrived.
TEST 1 of 4
Winograd Schema Challenge
Name after pioneering AI researcher Terry Winograd, a "Winograd schema" is a simple but ambiguously worded natural-language question. Answering correctly requires a "commonsense" understanding of how agents, objects and cultural norms influence one another in the world.
Winograd's first schema, which he wrote in 1971, sets a sceen ("The city councilmen refused the demostrators a permit because they feared violence?"). This is known as a pronoun disambiguation problem (PDP): in this case, there is ambiguity about whom the word "they" refers to. But Winograd schemas are subtler than most. PDP's because the meaning of the sentence can be reversed by changing a single word. (For example:) "The city councilmen refused the demostrators a permit because the advocated violence). Most people use "common sense" or "world knowledge" about typical relationships between city councilmen and demostrators to solve this problem. This challenge uses an initial round of PDP's to weed out less intelligent systems; ones that make the cut are given true Winograd schemas.
Pros: Because Winngrad schemas rely on knowledge that computers lack reliable access to, the challages is robustly Google-proof - that is, hard to game with internet searches.
Cons: The pool of useable schemas is relatively small. " They're not easy to come up with," Say's Ernest Davis, a professor of computer science at New York University.
Difficulty Level: High. In 2016 four systems competed to answer a set of 60 Winograd schemas. The Winner got only 58 % of the questions correct-far short of the 90 % threshold that researchers consider a passing grade.
What is It Useful For: Dingstinguishing comprhension from mere simulation of it. "(Apple's digital assistant) Siri has no understanding of pronouns and cannot disambiguate," explains Leora Morgenstern, a resercher at Leidos who worked on the Winograd Schemas Challange with Davis. That means "you really can't carry on a dialouge (with the system), because you're always referring to something prevfious in the conversation.
ref. Scientific American, March 2017 (pg.59) Am I Human,Gary Marcus
Next Week Look Out For Test 02. Standarize Testing For Machines
