It's Not Just Practice AI, It's Real!
Test 02. Standarize Testing For Machines
www.WeSTEMulate.Org
AI would be given the same standardized, written educational test tha we give to elementary and middle school students. without any hand holding. This is not just practice, It's real AI!
Identifying the nuances of improving performance, should be the goal here. The method would assess a machine's ability to link facts together in novel ways through semantic understanding. Much like Turing's original imitation game, the scheme is ingeniously direct.
Simply take any sufficiently rigorous standardized test (such as multiple-choice parts of New York State's fourth-grade Regents science exams), equip the machine with a way of ingesting the test material (such as natural language processing and computer vision) and let'er rip.
Pros: Versatile and pragmatic. Unlike Winograd schemas, standardized test material is cheap and abundant. And because none of the material is a adapted or processes for the machine's benifit, test questions require a wealth of versitile, commonsense world knowledge just to parse, much less answer correctly.
Cons: Not as Google-proof as Winograd schemas, and as with humans, the ability to pass a standardized test does not necessarily imply "real" intelligence.
Difficulty Level: Moderately high. A system called Aristo, designed by the Allen Institute for Artificial Intelligence, achieves an average 75 percent score on the forth-grade science exams that its has not encountered before. But this is only on multiple choice questions without diagrams. "No system to date comes even close to passing a full 4th grade science exam," the Allen Institute researchers wrote in a technical paper published in AI magazine.
What Is It Useful For: Administering reality checks. "Fundamentally, we can see that no program can get above 60 percent on and eight grade science test, but at the same time, we might read in the news that IBM's Watson is going to medical school and solving cancer, "says Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence.
"Either IBM had some startling breakthrough, or perhaps they're getting a bit ahead of themselves.
Scientific American, March 2017, John Pavlus
