http://www.technologyreview.

com/view/538111/why-and-how-baidu-cheated-an-artific
ial-intelligence-test/
Tom Simonite
June 4, 2015
Why and How Baidu Cheated an Artificial Intelligence Test
Machine learning gets its first cheating scandal.
The sport of training software to act intelligently just got its first cheating
scandal. Last month Chinese search company Baidu announced that its image recogn
ition software had inched ahead of Google s on a standardized test of accuracy. On
Tuesday the company admitted that it achieved those results by breaking the rul
es of that test.
The academic experts who maintain that test say that makes Baidu s claims of beati
ng Google meaningless. Ren Wu, the Baidu researcher who led work on the software
in question has apologized and said the company is reviewing its results. The c
ompany has amended a technical paper it released on its software.
We don t know whether this was the action of one individual or a strategy of the t
eam as a whole. But why a multibillion dollar corporation might bother to cheat
on an obscure test operated by academics on a voluntary basis is actually quite
clear.
Baidu, Google, Facebook and other major computing companies have spent heavily i
n recent years to build research groups dedicated to deep learning, an approach
to building machine learning software that has made great strides in speech and
image recognition. These companies have worked hard to hire leading experts in t
he small field - often from each other (see "Is Google Cornering the Market on D
eep Learning"). A handful of standardized tests developed in academia are the cu
rrency by which these research groups compare one another s progress and promote t
heir achievements to the public.
Baidu got an unfair advantage by exploiting the test s design. To get your softwar
e scored against the ImageNet Challenge you first train it with a standardized s
et of 1.5 million images. Then you submit the code to the ImageNet Challenge ser
ver so its accuracy can be tested on a collection of 100,000 "validation" images
that the software has never seen before.
The Challenge rules state that you must only test your code twice a week, becaus
e there s an element of chance to the results.
Baidu has admitted that it used multiple email accounts to test its code roughly
200 times in just under six months - over four times what the rules allow.
Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence, likens wha
t Baidu did to buying multiple lottery tickets. "If you get to buy two tickets a
week you have a certain chance if you buy 200 a week you have more of a chance,
" he says. On top of that, testing slightly different code over many tests could
help a research team optimize its software for peculiarities of the collection
of validation images that aren t reflected in real world photos.
Such is the success of deep learning on this particular test that even a small a
dvantage could make a difference. Baidu had reported it achieved an error rate o
f only 4.58 percent, beating the previous best of 4.82 percent, reported by Goog
le in March. In fact, some experts have noted that the small margins of victory

in the race to get

ss. That Baidu and
ay even be willing
earning matters to

better on this particular test make it increasingly meaningle

others continue to trumpet their results all the same - and m
to break the rules - suggest that being the best at machine l
them very much indeed.

CONTINUE
HP Puts the Future of Computing On Hold
The Emerging Science of Human Computation
First Demonstration of a Surveillance Camera Powered by Ordinary Wi-Fi Broadcast
s
Virtual Eyes Train Deep Learning Algorithm to Recognize Gaze Direction
Recommended from Around the Web (Week Ending May 30, 2015)
Is This the First Computational Imagination?
Computational Aesthetics Algorithm Spots Beauty That Humans Overlook
2 comments
andrewppp 4 hours ago
In the domain of high tech the Chinese are well-known for widespread stealing. W
hy then would cheating come as any surprise?
dobermanmacleod 7 hours ago
What surprises me is that you can't supercharge any neural net image recognition
software with linked GPU hardware.
http://www.wired.com/2015/02/hot-yet-little-known-trend-thatll-supercharge-ai/
Frankly, I think that big companies like Baidu aren't poised to take the lead in
AI, because of the nature of creative R&D. Ossification and inside-the-box thi
nking is rampant at such corporations, and even if they do hire the best talent,
odds are they will mismanage it.