It's a robotics term, coined by Japanese roboticist Masahiro Mori. It is the point at which a human-looking robot is so human-looking, it starts to make you feel uncomfortable. Kind of like Arnold in Terminator.
As usual, I found this out at wordspy.
Early in their collaboration, in the spring of 2002, Winston and Breazeal selected a name: Leonardo, "because this creature represents the ideal collaboration of art and science—an artist and a scientist working together to create something real," Winston said. Then, in Los Angeles, Winston went to work on Leo's body and face. One of the few guidelines from Breazeal was that Leo not look too human, lest he fall into the "uncanny valley," a concept formulated by Masahiro Mori, a Japanese roboticist. Mori tested people's emotional responses to a wide variety of robots, from non-humanoid to completely humanoid. He found that the human tendency to empathize with machines increases as the robot becomes more human. But at a certain point, when the robot becomes too human, the emotional sympathy abruptly ceases, and revulsion takes its place. People began to notice not the charmingly human characteristics of the robot but the creepy zombielike differences. —John Seabrook, "It cam from Hollywood," The New Yorker, December 1, 2003