Imagine a world in which the robots we see dancing on viral videos do so, not because they were programmed to, but because they had seen it in a video. Computer Science PhD alumnus Georgios Pavlakos seeks to help make this a reality.
The current BAIR postdoc has received the 2021 Morris and Dorothy Rubinoff Award for his project titled “Learning to Reconstruct 3D Humans,” advised by Professor Kostas Daniilidis. The School of Engineering and Applied Science presents this award yearly “to a graduate degree candidate whose dissertation has resulted in or could lead to innovative applications of computer technology.”
“The goal is to make systems [that are] automatically able to perceive the human pose, humans from images, the way a human would,” said Pavlakos. “We started with more coarse representations, using just the hip points to represent the main joints of the human body, then increasing the detail, reconstructing the full surface of the body, and then finally, going into even more detail and expressive representations.”
The findings of such work can mean amazing breakthroughs in the realm of American Sign Language interpretation. A system would be able to look at the video, translate the human pose, and then map that pose to language, according to Pavlakos. Successful practical application of the research would also produce the opposite: a system able to translate language into real-time model animations.
Although the work is relatively far from these solid outcomes, crucial foundations have been laid.
“We can capture facial expressions, we can capture hand gestures,” said Pavlakos. “These are crucial to get the subtleties of sign language.”
Pavlakos has always been excited by the possibility of creating systems that can perceive human physicality without hardcode, but Kostas Daniilidis has made this particular project possible.
“This came from the excitement of my advisor, Kostas Dan***,” said Pavlakos. “This was a line of work and direction that he was always very excited about, a line of work that was active in the lab.”
The possible practical applications, as well as the thrill of scaling empirical mountains, is what continues to reel him in.
“The pure challenge of the problem, scientifically, and the excitement of the things that you can do with a system like that: this kind of problem used to always amaze me,” said Pavlakos. “And it still does. “