The Pure Challenge of Perception: Georgios Pavlakos receives Rubinoff Award for his work in 3D reconstruction

Courtesy of Georgios Pavlakos

Imagine a world in which the robots we see dancing on viral videos do so, not because they were programmed to, but because they had seen it in a video. Computer Science PhD alumnus Georgios Pavlakos seeks to help make this a reality.

The current BAIR postdoc has received the 2021 Morris and Dorothy Rubinoff Award for his project titled “Learning to Reconstruct 3D Humans,” advised by Professor Kostas Daniilidis. The School of Engineering and Applied Science presents this award yearly “to a graduate degree candidate whose dissertation has resulted in or could lead to innovative applications of computer technology.”

“The goal is to make systems [that are] automatically able to perceive the human pose, humans from images, the way a human would,” said Pavlakos. “We started with more coarse representations, using just the hip points to represent the main joints of the human body, then increasing the detail, reconstructing the full surface of the body, and then finally, going into even more detail and expressive representations.”

The findings of such work can mean amazing breakthroughs in the realm of American Sign Language interpretation. A system would be able to look at the video, translate the human pose, and then map that pose to language, according to Pavlakos. Successful practical application of the research would also produce the opposite: a system able to translate language into real-time model animations.

Courtesy of Georgios Pavlakos

Although the work is relatively far from these solid outcomes, crucial foundations have been laid.

“We can capture facial expressions, we can capture hand gestures,” said Pavlakos. “These are crucial to get the subtleties of sign language.”

Pavlakos has always been excited by the possibility of creating systems that can perceive human physicality without hardcode, but Kostas Daniilidis has made this particular project possible.

“This came from the excitement of my advisor, Kostas Dan***,” said Pavlakos. “This was a line of work and direction that he was always very excited about, a line of work that was active in the lab.”

The possible practical applications, as well as the thrill of scaling empirical mountains, is what continues to reel him in.

“The pure challenge of the problem, scientifically, and the excitement of the things that you can do with a system like that: this kind of problem used to always amaze me,” said Pavlakos. “And it still does. “

Click here to view the full dissertation.


Cyber comfort and security: Sebastian Angel receives NSF Career Award

Raj and Neera Singh Term Assistant Professor Sebastian Angel is the latest in a line of astute educators to be granted funding through the NSF’s Faculty Early Career Development Program.

Angel and his team received the funds, roughly $125k received to date, to continue research on their project “Tools for building online services that hide metadata“.

“The project is all about finding different ways to give companies the tools that they need so they can continue to operate as they do today, but without having to acquire as much information about users,” said Angel. “For example, how can we build social networks where Facebook doesn’t have to know everything about you or your friends.”

According to the project abstract, although it is “grounded by the needs of companies, this project designs algorithms, tools, and infrastructure to help services continue to work without accessing or collecting certain metadata (e.g., the social graph), limiting the harm of data breaches and insider attacks.”

While benefits to the user that include both comfort and security are apparent, the next phase of the project will hone in on benefits for companies, said Angel. Tracking these metrics is profitable for companies for many reasons.

“One reason they do this is because they want to get a better understanding of what content is popular, so they can promote it,” said Angel. “The point of this research is can we still allow companies to provide relevant advertising, to figure out which sites or which articles are popular, without the provider necessarily learning information about individuals.”

The research aims to confront the challenge of scaling up, as well as one more specific: “how much information can we actually give [companies], while still providing strong privacy guarantees for users?” said Angel.

When the ideal balance is struck, users could possibly plug this work into their lives in many ways, including their streaming and online news experiences.

“You go to the New York Times and you want to read an article,” said Angel. “You might want to do that without revealing to the New York Times which article you’re reading.”

The next phase of the project, to be funded with this latest NSF grant, will focus on enticing companies with mockups of what their services could be.

“So now we’re working towards building this online advertising ecosystem,” said Angel. “Then we want to work on building versions of existing services that don’t require collecting metadata, but still work as users would expect.”

“Our hope is that once [companies] see that it is possible, that privacy and financial viability are not necessarily in tension, then they might be willing to employ some of our ideas.”


Kevin Johnson Appointed Penn Integrates Knowledge University Professor

Kevin Johnson has been named the University of Pennsylvania’s 27th Penn Integrates Knowledge University Professor (PIK).

Professor Johnson joins a distinguished group of interdisciplinary educators who have been appointed to two or more Penn Schools since Pres. Amy Gutmann launched the PIK initiative in 2005. The program’s agenda: “to recruit to Penn renowned scholars whose work draws from two or more academic disciplines and whose achievements demonstrate a rare ability to thrive at the intersection of multiple fields.”

Johnson will start in November with joint appointments in the Department of Biostatistics, Epidemiology, and Informatics (DBEI) and CIS, a secondary appointment with the Annenberg School for Communication, and will also serve as Vice President for Applied Informatics at UPHS and Professor of Pediatrics at CHOP.

“Kevin Johnson is a gifted physician-scientist who has harnessed and aligned the power of medicine, engineering, and technology to improve the health of individuals and communities. He has championed the development and implementation of clinical information systems and artificial intelligence to drive medical research, encouraged the effective use of technology at the bedside, and empowered patients to use new tools to better understand how medications and supplements may affect their health. He is a board-certified pediatrician, and his commitment to patient health and welfare knows no age limits. In so many different settings, Kevin’s work is driving progress in patient care and improving our health care system. He is a perfect fit for Penn, where our goal is to create a maximally inclusive and integrated academic community to spur unprecedented global impact.”

Penn President Amy Gutmann, as featured in Penn Today


Vraj Shroff’s Magic Connects allows students to choose the connection they want

A cool and colorful illustration from the Magic Connects website.

The past year has brought us no shortage of resourceful students and faculty developing tools to help mend the social rifts caused by the pandemic. In January, Vraj Shroff (MSE in SE, BSE in CSCI) and two of his teammates joined those ranks.

The 3 students developed a new website and app called Magic Connects. Earlier in the year, Shroff told the Daily Penn that the platform is “distinct from other popular platforms such as LinkedIn, Snapchat, and Tinder, because it matches students based on their interests.”

The article reported that, at the time of publication, Magic Connects had received over 100 pre-registered students, one month before its launch. CIS decided to check in with a few of the platform users several months after its release for some feedback.

Varun Jana, Junior in CIS
“I actually don’t know of any other platform that kind of asked me about what kind of relationships I want and what kind of people I want to connect with. Magic Connects asked me if I want to meet the people from the same school or clubs or majors specifically.”

Joy Cai, Senior in Business
“You can do friendship, dating and professional connection and each of these relationships are concrete and already established so everyone knows why they’re there and there’s no ambiguity. It’s all one single platform, so it makes it all the more powerful because it’s convenient and it has multiple uses and all of the users are specified.”

Anannya Shandilya. Senior in Finance
“A lot of other media platforms like Facebook encouraged connection, but virtually. In Magic Connects it has this feature where we can select that, I want to meet people in my own university it’s kind of taking over that barrier, because once i’m connected with people, then I can share your meetings. I think that’s a really good advantage that’s not there on other platforms.”

Radhik Patel, Wharton ’22
“The reason that I joined Magic next is because I’m really interested in entrepreneurship and it’s really helped me connect with other students who have similar interests at different universities. I’m part of the entrepreneurship club, but that only limits me to the students that are here at Penn. I think the scale at which I can meet other individuals who are like-minded as me is something that certainly helped me meet new people and build my network.”

Vraj Shroff, Founder, shares key updates based on user feedback:
“I put a lot of thought in the security side of it and making sure it’s a secure and safe community when they’re using this platform. A few of my female friends told me they don’t feel secure sharing their location, so now, you can enter your location, rather than allowing your phone to access a location. Now you have to sign up with your school email so it’s not up to you to make sure there’s a real person on the other side of your phone, and you’re talking to a student.”


Prof. Thomas Farmer steps up as MCIT Online Program Director

Professor Thomas Farmer has served the CIS and ESE Departments as a lecturer in some capacity for almost a decade. Now CIS is pleased to announce that, in addition to to his role as Senior Lecturer, Prof. Farmer has stepped into the role of Program Director for MCIT Online.

His course CIS 593, “Introduction to Computer Systems,” is a student favorite and, according to CIS Department Chair Zach Ives, “one of the highest rated courses in MCIT Online.” 

Prof. Farmer — whose focus areas of research have included RFIC, power amplifiers and mixed-signal design — teaches mainly courses that center hardware development and functionality.

The CIS Department has adapted many successful strategies from MCIT in our transition to remote learning, and Prof. Farmer has been a valuable member of MCIT since its inception. Prof. Ives notes that, in addition to strengthening student learning and community experiences, “he will also help shape the curriculum and admissions process.”

“We look forward to Tom’s leadership of the program, and thank him on behalf of our students,” said Ives. “Congratulations Tom!”