Imagine a world in which the robots we see dancing on viral videos do so, not because they were programmed to, but because they had seen it in a video. Computer Science PhD alumnus Georgios Pavlakos seeks to help make this a reality.
“The goal is to make systems [that are] automatically able to perceive the human pose, humans from images, the way a human would,” said Pavlakos. “We started with more coarse representations, using just the hip points to represent the main joints of the human body, then increasing the detail, reconstructing the full surface of the body, and then finally, going into even more detail and expressive representations.”
The findings of such work can mean amazing breakthroughs in the realm of American Sign Language interpretation. A system would be able to look at the video, translate the human pose, and then map that pose to language, according to Pavlakos. Successful practical application of the research would also produce the opposite: a system able to translate language into real-time model animations.
Courtesy of Georgios Pavlakos
Although the work is relatively far from these solid outcomes, crucial foundations have been laid.
“We can capture facial expressions, we can capture hand gestures,” said Pavlakos. “These are crucial to get the subtleties of sign language.”
Pavlakos has always been excited by the possibility of creating systems that can perceive human physicality without hardcode, but Kostas Daniilidis has made this particular project possible.
“This came from the excitement of my advisor, Kostas Dan***,” said Pavlakos. “This was a line of work and direction that he was always very excited about, a line of work that was active in the lab.”
The possible practical applications, as well as the thrill of scaling empirical mountains, is what continues to reel him in.
“The pure challenge of the problem, scientifically, and the excitement of the things that you can do with a system like that: this kind of problem used to always amaze me,” said Pavlakos. “And it still does. “
“The project is all about finding different ways to give companies the tools that they need so they can continue to operate as they do today, but without having to acquire as much information about users,” said Angel. “For example, how can we build social networks where Facebook doesn’t have to know everything about you or your friends.”
According to the project abstract, although it is “grounded by the needs of companies, this project designs algorithms, tools, and infrastructure to help services continue to work without accessing or collecting certain metadata (e.g., the social graph), limiting the harm of data breaches and insider attacks.”
While benefits to the user that include both comfort and security are apparent, the next phase of the project will hone in on benefits for companies, said Angel. Tracking these metrics is profitable for companies for many reasons.
“One reason they do this is because they want to get a better understanding of what content is popular, so they can promote it,” said Angel. “The point of this research is can we still allow companies to provide relevant advertising, to figure out which sites or which articles are popular, without the provider necessarily learning information about individuals.”
The research aims to confront the challenge of scaling up, as well as one more specific: “how much information can we actually give [companies], while still providing strong privacy guarantees for users?” said Angel.
When the ideal balance is struck, users could possibly plug this work into their lives in many ways, including their streaming and online news experiences.
“You go to the New York Times and you want to read an article,” said Angel. “You might want to do that without revealing to the New York Times which article you’re reading.”
The next phase of the project, to be funded with this latest NSF grant, will focus on enticing companies with mockups of what their services could be.
“So now we’re working towards building this online advertising ecosystem,” said Angel. “Then we want to work on building versions of existing services that don’t require collecting metadata, but still work as users would expect.”
“Our hope is that once [companies] see that it is possible, that privacy and financial viability are not necessarily in tension, then they might be willing to employ some of our ideas.”
Johnson will start in November with joint appointments in the Department of Biostatistics, Epidemiology, and Informatics (DBEI) and CIS, a secondary appointment with the Annenberg School for Communication, and will also serve as Vice President for Applied Informatics at UPHS and Professor of Pediatrics at CHOP.
“Kevin Johnson is a gifted physician-scientist who has harnessed and aligned the power of medicine, engineering, and technology to improve the health of individuals and communities. He has championed the development and implementation of clinical information systems and artificial intelligence to drive medical research, encouraged the effective use of technology at the bedside, and empowered patients to use new tools to better understand how medications and supplements may affect their health. He is a board-certified pediatrician, and his commitment to patient health and welfare knows no age limits. In so many different settings, Kevin’s work is driving progress in patient care and improving our health care system. He is a perfect fit for Penn, where our goal is to create a maximally inclusive and integrated academic community to spur unprecedented global impact.”
A cool and colorful illustration from the Magic Connects website.
The past year has brought us no shortage of resourceful students and faculty developing tools to help mend the social rifts caused by the pandemic. In January, Vraj Shroff (MSE in SE,BSE in CSCI) and two of his teammates joined those ranks.
The 3 students developed a new website and app called Magic Connects. Earlier in the year, Shroff told the Daily Penn that the platform is “distinct from other popular platforms such as LinkedIn, Snapchat, and Tinder, because it matches students based on their interests.”
The article reported that, at the time of publication, Magic Connects had received over 100 pre-registered students, one month before its launch. CIS decided to check in with a few of the platform users several months after its release for some feedback.
Varun Jana, Junior in CIS “I actually don’t know of any other platform that kind of asked me about what kind of relationships I want and what kind of people I want to connect with. Magic Connects asked me if I want to meet the people from the same school or clubs or majors specifically.”
Joy Cai, Senior in Business “You can do friendship, dating and professional connection and each of these relationships are concrete and already established so everyone knows why they’re there and there’s no ambiguity. It’s all one single platform, so it makes it all the more powerful because it’s convenient and it has multiple uses and all of the users are specified.”
Anannya Shandilya. Senior in Finance “A lot of other media platforms like Facebook encouraged connection, but virtually. In Magic Connects it has this feature where we can select that, I want to meet people in my own university it’s kind of taking over that barrier, because once i’m connected with people, then I can share your meetings. I think that’s a really good advantage that’s not there on other platforms.”
Radhik Patel, Wharton ’22 “The reason that I joined Magic next is because I’m really interested in entrepreneurship and it’s really helped me connect with other students who have similar interests at different universities. I’m part of the entrepreneurship club, but that only limits me to the students that are here at Penn. I think the scale at which I can meet other individuals who are like-minded as me is something that certainly helped me meet new people and build my network.”
Vraj Shroff, Founder, shares key updates based on user feedback: “I put a lot of thought in the security side of it and making sure it’s a secure and safe community when they’re using this platform. A few of my female friends told me they don’t feel secure sharing their location, so now, you can enter your location, rather than allowing your phone to access a location. Now you have to sign up with your school email so it’s not up to you to make sure there’s a real person on the other side of your phone, and you’re talking to a student.”
Professor Thomas Farmer has served the CIS and ESE Departments as a lecturer in some capacity for almost a decade. Now CIS is pleased to announce that, in addition to to his role as Senior Lecturer, Prof. Farmer has stepped into the role of Program Director for MCIT Online.
His course CIS 593, “Introduction to Computer Systems,” is a student favorite and, according to CIS Department Chair Zach Ives, “one of the highest rated courses in MCIT Online.”
Prof. Farmer — whose focus areas of research have included RFIC, power amplifiers and mixed-signal design — teaches mainly courses that center hardware development and functionality.
The CIS Department has adapted many successful strategies from MCIT in our transition to remote learning, and Prof. Farmer has been a valuable member of MCIT since its inception. Prof. Ives notes that, in addition to strengthening student learning and community experiences, “he will also help shape the curriculum and admissions process.”
“We look forward to Tom’s leadership of the program, and thank him on behalf of our students,” said Ives. “Congratulations Tom!”
As men went off to WWII, women began to play an integral part in programming. Here are a few programmers working diligently with the ENIAC. Courtesy of eniacday.org.
On February 15, the global computing community celebrated the 75th anniversary of the launch of “Electronic Numerical Integrator and Computer,” or ENIAC for short. The machine, unveiled here at Penn in 1946, was the world’s first all-electronic, programmable computer.
CIS RCA Professor of Artificial Intelligence Emeritus Mitch Marcus fittingly opened his “History of ENIAC” webinar with a classic lecture ritual.
“I thought that, because I was a professor, I would give you guys a pop quiz,” said Prof. Marcus. “The nice thing is that it doesn’t get graded.”
“On February 15, 1946, the ENIAC was unveiled at the University of Pennsylvania. As part of that unveiling, the machine was demonstrated to those present. Watch a reenactment of that demonstration based on a press demonstration given and oral history collected by the ENIAC Programmers Project.” Courtesy of eniacday.org.
He asked of attendees via multiple choice: 1. How long was the ENIAC, 2. What was ENIAC’s total memory, and 3. What was the first program to run on ENIAC? Prof. Marcus not only provided the audience with the answers (1. About the size of a blue whale, 2. 20, 10-bit numbers, and 3. atomic bomb simulation), but went on to give a concise background of the legendary machine, from inspiration to execution.
Screenshots of the …errrr…animals(?) drawn in the game Fake Artist during the PhD Valentine’s Day party.
First-year CIS PhD student Alyssa Hwang was yearning for a bit of connection when she decided a Valentine’s Day celebration was in order.
“I kind of wanted an excuse to have a party and get together with everyone,” said Hwang. “I like having these big get-togethers on Zoom because it’s fun and I also get to meet people.”
Alyssa came aboard the CIS team in September of last year, and as such, has not had the opportunity to meet many of her peers in person. She teamed up with fifth-year CIS PhD student Omar Navarro Leija to spread a little Valentine’s Day cheer.
“Omar really helped a lot with the organization of the party,” said Hwang. “And I helped out with the game, because there was a really fun game I wanted to play with everyone.
The game, called Fake Artist, is popular within the relatively new realm of e-meetings and e-meetups. Alyssa likens it to other popular whodunit-type games such as Among Us and Spyfall, except with drawing.
Unfortunately, great games and company aren’t always enough to lure people to the party.
As part of the organizing duties, Omar felt it was a good idea to add some appealing incentives. They decided to give everyone $15 Grubhub vouchers, and also raffled off 10 VISA gift cards. On the CIS dime, of course.
“We used to have this weekly free food and beer event. It used to feed 75-90 people,” said Leija, of the CIS PhD TGIF gatherings that used to take place pre-pandemic. “That’s one of the reasons I don’t mind asking the department for money.”
Still, Alyssa and Omar are hoping that the word will spread throughout the department: the more the merrier.
“We have an entire Doctoral association of students dedicated to making these things happen,” said Hwang. “I wish people would tell us that they like things or what they want to see because we’re excited to do more.”
Professor Sharath Guntuku has not only been deepening his photography skills while working from home this past year, he’s also been reading a whole lot. Of Malcolm Gladwell, specifically.
“I’m really amazed by how he uses data to tell stories,” said Sharath. “That makes a lot of sense to me.”
Although he’s worked with UPenn as early as 2017, Sharath was brought onboard the CIS team as a Research Assistant Professor in May 2020. According to him, one of the most appealing and challenging aspects of the Penn CIS program is its lush interdisciplinary environment.
“I think access to experts in public health, experts in psychology, experts in psychiatry – I think it’s one of the few places in the US where we have this,” said Sharath.
And it is in these fields, most notably public health and psychology, that the points of Professor Guntuku’s research meld together. With regards to public health, he has been utilizing large scale social media data sets to track well-being over time.
“March 2020 had been really drastic in terms of how much people stopped moving,” said Sharath. “An average person in Philadelphia was moving about 2 miles a day before March 2020. And after March 2020, that reduced to…just the apartment.”
Various state and local governments, as well as Sharath and his research team, were curious about the effects that substantially less physical movement would have on mental health. Surveying, however, was too costly and time-consuming. So Sharath and his team turned to Twitter data.
“We were interested in using more real-time sources to track this more seamlessly,” said Sharath. “We looked at what people are saying in different counties, [and how] that reflected how they’re feeling. We started sharing this with 3 different state departments: Washington state, Pennsylvania and Colorado.”
While studying Twitter data is less expensive and requires less time than surveys, this method isn’t without its own technical difficulties.
“There are technical challenges in terms of how representative Twitter is because not everyone is on Twitter,” said Sharath. “It’s a skewed demographic, so how do you correct for it? It’s mostly building algorithms that can be effective in a public health space.”
Professor Guntuku rocking a cool (yet seemingly warm) Penn Med Center for Health Care Innovation jacket.
Another public health project that Sharath is working on involves identifying concerns that people have whenever public health emergencies occur. Many folks have reasons to doubt the public health info that’s widely available. Sharath cites the country’s mishandling of racial justice, and the contradicting mask advice that circulated early in the pandemic, as just a few examples.
“We’re looking at how we can use social media data to identify concerns that are very specific to local populations, and how we can tailor public health campaigns to address those concerns,” said Sharath.
That’s where the interdisciplinary nature of this work becomes crucial.
“I, as a computer scientist, don’t really know what messaging or communications are effective,” said Sharath. “But that’s why I work with people in Annenberg or say, Penn Medicine: to tell me what makes sense to the end user. My role here is to address challenges in developing algorithms that don’t propagate bias, but that give a reasonable presentation of what people are concerned about.”
Upon first meeting Professor Lyle Ungar and learning that at Penn, computer scientists, sociologists and psychologists often band together to work through solutions, Sharath was blown away.
“This has been sort of revolutionary for me because I did my graduate studies in Singapore,” said Sharath. “It was a pretty good school, but the level of interdisciplinary activity at Penn is at a completely different level.”
When asked what he was most looking forward to as a part of the CIS community, Sharath responded:
“I’m really excited about this intersection, how we build algorithms that actually have an impact, and how we can work with students that are interested in this intersection.”
The MCIT Bar in Gather.town: Where everybody knows mainframe
“What bar in the world has a bunch of nerds hanging out and doing stuff on whiteboards?” CIS Senior Lecturer and MCIT Director Arvind Bhusnurmath poses this as a rhetorical question.
The short answer is, one co-created by Bhusnurmath himself, and MCIT Program Coordinator Redian Furxhiu.
The MCIT Bar is a digital drink hole, of course, created within the CIS microcosm on the Gather.town platform. It features a snazzy night sky setting outside the windows (lest one need to comfort oneself for indulging in a 2PM e-drink) and program preset seating arrangements where each table serves as placings for private conversations.
While Bhusnurmath uses Gather.town primarily for office hours, the e-bar has allowed MCIT students and faculty to interact and socialize on a less regulated level. He believes that the student queue outside of his digital office doesn’t function as well as the real deal.
“The part that’s missing? I’m not sure if they’re talking to each other while in the line,” says Bhusnurmath of the usual hubbub and informal info passing that occurs during office hours.
So, the MCIT Bar provides a space where students and faculty can meet with the intent to connect socially over academic interests. Although whiteboard features like the one in Gather.town do exist on Zoom and other platforms, the one in Gather.town seems to encourage an easier flow of natural communication, that for Professor Arvind, more closely mimics the use of an actual whiteboard in an actual classroom.
The whiteboards, Bhusnurmath’s personal design addition, play a key part in the various events hosted at the bar, notably the MCIT 2021 Winter Hackathon that took place a few weeks ago.
“Students hung out, people presented their projects, we had like a little awards ceremony,” says Professor Arvind.
Prizes included an award for most complex, most creative, most beneficial to MCIT and an overall presentation winner.
Place your bets…as soon as the other players arrive! The virtual poker table at the MCIT Bar
MCIT has also hosted alum mock interviews at the bar, where alumni come visit and put MCIT students on the hypothetical spot.
Even with interactive game table features, where students and faculty can enjoy a game of poker or pool, Bhusnurmath isn’t certain how much extracurricular use the bar gets outside of specific department events. He notes one time he popped in to curiously observe.
“There were a bunch of the kids who were at a couple of those tables,” says Professor Arvind. “It was just cool to see that they seemed to be working together. I think the best thing we can do during this pandemic is to say that ‘these are the things that are there,’ if someone wants to use them, they can.”
