In the past year, the Department of Computer and Information Science has welcomed an unprecedented number of academic professionals to join Penn’s faculty. One of the Assistant Professor’s who has joined both CIS and ESE this past Fall is Mingmin Zhao, an MIT graduate with a PhD focusing on building wireless sensing systems with artificial intelligence.
The collaboration between CIS and a number of departments at Penn is what encouraged Zhao to further his research and teaching career here.
“Penn provides a fertile ground for interdisciplinary research not only within the CIS department but also with other departments, including ESE, medical school, nursing school, etc.” said Zhao, “I am very excited about collaborating with people at Penn and working on highly-impactful interdisciplinary research.”
Zhao’s research interests include building wireless sensing systems that can capture a human’s functionality through physical surfaces. He explains that his research “uses machine learning to interpret and analyze wireless reflections to detect humans through walls, track their movements, and recognize their actions, enabling a form of x-ray vision.”
With these wireless sensing systems, he has also developed a way for healthcare professionals to track a person’s functions including sleep, respiration, and heart rate. “These technologies allow us to continuously and without contact monitor people’s health without wearable sensors or physical contact with the user.” In the startup he joined after graduating, Zhao stated that they are building upon his own research to “work with pharmaceutical companies to run clinical trials in people’s homes.”
When asked about what made him passionate about the work that he does, Zhao explained that he is passionate about developing sensing tech that focuses on better understanding humans and their wellbeing.
“New sensing technologies (e.g., contactless monitoring of physiological signals) could help doctors understand various diseases and how patients are doing after taking medications,” said Zhao. “They could enable new digital health and precision medicine solutions that improve people’s life.”
Mingmin Zhao is currently teaching CIS 7000 focusing on wireless mobile sensing and building AIoT (Artificial-intelligence Internet of Things) systems. He is looking forward to educating his students to apply what they have learned in building “hardware-software systems” to solving practical problems that can impact the world.
So far in this semester, our department blog has talked about the growths in our labs and our faculty members careers. We have touched on the incredible seminars that have been taking place as well as the exciting research projects that our students are involved in. The blog’s overall purpose is to showcase the technological strides that Penn Engineering is making as well as the significant academic achievements of our faculty and student body. The C.I.S. blog is also a platform that strives to implement humanity and relatability to those who are a part of the Penn community and those outside of the University.
As many of you may know, DALL-E 2 developed by OpenAI was launched in April of this year and has just recently been made available to anyone. You get an allowance of free credits that enable you to type in a detailed description of anything that comes to mind and the machine learning models generate digital images that reflect the prompt. We were interested in the combination between artificial intelligence and the ability to generate realistic images using human ingenuity.
DALL-E is a great example of AI and human collaboration working to break barriers and expand horizons through artistic creativity. This platform also gives people the ability to play and be as out there and imaginative as they want. All in all, DALL-E gives us the opportunity to have fun and explore our hobbies, interests, and studies in the form of art. To showcase this AI system and what it can do I had asked all students from the C.I.S. department, from Undergrad to Ph.D., to send in the descriptions that they prompted and to have fun with it.
With that being said, this Gallery of DALL-E generated art was made possible by some of our wonderful students in the C.I.S. Department!
Ani Petrosyan, she/her
Computer Science major, 2026
Purple mountains with Armenian waterfalls
"I am an international student from Armenia. It's a very mountainous country, with waterfalls and wonderful nature. One fun fact about me: most of the dreams I see have purple color in them, so I am dreaming of my country and seeing it in purple.
Edward Hu, he/him
Ph.D. in Robot Learning
A. Bob Ross in the style of Picasso uncanny unreal engine
"Bob Ross is an iconic painter, so I would like AI painters to pay homage to him."
B. Darth Vader cooking in Hell's Kitchen
"Hell's Kitchen is one of my favorite shows. I think Darth Vader's past with high heat and pressure scenarios would make him an excellent contestant."
Yuxin Meng, she/her
A. Hacker, another dimension, digital art
"This one was intended to be "software engineer..." or "coding in..." but these keywords seemed to be less instructive compared to "hacker". I wanted to see us working on the same thing in another universe."
B. Bionic sheep, blueprints
"Love the book: Do Androids Dream of Electric Sheep? Electric sheep in the book look like real sheep, so I guess bionic is proper."
C. In lab, machine reading brain, codes on computer science, digital, by Beeple
"Not so much what I pictured in my mind. I've been obsessed with Pantheon (science fiction drama) recently. Briefly, I expected a picture of machine scanning human brain as code."
Rotem Dror, she/her
Postdoc in the Cognitive Computation Group
Two computers compete in a running competition
"I needed an image for my job talk presentation that would show two models competing who are going to be state of the art. My research involves developing statistical methods for comparing NLP models to determine which is better."
Hannah Gonzalez, she/her
MSE and BSE in Computer Science, 2023
A. A red fox surfing The Great Wave off Kanagawa by Katsushika Hokusai
B. Huskies sledding in Alaska by Monet
C. Macro 35mm film photography of a floating otter wearing a space suit with the Van Gogh "Starry Night" painted background
D. An Andy Warhol style painting of a corgi winking
Gaoxiang Luo, he/him
Ph.D. in Machine Learning
"I generated an image using my hobbies as keywords: cat, guitar, and latte art. I was very impressed and surprised that the AI considered the cat element as latte art!"
This year’s Halloween celebration for Penn Engineering was a great success! Held in Quain Courtyard this past Monday, October 31, students, staff, and faculty gathered for trick or treating, hot apple cider and hot chocolate, and a costume contest. There was a great turn out of students coming in between classes and during their lunch break. In addition, incredibly decorated tables from each department covered the courtyard and there was wonderful staff participation. Even some faculty members came out in costume! It was a great experience interacting with students for some spooky and relaxing fun!
Scroll down to see the department tables and awesome costumes that came out for another fun year of Halloween festivities!
Many students and faculty alike may recognize the face above as Osbert Bastani. Well that’s because this Assistant Professor is not a new member of the Penn Engineering team. Osbert joined the Computer and Information Science Department as a Research Assistant Professor in 2018 specializing in programming languages and machine learning.
“Penn has a great group of faculty working on interesting research problems, and they are all incredibly supportive of junior faculty. I’ve been fortunate enough to collaborate with Penn CIS faculty in a range of disciplines, from programming languages to NLP to theory, and I hope to have the chance to collaborate with many more.” (Osbert Bastani)
Osbert actually began his research career in programming languages. This major challenge in this research is “verifying correctness properties for software systems deployed in safety-critical settings.” He explains that because machine learning is progressively being incorporated into these systems, it has become a greater challenge facing verification. In his research, he is tackling this overarching question; “How can one possibly hope to verify that a neural network guiding a self-driving car correctly detects all obstacles?” While there has been progress made in trustworthy machine learning, there is still a long road ahead to finding solid solutions.
His enthusiasm in working with the Ph.D. students on various topics and research projects is what he has looked forward to most as he entered into this new role in his teaching career at the start of this Fall semester. Since the school year began, he has been teaching Applied Machine Learning (CIS 4190/5190) with Department Chair, Zachary Ives. When asked about how the semester is going Osbert replied:
“I’ve been very fortunate to have strong students with very diverse interests, meaning I’ve had the opportunity to learn a great deal from them on a variety of topics ranging from convex duality for reinforcement learning to graph terms in linear logic. An incoming PhD student and I are now learning about diffusion models in deep learning, which are really exciting!” (Osbert Bastani)
While teaching, Osbert is also involved in several research projects that are dealing with trustworthy machine learning within real-world settings. One project that raises several questions about fairness and interpretability includes “building a machine learning pipeline to help allocate limited inventories of essential medicines to health facilities in Sierra Leone.” In addition, during a summer internship at Meta, one of Osbert’s students has been in the process of “developing deep reinforcement learning algorithms that can learn from very little data by pretraining on a huge corpus of human videos.”
Osbert Bastani wears many hats in the CIS Department. Not only is he involved in teaching and research projects with students, he is also a member of several groups within the department. Those include PRECISE, PRiML, PLClub, and the ASSET Center and he encourages all students to attend the seminars that each club holds and get the opportunity to learn about research in their areas or outside of their own.
Just as Osbert works to problem solve within the classroom and in his research, he does just about same outside of work as well! He expresses that he is an avid board game player and frequents the restaurant just down the street from Penn called “The Board and Brew”. He and his wife have played through the restaurant’s entire collection of the game “Unlock!”. The Board and Brew has great food and several hundred games to choose from. It is highly recommended by Osbert himself!
As the School of Engineering and Applied Science grows, so do the rooms that house all of the creative minds within our departments. Just in time for the Fall 2022 semester, two labs that are a part of the C.I.S. Department and are located in the Moore Building have gotten a makeover this past summer and the faculty and students are thrilled. The Distributed Systems Laboratory (DSL) in Moore 100 and The SIG Center for Computer Graphics in Moore 103 have both undergone renovations for more student space to collaborate, create and liven up the space!
Cheryl Hickey, the Administrative/Event Coordinator in C.I.S., collaborated with the directors of the labs and a design team. She was tasked with choosing the furniture, color scheme, and layout that would best suit the space and the students. The Distributed Systems Lab received all new desks as well as a redone kitchen area for faculty and students. Jonathan M. Smith, Olga and Alberico Pompa Professor of C.I.S. said, “The DSL renovation allow for maximum flexibility in placement of student desks and experiments. Moving the network drops into the DSL machine room freed up space in the main areas for scholarly interaction and increased physical security.” The lab now encourages students to work together within the space better than ever before.
“The DSL renovation is nicely done and timely. I know many of our students really appreciate the new space. The new desks are superb. These days, when I walk past DSL, I’m pleased to see many students interacting with each other. The vibrancy and bustle are back in DSL! Our DSL seminars have also been packed with faculty and students listening to each other’s research. Nothing beats in-person interactions, and the best ideas happen when students talk and iterate through ideas in close proximity.” (Boon Thau Loo, Director of DSL)
Students have noticed the changes to the DSL lab as well since they began the semester just over a month ago. Jess Woods, a C.I.S. PhD being advised by Sebastian Angel, stated that, “It’s nice having our own functional cabinets/lockers. There are more desks and less clutter, which seems to encourage more people to come to campus to work, which makes the environment more collaborative and productive”. Having a space post-Covid that is inviting and open for all students to take advantage of is incredibly positive for the department.
While a part of the SIG Center for Computer Graphics is fully renovated as of this past summer, the lab is still continuing to grow. New technologies are in the works in addition to a couple of new faculty members who will be officially joining the department this coming Spring 2023 semester. Incoming assistant professor in Computer Graphics, Lingjie Liu, stated that there will be many upgrades to the space and equipment including GPU clusters and data storage space in support of research development.
Upon her arrival to Penn Engineering, Lingjie will “we will also purchase and set up a multi-view volumetric camera capture stage for capturing human motions and human-object interactions with multiple synchronized cameras (about 40 cameras).” This set of equipment will be an addition to the labs’ Vicon 16-camera motion capture system.
“The research focus of the new SIG Lab will be developing Artificial Intelligence (AI) technologies for perceiving, understanding, and interacting with 3D objects, people, and environments. Specifically, we will pursue three research goals: (1) High-quality reconstruction of real-world scenes from sparse RGB (camera) images; (2) Photo-realistic image synthesis of real-world scenes with 3D control; and (3) Large-scale 3D scene generation for 3D machine learning tasks.
Lingjie states that the SIG Lab’s new research focus will be “developing Artificial Intelligence (AI) technologies for perceiving, understanding, and interacting with 3D objects, people, and environments.” There are three research goals that the lab will pursue which include:
High-quality reconstruction of real-world scenes from sparse RGB (camera) images
Photo-realistic image synthesis of real-world scenes with 3D control
Large-scale 3D scene generation for 3D machine learning tasks.
“We will approach these goals by designing new algorithms that incorporate AI advances into classical computer graphics methods. We believe that with the new research focus and the facilities upgrade, we will create new research achievements and continue the success of our SIG Lab.”
The finished and fully furnished side of the SIG lab will now be filled with another incoming assistant professor of C.I.S. and ESE, Mingmin Zhao‘s students. “Working with the renovation and designer team was a pleasure, and the new lab is absolutely gorgeous. I am excited to attract and recruit more students and spend time with them in the new lab.”
While touring SIG lab earlier this month, there were several students hard at work and raved about how much they enjoy spending time in the space. One PhD student who is being advised by Mingmin, Gaoxiang Luo, said, “The renovation of Moore 103B is way beyond my expectation. The workspace shelves allow us to store electronic components, while the wall pegboard allows us to hang tools. While the space is large enough for us to conduct wireless experiments using radar with a certain distance, the ESD flooring further ensures our safety. It’s also worth mentioning that I love the ergonomic height-adjustable desk and chair, which is particularly useful for our health as we work with our computer frequently nowadays.”
It is incredible to see the impact that this renovation has had on the students and faculty. The department is looking forward to seeing the work and successes that come out of these two labs for this new generation of students in Penn Engineering!
Picture this: you’re getting ready to watch a movie on Netflix, popcorn in hand, and several films pop up that have been curated just for you. What are you going to do: choose one from the list recommended by the underlying AI algorithm, or worry about how this list was generated and whether you should trust it? Now, think about when you are at the doctors’ office and the physician decides to consult an online system to figure out what dosage of medicine you as the patient should take. Would you feel comfortable having a course of treatment chosen for you by artificial intelligence? What will the future of medicine look like where the doctor is not involved at all in making the decision?
This is where the ASSET Center comes into play. This initiative, led by the C.I.S. Department in Penn Engineering, to focuses on the trustworthiness, explainability, and safety of AI-systems. The faculty members and students who are a part of this Center have tasked themselves with finding ways to achieve this trust between an AI-system and a user. Through new collaborations between departments all throughout Penn, innovative research projects, and student engagement, ASSET will be able to unlock AI abilities that have never been achieved before.
I recently spoke with Rajeev Alur, Zisman Family Professor in the C.I.S. Department and inaugural director of ASSET. He elaborated on our Netflix example to explain the trust between an AI-system and a user and when it is absolutely critical for the adoption of AI by society. Based on movies and shows that the user watches, Netflix is able to give several recommendations, and it is the user’s choice as to whether they will go for something new. While the recommendations may be decent picks to the user “there is no guarantee or assurance that what they are recommending is foolproof or safe”, says Rajeev. Although AI is found to be useful in the case of choosing what to watch, the user needs a higher level of assurance with the system in more critical applications. An example of this could be when a patient is receiving treatment from a doctor. This high assurance can become important in two cases. One is when the system is completely autonomous, or what is called a “closed loop system,” and the other case is when the system is making a recommendation to a physician who decides what course of action to take. For this latter case, the AI does not make the decision directly, but its recommendation may still be highly influential. In many clinical settings, there are AI-systems already in place that dole out courses of treatment that best suits the patient, and a physician consults and tweaks these choices. What ASSET is looking to implement in the medical field are autonomous AI-systems that are trustworthy and safe in their decision making for the users.
“The ultimate goal is to create trust between AI and its users. One way to do this is to have an explanation and the other one is to have higher guarantees that this decision the AI-system is making is going to be correct,” Rajeev explains.
For ASSET to succeed, the center must nurture connections throughout Penn Engineering and beyond its walls. Within C.I.S., machine learning and AI experts are working together with faculty members in formal methods and programming languages to come up with tools and ideas for AI safety. Outside of C.I.S., Rajeev explains that robotics faculty in ESE and MEAM are interested in designing control systems and software that uses AI techniques in the Center. Going beyond Penn Engineering, ASSET is dedicated to making connections with Penn’s Perelman School of Medicine. “There is a great opportunity because Penn Medicine is right here and there are lots of exciting projects going on. They all want to use AI for a variety of applications and we have started a dialogue with them…This will all be a catalyst to having new research collaborations”, says Rajeev.
In keeping with the idea of autonomous AI that was discussed earlier, one of ASSET’s flagship projects is called Verisig. The goal of this project involving the collaborative efforts of Rajeev Alur, Insup Lee, and George Pappas, is to “give verified guarantees regarding correct behaviors of AI-based systems” (Rajeev Alur). In a case study being performed by the Center, researchers have verified the controller of an autonomous F1/10 racing car to check the design and safety of the controller so that the car is guaranteed to avoid collisions. The purpose of this project is to further understand assured autonomy; if a controller of a small car can be found trustworthy and safe, these methods may eventually be generalized and used in AI applications within the medical field.
How to get involved
The best way for students to get involved with ASSET is engaging in the center’s Seminar Series. They happen every Wednesday in Levine 307 from noon to 1pm, and the great thing about them is that any Penn student can join. There are incredible speakers lined up through the Fall and Spring semesters this school year, so instead of turning on Netflix and letting the system choose your next bingeworthy show, join ASSET every Wednesday for exciting talks about creating safe and trustworthy AI!
What happens when AI goes wrong? Probably not the Terminator or the Matrix – despite what Hollywood suggests – but rather, something that could still harm a human, such as a self-driving car that gets into an accident, or an algorithm that discriminates against certain people. Fortunately Penn has innovative researchers like Eric Wong, who build tools to make sure AI works correctly!
You may have already seen Eric on campus or perhaps teaching his advanced graduate class. Just like the Class of 2026 who are quickly learning their way around Levine Hall, Eric is one of the C.I.S. Department’s newest faculty members. An Assistant Professor who works in Machine Learning, Eric is a Carnegie Mellon Ph.D. graduate and a former MIT post-doctoral researcher in the Computer Science and Artificial Intelligence Lab.
As this semester is in full swing, Eric Wong is busy at work teaching course 7000-05: Debugging Data and Models. When asked what he is looking forward to most about teaching in Penn Engineering, Eric stated,
“One of the key skills that students will learn is how to tinker with AI systems in order to debug and identify their failure modes. I’m excited to see the new ways in which Penn Engineering students will break AI systems, as well as the innovations they come up with to repair them!”
The initiatives that Penn Engineering has launched in recent times are what drew Eric to the C.I.S. Department, specifically the ASSET Center. “Penn Engineering is well-situated to ensure that the tools and systems we develop as computer scientists actually satisfy the needs and requirements of those that want to use them.”, said Eric. He will be one of many faculty members working with ASSET to develop reliable and trustworthy AI-systems which coincides with his own research.
Some of Eric’s specialized interests in this field include “verifying safety properties of an AI-system, designing interpretable systems, and debugging the entire AI pipeline (i.e. the data, models, and algorithms).” His research goals are working towards debugging AI-systems so that the user is able to understand the decision process of a system and learn how to inspect its defects. Eric is also interested by the interdisciplinary work of connecting these methods to other fields outside of engineering. Collaborators in medicine, security, autonomous driving, and energy would “ensure that the fundamental methods we develop are guided by real-world issues with AI reliability.”
As AI is being developed and deployed at a rapid rate, Eric worries that, “it is only a matter of time before the ‘perfect storm’ induces a catastrophic accident for a deployed AI system.” In teaching methods of debugging AI-systems, he strives to give his students the tools and knowledge toward building safer and more trustworthy AI for the future. He hopes that with his research and teachings in the classroom, students take the time to “critically examine their own system” before sending them out into the world.
When Eric is not spending time making sure AI-systems are at the top tier of trustworthiness and reliability, he enjoys trying to recreate the recipes of meals that he orders at restaurants. Trying to “reverse engineer its creation process” is harder than it might seem. Eric mentioned that, “It does not always look the same as the original, nor does it always taste as good, but sometimes it works!”. Maybe someday that too will be something an AI can do (correctly)!
The 2022-2023 academic year has kicked off last week and summer has officially come to a close. We would like to welcome back returning Computer and Information Science students as well as the Class of 2026! Penn Engineering is excited to have you on board.
With a new school year comes new changes for Penn Engineering and the CIS department. In the past year, we have hired an exceptional number of faculty, brought in new research initiatives, renovated our spaces, and broke ground on a state-of-the-art facility. We are also incredibly thankful to be able to see so many faces in person this year as the circumstances surrounding Covid-19 continues to improve and in-person activities can commence.
Several brand new assistant professors have joined the department this semester, while the rest will arrive in January and next Fall. Rejoining the CIS department, in a new role as an Assistant Professor, is Osbert Bastani who develops innovative techniques for programming and building software that incorporate machine learning components. Two new faces to Penn Engineering features Danaë Metaxa, who works in areas of human-computer interaction and communications, and Eric Wong, who works on robust and reliable machine learning.
“We are delighted to welcome an unprecedented 10 new assistant professors arriving over the next year. Each brings new innovations to the curriculum and more opportunities to get involved in undergraduate research projects.”, says Zachary Ives, Chair of the CIS Department.
Each new faculty member entering into Computer and Information Science has arrived to break barriers and help our students grow.
As our department is growing, our spaces have been transforming as well. This summer, the Distributed Systems Lab (DSL) and the SIG Lab for Computer Graphics on the first floor of Moore have both undergone major renovations. On the second floor of Levine Hall, the brand new Penn Human-Computer Interaction Lab, led by Andrew Head and Danaë Metaxa, just opened.
“We couldn’t be more excited for the start of this year and the official launch of our group. We’re looking forward to teaching Penn’s first Human-Computer Interaction courses at all levels, welcoming our first cohort of PhD students and opening our physical space- the HCI Lab- in Levine 255. We welcome interested students to reach out to us!” -Danaë Metaxa, Assistant Professor, CIS Dept, University of Pennsylvania.
In addition to our growth in space, Penn Engineering is not stopping there. The construction of Amy Gutmann Hall has begun during the summer. The creation of new space, study rooms, and research labs is anticipated to be opening in September 2024.
Since the launch of this past year’s interdisciplinary research initiative, Innovation in Data Engineering and Science (IDEAS), the CIS Department announced the ASSET Center directed by Rajeev Alur. ASSET (AI-enabled Systems: Safe, Explainable, Trustworthy) focuses on implementing tools and science to guarantee AI systems do exactly what they are designed to do. Getting students involved in this new initiative is a top priority for the Center.
“The best way to get involved is to join our seminars. It’s every Wednesday at noon and we have a great line-up of speakers. There is a number of faculty from our department, other Penn faculty, and also outside speakers. Some of the topics will include applications to healthcare, explainablility, and safety.” -Rajeev Alur, Director of ASSET, University of Pennsylvania.
With every new faculty member, space renovation, and research initiative; all of these things are implemented to give students the best opportunities for success. We have another exciting year just beginning at Penn Engineering. Let’s make it count.