Computer Science Research Profile:
Image and Video Processing and Analysis

From the Fall 2012 issue of of Threads

From the moment of his introduction to it, Guillermo Sapiro was hooked on image processing.

"I went to all the lectures, and this was the one that excited me the most," the professor of Computer Science and Electrical and Computer Engineering recalled of a lecture series he attended while an undergraduate at Technion: Israel Institute of Technology. The next semester, he was learning image processing. "That's the power of good teachers," he said, laughing.

Two decades later, Sapiro is still excited by the field he leapt into. "It's just a fun area to work, and there is contribution," said the professor, who joined Duke this fall with a secondary appointment in Computer Science. He worked for 15 years at the University of Minnesota and prior to that at Hewlett Packard Labs. In his main research area in image and video processing and analysis, Sapiro works in a number of venues and with colleagues from multiple disciplines, focusing primarily now on consumer, brain and biomedical imaging as well as behavioral analysis.

Brain imaging for neurosurgery

He is particularly excited by a project in which electrodes are implanted deep in the brains of Parkinson's disease patients. Constant electrical impulses help suppress their tremors and other symptoms. "It's actually extremely impressive," Sapiro said, showing a video of a patient speaking about his implant. In the video, the patient turns off a battery that controls the electrode's constant impulses and begins shaking. He restarts the stimulation, and the shaking stops. "It's just absolutely amazing," Sapiro said. "It's the kind of movie that Congress should show to others in the support of science." By developing image processing to analyze the magnetic resonance imaging of a patient's brain, Sapiro is helping neurosurgeons and neurologists with individual target localization. That pre-surgery localization gives direction in implanting the electrode in a particular patient. Post-operation, it helps the neurologist determine which end of the electrode to activate.

Image processing for HIV research

In other medical imaging, Sapiro is helping the National Institutes of Health with disease detection. His research group in Minnesota worked for four years to develop computational techniques to find HIV in low-quality images; determine the virus's shape; and automatically create 3-dimensional reconstructions of its outer layer, or envelope. Knowledge of the virus's shape could lead to a vaccine that wraps the envelope and prevents it from attaching to other cells. Sapiro since has been extending the computational work to other viruses, including influenza, while continuing to improve the ability to process information at smaller scales.

He also is continuing his work to enhance consumer images - such as making a blurry photo sharper - while also trying to find cheaper methods for producing quality images. An algorithm he created to compress images while at Hewlett Packard Labs is being used in the current Mars rover expedition as well as a previous one. Other techniques he has developed are used in movie production, such as to remove mistakes or scratches from a film without any perceivable trace of a change. The ability to transfer the art of image inpainting - long used in art restoration - to video is the result of an algorithm he and his team developed. It also allows black-and-white movies to be transferred to color, which has gained the interest of a French company that wants to make history films attractive to teens. In addition, Sapiro has developed techniques for special effects in film, allowing parts of a scene to be segmented out to create shadow effects by replicating the segmented images or to keep focus on a particular action by blurring the rest of the scene. The techniques, which also allow a scene's background to be changed, were co-developed with Adobe and featured in the 2010 upgrade of its After Effects software.

Sapiro's work in video also includes analysis and the ability to perform automatic activity recognition. Because of its security applications, such as airport surveillance or crowd monitoring, a lot of his research is supported by the Department of Defense. Through methods he has developed, a video frame that shows activity out of the normal range might be highlighted in a different color, allowing abnormal activity to be detected easily without excessive manpower.

This ability to track activity in film also is used in his work in behavioral analysis. He has been involved in a project with psychiatrists in Minnesota to detect autism early. At Duke, he is starting a project with Helen Egger, head of the Division of Child and Adolescent Psychiatry, to detect anxiety in preschool-aged children. For the Minnesota project, he developed a technique using dots to automatically track the eyes, nose and ears of children involved in standard tests for autism that look for eye contact and response to a person. "Instead of just having the expert have to decide, 'Is he looking at me or not?' - or maybe a nonexpert - we basically can do that automatically," Sapiro said. The goal is to develop the techniques further to allow for earlier detection and use by nonexperts. Children typically are 3 before being diagnosed with autism. "I'm not saying it's too late, but it's 2½ years later than the experts want it," Sapiro said. "That creates a lot of suffering in the family, of course. But there are a lot of studies that indicate that if you start intervening early, you might actually help in the progress. If you catch it early, you can change the course."

Sapiro plans to use some of his tracking tools with Egger, who has a good deal of data and videos from anxiety testing of kids. Part of the goal is to mine data. "If we manage to develop these tools and spread them, we might be able to observe thousands and to basically learn from that a lot," Sapiro said. "It's an exciting project because the contribution to society could be huge. And that's one of the things that are always exciting with medical imaging projects."

"If we manage in these projects that we have," he noted about all of his work, "the results for humanity are immediate. We all contribute a small piece, but the applications are very clear."

Return to Reseach Profiles home page