TECH

3D vision technology powers factory automation

One night in 2010, Mohit Gupta decided to try something before leaving the lab. Then a Ph.D. student at Carnegie Mellon University, Gupta was in the final days of an internship at a manufacturing company in Boston. He'd spent months developing a system that used cameras and light sources to create 3D images of small objects. "I wanted to stress test it, just for fun," said Gupta, who would begin his postdoctoral research at Columbia Engineering a few months later.

He folded a piece of paper into a cone and placed it under the lights and cameras. When the cone was pointed toward the system, it captured the conical shape perfectly. But when light shone into the cone, the system failed. The problem, Gupta determined, was light bouncing off the cone's walls. Those "interreflections" added sources of light that the system wasn't built to understand.

He was far from the first researcher to encounter this problem. Five years earlier, Shree Nayar, now the T. C. Chang Professor of Computer Science at Columbia Engineering, had written a paper exploring a possible solution.

"That paper had planted a seed in my head," said Gupta, who is now a professor of computer sciences at the University of Wisconsin–Madison. "It made me think that there might be a way to solve the interreflection problem at scale."

After completing his Ph.D., Gupta came to Columbia Engineering and joined Nayar's research group, the Columbia Imaging and Vision Laboratory (CAVE). In just a few months, the engineers developed a breakthrough solution that has transformed the field of visual inspection. Today, technology based on their work is deployed in factories across the world.

"In a university research lab, we tend to approach problems from our academic viewpoint rather than the perspective of an industry insider," Nayar said. "It is therefore truly gratifying to see one of our results directly impact the manufacturing of a variety of products that are widely used in society."

Solving the reflection problem...Manufacturers produce more than a trillion electronic components every year. Many of these products, like chips and circuit boards, are small and complex, with extremely little tolerance for error. A mistake as small as a micron could render an entire board unusable. If problems aren't caught early in the manufacturing process, they can prove costly for businesses and their customers.

By the early 2000s, electronics had become so small and tightly packed that manufacturers began using computerized systems that used cameras and light projectors to measure the 3D structure of components. This new industry—called automated visual inspection—developed sophisticated imaging systems capable of inspecting the highly complex chips and boards found in consumer products such as phones, computers, and cars.

Now worth billions of dollars, the automated visual inspection industry has typically relied on a technique called structured light to create high-resolution 3D images of components. However, that technique has become less effective as electronics have become smaller. One reason is that tightly packed components tend to reflect light onto each other. Since automated systems are designed to measure light from projectors, spurious reflections can cause significant errors in 3D images.

In 2012, Nayar and Gupta developed a new imaging method that is resilient to these optical effects. They showed that by projecting light patterns in a narrow frequency band, structured light methods can be made robust to phenomena such as interreflections.

Their method, called Micro Phase Shifting (MPS), which was published at a premier computer vision venue (IEEE CVPR—Conference on Computer Vision and Pattern Recognition), demonstrated unprecedented accuracy in the 3D images it produces. In the realm of printed circuit boards, the method can reconstruct depth maps with micron-level accuracy, opening the door to precise, high-speed 3D imaging, which is valuable across many aspects of manufacturing.

Scaling the technology...In 2018, Omron, a leading company in the industrial automation and machine vision market, licensed the MPS technology from Columbia and developed an inspection system that launched in 2020. The product is now used by several leading automotive and consumer electronics manufacturers.

Dr. Masaki Suwa, the head of corporate research and development at Omron, and the president and CEO of OMRON SINIC X Corporation, said, "Our Automated Optical Inspection (AOI) solutions play a central role in ensuring the quality of printed circuit boards. Because Columbia's MPS technology is robust to spurious reflections when inspecting mirror-like surfaces such as solder joints, dies, and chip surfaces, it has proved essential to reliable 3D inspection. As electronic components continue to miniaturize, a technology like MPS that can capture 3D shapes with high precision will become increasingly important to printed circuit board manufacturing."

It is rare for a technology developed in a university laboratory to achieve large-scale adoption in a fast-moving and highly demanding field such as factory automation.

"The successful commercialization of Micro Phase Shifting underscores both the strength of Columbia's creative fundamental research and the value of close collaboration between academia and industry to bring breakthrough innovations into real-world manufacturing environments," said Ofra Weinberger, director of Columbia Technology Ventures at Columbia University.

"When we began this research project, we were motivated by a fundamental question: How do you recover accurate 3D information when light behaves in complex and non-ideal ways?" said Gupta. "We showed that by coding light smartly, one could separate the true 3D signal from the noise due to interreflections—a long-standing open problem in 3D imaging. Seeing that idea evolve into a method deployed at scale to help ensure the reliability of critical technologies has been a career highlight."

By creating an approach adopted by industry, the researchers demonstrated the value of academic research in bringing fresh ideas and rigorous thinking to business.

"Academic researchers explore a wide spectrum of problems, ranging from theoretical questions that seek to advance the knowledge base of the field to novel solutions to known practical problems," Nayar said. "It is exciting to see one of our innovations solving a critical problem in the manufacturing of products we use on a daily basis."

Provided by Columbia University