STMicroelectronics continues to enhance pixel performance and also focuses on advanced power management to keep ultra-low power consumption
Global-shutter CMOS image sensors were first launched roughly a decade ago, and since then have played a crucial role in enabling the accelerated throughput of high-speed manufacturing processes. The technology not only gave a digital output, but also avoided spatial distortion caused by the rolling-shutter effect.
STMicroelectronics is enabling the next generation of smart computer-vision applications with new high-speed image sensors that use “global shutter” – the preferred mode for capturing distortion-free images when the scene is moving or when near-infrared illumination is needed.
Global shutter sensors have a fast and high-resolution capture of the entire field of view. Therefore, they are suitable for applications in Machine Vision and Industry 4.0, measurements and inspections, intelligent traffic systems (ITS), automotive, medicine, research, 3D etc.
In a recent media briefing, Vincent LIN, Technical Marketing Manager, Imaging Division, Greater China and South Asia, STMicroelectronics shares more insights on global shutter image sensors, their market growth and its newest applications in automotive, AI, Metaverse.
What are global shutters and rolling shutters in machine vision cameras?
In machine vision systems the sensor is the heart of the camera. Responsible for much of the image capture responsibilities, image sensors come in many different forms. Broadly speaking, most image sensors are either Charged Coupled Device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensors. Each type of sensor uses either a rolling shutter or a global shutter.
The shutter controls the way the camera is exposed to a scene and how the charges built up in the pixel are read out. Rolling shutters and global shutters both control exposure, but complete pixel readout in different ways.
The purpose of rolling shutter is designed for the still image, to capture the still image, and also video recording, therefore you need camera with very high resolution, and also you need colour. Most of the camera you see around are rolling shutter, like your smartphone its main camera is rolling shutter, your selfie camera is rolling shutter, also in the building you have surveillance camera it’s rolling shutter, even the digital camera is all rolling shutter, because we need to capture colour image and also we need very high resolution, so all these occasions are using rolling shutter.
But there is one issue, if you’re using rolling shutter in the computer vision, the way rolling shutter capture image is like that, line by line. For example, this is one picture, the rolling shutter will take the picture line by line, so the picture capture will be finished in 1 frame time. If your target is a car, it’s a very fast-moving object, and the output of rolling shutter will become like this, the image will be deformed, or the picture will not be like the original things. If the image become deformed, it means it’s not the original one, it cannot be used for computer vision, because we don’t know if it’s a car or not.
That’s the reason that rolling shutter is normally not used in computer vision, instead, global shutter is totally different, for global shutter the way to detect picture is that the camera will take the picture at the same time, and the time to take picture is very fast and short, in this case even if you’re a fast moving object the image will be very accurate, and if the image is accurate then you can use it to do the computer vision.
Tell us about ST Imaging products and its key applications.
ST imaging products have a long history more than 20 years, we started in 1999 ST acquired a company called VISION, then we started to ship the camera module of mobile phone in 2002, until 2012 we started to move to other applications from mobile to medical, space and PC and other fields, and then in 2014 we started to ship ST ToF sensor, and then in 2018 we started to ship ALS sensor, and then in 2020 we started to ship new global sensor.
ST imaging products have 3 different product families, the first one is time of flight product which provides both direct ToF and indirect ToF, the second one is global shutter .
In the current generation we have 2 products for the consumer industrial application, the first product is VD55G0 which is 0.4Mp, and another one is called VD56G3 which is 1.5Mp, both products are using ST 3D technology, it means in the original design we have 2 layers of wafer, ST will combine the wafer into one, for customer it’s still a single wafer.
There are 3 key points for these products: The first one is that with our sensor we have highest quantum efficiency at 940nm, it means we have very high sensitivity in NIR domain, because for computer vision sensor normally it’s in NIR domain. And the other one is the resolution, you see the camera resolution is very close to the square, why do we use square? Because computer vision normally target is square, take an example the human face is close to a square face, The third one is the sensor can provide very short image capture to ensure very correct image and also we can enable very low system power consumption.
Our global shutter sensor finds application in smart home and smart building. Be careful with smart home, because you’re at home, normally you don’t want a camera just monitoring you all the day, but in the home you still want to enjoy the benefits from the camera or from the sensor, to make you more comfortable, so you still need a sensor, global shutter sensor is to bring you this benefit.
Next one is global shutter sensor for laptops and PC camera. ST provides a PC camera reference design, we input all the ST image product, as we mentioned earlier we have global shutter product, we have ToF, we have ALS, all image component are in reference design.
With global shutter in PC you can use it to do facial recognition, you can do gesture and gaze detection, and comfort level detection and privacy. And the Function of ToF and ALS sensor is doing user presence detection, power saving, display adaption and privacy.
Another reference by ST we call it ST AI camera, this platform included 3 major product group, the first one is sensing, the second is computing, and the third one is transmitting. Of course most of computer vision you need these 3 steps, first you need a lot of sensor for example global shutter sensor ToF sensor or IMU sensor to detect information, then you need to send the information to MCU and AI deep learning algorithm to get some output, then you need to use wireless device to send the data to cloud.
Optical flow on VD56G3
Please brief us about GS sensor applications in Automotive in-cabin cameras.
In automotive there are a lot of camera sensors, mainly 4 types, starting with in-cabin camera and there are some ADAS camera and viewing camera and aftermarket camera i.e. dash camera, these markets are all very mature, only in-cabin camera is a new field and a new market.
In-cabin camera is not only one type, there are two major types: Euro NCAP is a new car assessment program that defines two in-cabin monitoring system, the first one is driver monitoring also called DMS, the other one is cabin occupancy monitoring we also call it CMS or OMS.
With cabin or occupancy monitoring we’re more focused on passenger monitoring, especially the children, because somebody may leave the children in the car when they leave car.
DMS is the driver monitoring system. Normally the camera is installed in front of the driver, but since we only need to monitor the face, the lens doesn’t needs to be very large, 50 or 60 degree will be okay, resolution can be 1-2.3Mp. One important point is that with driver monitoring you must use NIR camera, the reason is the a driver is not only in day time, you also need to monitor in nighttime. Because you may still be driving at night, when you’re driving at night in very dark environment, you need to have illumination source on your face, in this case you cannot use visible light, you need to use invisible light, which means you need to use NIR light. That’s why you can see the image here is kind of a NIR image, here is the normal typical camera module, it looks like that, you have 1 sensor and also you have a NIR light source.
Occupancy monitoring is to monitor the passenger in the whole cabin, in this case it’s different, because we need to monitor the whole cabin, and because we also need to do the object detection or recognition so we need colour image, and as the space is wider of course our lens need to be wider and the resolution needs to be higher because we want to know all the details in this field.
Camera module
Another concept is all-in-one camera, it means we combine all the camera including the CMS, OMS and DMS by single camera, as we mentioned in the occupancy monitoring we need RGB Colour image for cabin and for driver monitoring we need NIR image, in this case if you want to use single camera to implement all-in-one system these camera need to be RGBNIR, single camera support both RGB Colour and NIR Mono.
How does ST’s Global Shutter image sensors work in vehicle safety applications and metaverse as compared to other industry players?
Regarding the vehicle safety, we already meet our major customer requirements. For example, with the product already in mass production like generation 1, we have already provided auto diagnostic feature to help the system know health conditions of the sensor. And for next generation safety, we will also implement ASIL-B level in automotive devices. Also, we will implement cyber security features, which means that the sensor will encrypt your image. When your image is encrypted, other people cannot see your face. This is what ST is implementing for the safety. So, regarding the Metaverse, ST provides customer and industrial grade sensors. So, we have two products as we already mentioned. We have0.4 mega product and 1.5 mega product. So, the 0.4 mega product is mainly used for the eye tracking, and gaze tracking. The 1.5 mega product is used outside for the wall tracking, collision avoidance or other applications. These two products are perfect fit for Metaverse applications.
What is your next focus for global shutter image sensors?
Regarding to next generation products, ST will continue to enhance pixel performance and also focus on better power management to keep ultra-low power consumption. Also, we will increase the number of embedded systems or processors in the sensor. So, we are also meant to reduce pixel size to ensure smaller camera dimension with higher resolution to make sure our customers can use the sensor, thanks to the high resolution, to provide accurate computer vision for applications. Those points are our focus for next generation products.
How do you see the market growth for Global shutter image sensors especially for Asia pacific and which sensors are showing the demand and adoption?
The global shelter sensor market in Asia is surely growing very fast across many traditional and emerging applications, especially in consumer, industrial and automotive applications. For example, the global shutter sensor is popular in face ID application on smartphones, smart locks or access control system. For example, in the robotics application, customer started to use global shutter sensor to generate 3D image for SLAM or simultaneous location and mapping. And AR/VR, youcan find GS sensor in eyetracking, wall tracking, and barcode scanning. And regarding automotive, DMS and OMS market is also growing in Asia Pacific.
How global shutter image sensors help in smart manufacturing and industrial robotics applications? With regard to smart manufacturing and robotics. For smart manufacturing, there are already some applications online, like high-speed inspection. I think there are some trends towards using the computer vision in robots, for example, robotic arms or the other robotic machines and production lines. When you have computer vision in these devices, the speed and accuracy will get improved. So, the second question is about the robotic applications. I think right now, in the market, you have solutions based on different technologies, like Lidar-based designs and camera-base dones. For ST, we work with customer on the image-based solutions that allow us to work with our customers using the two different kinds of global shutter sensors we offer. we can provide two monochrome global shutter sensors for our customers to build a stereo camera system for robots that provides very accurate obstacle detection, including very small objects. Of course, you can also help a map processor to build simultaneous location and mapping (SLAM). Another one is in the near future,ST may also provide consumer global shutter sensors with RGB filter. In this case, customers in the robots can use global shutter RGB sensors to implement the object detection. This is what we see for the robot and smart manufacturing.
What innovations are needed next for ST’s Global Shutter sensors to help boost adoption of automated driving?
For the global shutter sensor, of course, it will be linked to autonomous cars, especially one sat levels from 2, 3 to 4…when you are at levels 2-3 or 4, you may still have two drivers at the same time, sometimes drive by driver, sometimes you drive by car. To help carmakers to implement that, ST continues to provide global shutter sensors with more features. For example, we mentioned we will increase our camera sensor resolution. We will increase the camera sensor sensitivity. And we will reduce the camera size. So, this can help our customers to design better performance cameras, and also provide the system a better image. When the sensor can provide a better image, the system can easily use it to detect driver behaviour and implement all those features.
