With the constant drive for better and cheaper cameras for industrial imaging applications, CMOS image sensors can make use of unique System on Chip (SoC) features. It is possible, due to 3D chip stacking and Back-Side Illuminated (BSI) CMOS image sensors, to integrate many image processing tasks into a single device. The future will see sophisticated machine learning and dedicated intelligence, computed on-chip, while simultaneously grabbing images for compact, higher speed, computer vision systems.
However, chip thermal management and power distribution are current obstacles that this new large-scale integration technology has to overcome.
In the meantime, advanced Front Side Illuminated (FSI) CMOS sensors are integrating analog and digital features to provide economic and performance oriented solutions. The key to achieving these objectives comes from judicious partitioning of what is useful for system performance aspects and/or cheaper to embed in the image sensor SoC. Here the imaging system application and the role of existing image processing devices such as CPU, FPGA or DSP has to be highly considered because redundancy offsets the economic concept. Creating a standard imager SoC product, that provides its target application market with a viable solution that is able to reach high quantity production (and therefore achieve lowest production cost), requires intense dialogue between the two communities.
Main market players have hardware, software, system architects and optical engineers, who are designing ever evolving end camera products that target specific application segments. Benefitting from this respective technology/application knowledge, they also have a multi-disciplined image sensor development team, to find innovative product solutions between the semiconductor technology and the final camera product.
This article will present an example of an application focused CMOS image sensor family for use in barcode scanning and other embedded vision applications, with examples of their use cases, and some future perspectives.
Barcodes – an overview of the most pervasive of coding systems, and associated reading technology
In the end of year holidays (the peak of consumer retail/on-line spending and associated logistics and transport, manufacturing and distribution etc.) well in excess of 5 billion barcodes are scanned per day. Considering that the first barcode was scanned in the 70s on a packet of chewing gum, it is clearly a formidable method of providing a machine-readable code, which is still growing in deployment into new fields of application. A 1D barcode symbol was the first to emerge (but now printed smaller and smaller as label printing technology advances) and still represents the majority used, especially for UPC (Unique pProduct Code) applications in retail, transportation and logistics, etc. The 2D barcode emerged in its various forms to offer a significant increase in code-able data versus the traditional 1D – typically 1D barcodes can have from 20-25 characters while 2D codes go beyond 2,000 characters depending on the specific barcode type. Apart from the obvious ability to store greater product info and details, check-sum and other error correction techniques can be coded into 2D barcodes to ensure increased tolerance to poorly printed or damaged codes. 2D barcodes are used heavily in some market segments such as for Direct Part Marking (DPM) onto individual components used for automated production in the manufacturing segment.
The 2D barcode reader technology transition began around 15 years ago but now represents the majority of today's market since it can also read existing 1D codes in addition to 2D symbols.
Figure 1: Barcode examples
Figure 2: Not all 2D barcode scanners are created equal.
Not all 2D barcode scanners are created equal…
The systems that identify and decode barcodes have advanced at a high pace and continue to improve to offer faster, smaller, cheaper and more robust readers.
While 1D laser-based scanners are still manufactured and deployed today, the most significant advances in the reading system came from the invention of 2D readers. A 2D reader offered significant evolution possibilities as a consequence of using an image sensor. It provides additional features not previously possible. This includes taking photographs or recording videos, and enables more advanced functions, like document scanning, OCR (Orthogonal Character Recognition), object detection, and dimensioning, as just a few popular application examples.
Teledyne e2v image sensors are unique in this market, offering benefits over other 2D sensor options. This comes largely from the fact that they are specifically designed for barcode reading applications as opposed to more general purpose, consumer or automotive applications. This translates to non-compromised, powerful solutions that provide all the ingredients toenable a market leading end barcode scanner reader.
Teledyne e2v has recently developed a range of new small pixel, low noise, global shutter CMOS image sensors with unique dedicated application features that offer significant cost reduction and/or performance improvements in Automatic Data Collection Systems (ADCS) and Auto Identification (AI) market applications. Sensor unit cost is clearly a top criteria in this market/application, but other tangible cost reducing factors must also be considered at system level such as illumination/optics reduction.
Figure 3: Teledyne e2v's Snappy - an innovative application specific approach to CMOS image sensing for industrial barcode readers.
2D reader systems require a very rapid frame capture to avoid blurring. This requires that the shortest possible integration time is used. Also, to get maximum Depth Of Field (DOF) or scanning range, often a very small optical aperture (typically F/8 or smaller) lens is used. The very small amount of photons that enter the pixel of the image sensor, combined with the very short integration time means that barcode reading can be considered a low-light application (see Figure 5). A global shutter is also advantageous when the barcode target is moving.
The principal sensor parameters that contribute to end-scanner performance are therefore unique to the application. Figure 4 lists some of the main sensor/barcode scanning performance criteria, and shows the Snappy sensor family, which we will use as an example of an application focused CMOS image sensor.
Figure 4: Teledyne e2v's Snappy sensor - key specifications that influence barcode scanning performance.
Figure 5: Teledyne e2v’s Snappy sensor - low-light SNR offers the benefit of reducing system illumination optical power and cost.
When the heat is on…
A closer look at the difference between the various components that make up the different noise parameters between 25°C junction temperature, and what happens to them at >65°C yields many limiting factors and should be considered in the choice of sensor. Spatial row and column fixed pattern noise is a particularly important parameter for barcode reading. Considering that FPN looks like straight horizontal/vertical lines, they can be mistaken for a barcode, or add miss-information to real barcodes in the image.
The semiconductor process used for the Snappy sensors shows just a few electrons of dark signal at 25°C, but even at above 65°C, only 77 electrons/second is observed. This assists the built-in FPN (Fixed Pattern Noise) cancellation algorithms of both row and column noise to achieve just a few percent of FPN even at high operating-temperature.
The total readout noise; which combines both temporal and spatial elements, is very low at typically 3 electrons and does not degrade beyond use at high-temperatures. Insufficient sensor performance at high temperature results in additional system cost from the only cure; since more light is needed.
Unique mono/color pixel filter patterns can combine the low-light sensitivity of monochrome pixels with the advantages of producing object color data.
Imagers can have color pixels which can add increased object/label recognition possibilities to provide additional security features to eliminate spoofing or in cases where the barcode itself can’t be read. Sensors used are predominantly monochrome, since color imagers are less sensitive due to the lower transmission properties of the organic color filter materials used and also represent a lower spatial resolution due to the need to combine red, green, and blue pixels to produce a ‘colored’ pixel. An interesting innovation can be found on Teledyne e2v’s Jade image sensor, which uses a combination of both monochrome pixels with a color pixel once in every four monochrome pixels. The much needed spatial resolution and sensitivity to read barcodes is preserved in this way, while offering a lower resolution color image to be captured simultaneously.
Figure 6. An innovative use of color sensing without compromising reading performance.
Innovative embedded application specific features
Achieving fast (snappy) barcode reading is not only a function of frame readout rate. While the maximum FPS could be a limiting factor, Teledyne e2v’s Snappy sensors do not compromise on this, by offering close to 120 frames per second in 8 bit operating mode. A unique power-up mode ensures that the device comes out of power up or standby with first acquired image (or FSE sub-image) at the datasheet SNR specification. This is not often the case for most automotive or other application oriented global shutter CMOS sensors, which require several complete frames to be wasted before they stabilize and reach their reported data sheet SNR performance values. This unique first frame read ability after power up is also a differentiating factor for cameras that result simply in highest speed barcode reading or 'snappyness' of scanning to the end-user, and higher productivity levels for their organization.
Two of the most innovative patented features that Teledyne e2v's Professional Imaging team has invented and now features in the latest Snappy family of sensors, are designed to offer the fastest barcode scanning, identification and decoding in scanning end-products are highlighted below.
1. Fast Self Exposure (FSE) mode, is featured on the Snappy 2MP and 5MP CMOS sensors:
FSE mode enables optimization of the exposure time in varying lights (see Figure 7). This brings huge benefits when compared with conventional auto exposure methods (in terms of convergence time and robustness), as the FSE mode is completely user programmable and provides a constant fast read benefit to end users, self-adapting to any kind of lighting or dynamic lighting environment, with almost no impact on the frame rate.
Figure 7: A novel approach to on-chip auto-exposure for barcode reading and all MV applications
The patented FSE mode uses a number of on-chip components to enable this feature:
(a) A unique column ADC architecture which can allow for 4 different integration periods to be set on consecutive columns, and then repeated across the array to produce 4 lower resolution images at different exposure values. It can also be used as a powerful HDR image capture feature
(b) Incremental sub-sampling in the row direction of up to 1/64 lines
(c) On chip statistics also count saturated pixels and provide in parallel a 16 bins histogram output, directly readable in the footer part of the image, for the currently active frame or region
(d) An RoI mode to be applied to the FSE sub-frame. Multiple regions are also possible, as well as regions within a region.
(e) A fine control to use either the histogram value, an average value or a combination
(f) Programmable registers to offer intuitive user-control and setup.
The end-application advantage again becomes scanning speed, since the FSE mode converges in typically much less than 10% of the frame period. Conventional embedded AEC controls in CMOS sensors use asymptotic techniques to avoid flashing and provide a slow convergence to the target image contrast level. However, the fact that numerous frames are expended in the process, then this is too slow for the barcode reading application.
2. Smart RoI mode, featured on the Snappy 5MP sensor:
Smart RoI uses an on-chip algorithm that detects a barcode or multiple barcodes in the current image. The barcode decoding image processing system requires that the area of the image containing a code is discerned from the rest of the image frame, in order to only analyze the area of interest. Whilst this operation is often performed on FPGA or CPU, the number of gates/RTC and processing horse-power devoted to this task is very high and adds significant cost and complexity constraints to the choice of processing engine.
Figure 8: Detecting multiple moving barcodes on the image sensor with the Smart RoI feature.
Embedding this barcode detection task in the sensor brings cost reductions by considerably reducing the processing overhead, but also provides some performance/stability advantages by performing this inside the sensor compared to much further down the digital signal flow chain. The detected 1D or 2D barcodes in the active frame are presented as blocks with X/Y coordinates reported as readable metadata in the footer (i.e. non-visible) region of the image data.
Many regions (or barcodes) can be detected simultaneously. Some other forms of repetitive signatures such as printed character strings can also be detected, for example in OCR applications. This feature also operates efficiently in applications where codes/objects/cameras are moving.
Significant cost savings and system simplification are the principal advantages. The use of 5MP sensors in barcode applications is limited mainly to high-end applications due to cost of larger optics and greater processing power needed to run real-time image processing/decoding on 5 million pixels, which offsets the principle advantage of offering greater scanning range or area. However, the small pixels of the Snappy sensor family and the reduction of processing overheads due to the on-chip Smart RoI feature, means lower cost points at system level are achieved. Power consumption and the ability to recognize many barcode symbols in a single image frame are becoming drivers for higher resolution sensors, particularly in the e-commerce logistics market.
Both the FSE and Smart RoI features can be run simultaneously in the Snappy 5MP sensor and ensures rapid, robust operation, even with constant changes in ambient light.The Snappy family of sensors are designed to address real application issues and provide useful cost/power saving features to make processing circuits less complex, less power-hungry and less expensive. We have also demonstrated that the performance can be enhanced when treating data at source within the sensor itself. Other machine vision applications such as inspection, metrology, OCR etc. can benefit from the performance and embedded features of the Snappy sensors, despite them being heavily focused on the barcode reading application/market. Some other areas of usage are in embedded vision Systems, IoT edge devices, drones, augmented reality and biometric systems.
Figure 9: Snappy sensors are compelling for many other MV, IIoT or other industrial computer vision applications.
Extending value by integrating optics and advanced MEMS based auto focus with Snappy sensors
Consumer camera optical modules are not options for the rugged long-term operation required in B2B industrial applications as the lens performance and optical characteristics required for decoding a barcode are often very specific. For example, where the working distance needs to be maximized, and optical aberrations such as optical MTF need to be minimized, in order to efficiently decode barcodes with smaller and smaller feature sizes below Nyquist limits. MIPI Optical Modules (MOM) that combine the powerful Snappy image sensors with high performance optics increase system value and save on development time and cost.
Figure 10: MIPI Optical Modules (MOM) with integrated fixed-focus optics feature the Snappy sensors
MIPI Optical Modules are ideal solutions for embedded vision applications, providing some customization of the optics available, to offer uncompromised performance and flexibility in a small 20mm x 20mm module. Teledyne e2v is starting to sample end customers with the first 2MP MIPI Optical Module and will announce both a 5MP version, and a 2MP Auto Focus version that uses a robust and lightweight MEMS based element to perform the auto focus. The singular benefit of auto focus comes from enabling bigger effective optical apertures to be used compared with a fixed focus optical system to achieve the same or improved scanning range or working distance, but requiring far less illumination power. Achieving better than two F stops of improvement, a future 2MP MEMS MIPI Optical Module will achieve additional performance and/or cost improvements in industrial imaging applications, barcode reading being one of the principal application segments that can make good use of this enhanced value. A product announcement is planned for mid-2020. In the meantime, an evaluation platform is available for the novel open-loop ‘multi-focus’ feature, that assures maximum working range with highest frame rates, and has the MEMS auto focus element fitted to a conventional Snappy 2MP demo kit.
New trends and changes caused by the e-commerce explosion
The huge double digit CAGR growth target (>25% per year) of the e-commerce boom is creating a shift in requirements in warehouse fulfilment centers, but also in the traditional 'bricks and mortar' retail point of sale market, as a response to retain and also grow their market share. Retail markets are undergoing major changes to improve the customer experience and minimize the delay through the checkouts by using un-manned automated 'self-scanning' systems. The success of these systems will depend on reliable barcode identification and decoding, but also on more sophisticated object recognition tasks that will see increased use of color imagers.
Figure 11: New trends and changes in CMOS sensor requirements and features caused by the e-commerce boom.
The requirements for higher speed or higher throughput of future scanners and cameras is a must-have if the high growth potential is to be fully realized. Higher resolution sensors with wider formats will allow for faster speeds in the future, and enable larger surface areas (containing many packages and barcodes in the image), to cover entire surface areas in warehouses of the future.
At sensor level, we will see the adoption of curved sensor technology with its advantage of reducing the complexity of external optics and alleviate some of the limitations with smaller pixels with regard to the diffraction limits of lenses today. There are two benefits; one that simplifies and reduces the cost of optics, and the other; which allows smaller pixels below 2.5µm to be employed without the MTF reduction impact of reaching optical diffraction limits.
The requirement to measure or dimension a package as well as scan its barcodes in modern warehouses is needed to meet the efficiency targets of emerging e-commerce fulfilment centers. Every square centimeter of valuable storage and transportation space must be used to its maximum. Monitoring the three-dimensional size of packages throughout the transport and logistics chain, in addition to reading associated codes/text labels using 2D scanning is currently performed by using two separate cameras; one for 3D (often using structured light or stereo vision based 3D techniques), in conjunction with a non-correlated 2D camera.
Efforts are now being made to invent a 2D plus 3D CMOS sensor that will provide a conventional 2D image and 3D point cloud simultaneously. Teledyne e2v is committed to being at the leading edge of innovation in these emerging market areas, and has future product roadmaps and IP aligned with the demands of next generation cameras and systems, that will continue to evolve into the foreseeable future, with focus on the specifics of their application.
Author: Gareth Powell, Senior Marketing Engineer, Teledyne e2v.