THine Value Developed an automatic generation tool for ISP firmware that is no longer dependent on FPGA
2018.08.31
- Article
- Column
An “image sensor” refers to a semiconductor device that takes images and videos. It is mounted on various electronic devices including digital cameras, smartphones, and automobiles. However, we cannot expect sufficient functions only with an image sensor. Combined use with an “Image Signal Processor (ISP)” that handles image processing is essential. The problem here is that the development of ISP firmware is extremely difficult. To solve this problem, THine Electronics began offering the “Camera Development Kit (CDK).” This article explains the background of CDK development and efficacy of its adoption.
ISP is essential for image sensors
The scope of application for image sensors is expanding. The mounting of image sensors on not only digital cameras and smartphones, but also on automobiles and surveillance cameras, industrial machine visions, document cameras (visualizers), web conference systems, medical electronic devices, and XR compatible equipment including AR (augmented reality), VR (virtual reality), and MR (mixed reality) is also progressing at a rapid pace.
When image sensors are mounted on these electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured with image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using the ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
The mounting of image sensors on electronic devices thus requires the selection and adoption of an appropriate ISP. However, forms of ISP differ depending on the electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in the chipset for mobile terminals are used. The adoption of a standalone IC is not common. For other types of electronic devices, however, the adoption of a standalone ISP chip or FPGA that implements an ISP IP core is common.
When image sensors are mounted on these electronic devices, an “Image Signal Processor (ISP)” is essential as raw image data captured with image sensors present an “incomplete picture.” Such processing as removing noise, adjusting brightness, and focusing on objects by using the ISP is required. Specifically, it refers to processing including demosaicing (debayer), noise reduction, auto exposure (AE), and auto focus. A “complete picture” is created by applying such processing (Fig. 1).
The mounting of image sensors on electronic devices thus requires the selection and adoption of an appropriate ISP. However, forms of ISP differ depending on the electronic devices to be mounted. In many cases for smartphones, ISP functions integrated in the chipset for mobile terminals are used. The adoption of a standalone IC is not common. For other types of electronic devices, however, the adoption of a standalone ISP chip or FPGA that implements an ISP IP core is common.
Skillful use is difficult
Regardless of the form of an adopted ISP, its skillful use is not easy as firmware must be developed to properly configure various parameters to create “complete pictures.” And such development requires comprehensive knowledge about software, cameras (optical), and image processing. It is never an easy job.
Major smartphone manufacturers can have good support by chipset vendors, because they make bulk purchases of chipsets for mobile terminals with an integrated ISP. Therefore, the development of firmware cannot be a high hurdle.
The development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as they cannot expect good support due to their small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission the development of firmware to semiconductor manufacturers that offer chipsets and ISP chips. Of course, they have to cover the development cost, which is not small. “If the purchase quantity for the application exceeds one million pcs. per year, it is possible to absorb the development cost; however, if the quantity is only about 100,000 pcs. per year, the development cost will impose a heavy burden” (THine Electronics).
The adoption of FPGA does not improve such a situation because it also requires the manufacturers to develop firmware. Furthermore, they can hardly expect support by ISP core vendors. For FPGA, a measure exists that customizes circuits and incorporates them as hardware according to the content of required processing. In this case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers than software engineers. As a result, such an approach may not necessarily offer a solution.
Major smartphone manufacturers can have good support by chipset vendors, because they make bulk purchases of chipsets for mobile terminals with an integrated ISP. Therefore, the development of firmware cannot be a high hurdle.
The development of firmware will be a very high hurdle, however, for medium-size smartphone manufacturers, industrial machine vision equipment manufacturers, and surveillance camera manufacturers, as they cannot expect good support due to their small amount of purchases of chipsets and ISP chips.
Needless to say, it is possible for such manufacturers to commission the development of firmware to semiconductor manufacturers that offer chipsets and ISP chips. Of course, they have to cover the development cost, which is not small. “If the purchase quantity for the application exceeds one million pcs. per year, it is possible to absorb the development cost; however, if the quantity is only about 100,000 pcs. per year, the development cost will impose a heavy burden” (THine Electronics).
The adoption of FPGA does not improve such a situation because it also requires the manufacturers to develop firmware. Furthermore, they can hardly expect support by ISP core vendors. For FPGA, a measure exists that customizes circuits and incorporates them as hardware according to the content of required processing. In this case, however, advanced technologies are required for hardware. Generally speaking, electronic equipment manufacturers typically have fewer hardware engineers than software engineers. As a result, such an approach may not necessarily offer a solution.
Automatic firmware creation
To solve these problems, THine Electronics, an ISP chip manufacturer, began to offer the Camera Development Kit (CDK) for their ISP chip “THP7312.”
The CDK consists of three major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “THine Tuning Tool,” and (3) firmware templates. The hardware consists of an ISP board with THP7312, a frame grabber, and a header board equipped with a camera module that has a built-in image sensor (Fig. 2).
The GUI-based tuning tool is an application operated on a PC. Its mechanism is that a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. Therefore, development can be achieved at low cost within a significantly reduced development period.
Although multiple semiconductor manufacturers now offer ISP chips, they rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what these simplified tools can perform” (THine Electronics). The situations surrounding FPGA are almost the same. “As far as we know, there are no vendors that offer both of ISP cores and GUI-based tuning tools for firmware development” (THine Electronics).
The combination of an ISP chip "THP7312" and the CDK can thus offer the best possible option at this moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. This means that they no longer have to depend on FPGA anymore. As a result, they also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. This is undoubtedly good news for XR compatible equipment that requires significant downsizing.
Technical details and methods of utilization of the GUI-based tuning tool included in the CDK will be introduced in subsequent articles.
The CDK consists of three major components: (1) hardware, (2) Graphical User Interface (GUI)-based tuning tool for firmware development named “THine Tuning Tool,” and (3) firmware templates. The hardware consists of an ISP board with THP7312, a frame grabber, and a header board equipped with a camera module that has a built-in image sensor (Fig. 2).
The GUI-based tuning tool is an application operated on a PC. Its mechanism is that a template is automatically rewritten and required firmware is newly developed by simply entering numerical values on a PC screen according to its menu or selecting parameters from a pull down menu (Fig. 3). Virtually no expertise regarding firmware development is required. Therefore, development can be achieved at low cost within a significantly reduced development period.
Although multiple semiconductor manufacturers now offer ISP chips, they rarely offer GUI-based tuning tools for firmware development. “Some semiconductor manufacturers offer simplified tools, but significant differences exist between what our GUI-based tuning tool can perform and what these simplified tools can perform” (THine Electronics). The situations surrounding FPGA are almost the same. “As far as we know, there are no vendors that offer both of ISP cores and GUI-based tuning tools for firmware development” (THine Electronics).
The combination of an ISP chip "THP7312" and the CDK can thus offer the best possible option at this moment for medium-size smartphone manufacturers, industrial machine vision equipment and surveillance camera manufacturers, and XR compatible equipment manufacturers. This means that they no longer have to depend on FPGA anymore. As a result, they also can obtain additional advantages including reduced power consumption and a downsized footprint on printed boards, as relatively higher-end FPGAs were required for mounting ISP cores. This is undoubtedly good news for XR compatible equipment that requires significant downsizing.
Technical details and methods of utilization of the GUI-based tuning tool included in the CDK will be introduced in subsequent articles.
Related Contents
- A new development environment for camera systems: CDK will solve problems caused by the fusion of cameras and AI
- Establishing a framework that multilaterally supports embedding camera functions into systems to achieve great progress toward realization of “SWARM/IoT of cameras”
- Details of GUI-based tuning tool of Camera Development Kit (CDK) for greatly enhanced ISP performance with no firmware coding