THine Value A new development environment for camera systems: CDK will solve problems caused by the fusion of cameras and AI

2019.05.21
  • Article
  • Column
Surveillance cameras, smart glasses, electron mirrors, monitoring cameras, biometrics, smartphones, car-mounted cameras, and machine vision. . . . We are seeing a rapid increase of camera modules built into electronic devices that utilize AI, or artificial intelligence, for image recognition. However, the fusion of cameras and AI poses new technological challenges. Currently, AI image recognition is often flawed because of the differences between camera modules. But THine Electronics is solving this problem with their release of a camera development environment, based on their Camera Development Kit, or CDK.

A “world full of cameras” is coming

A “world full of cameras” is coming to our planet (see Fig. 1).
Fig. 1. A “world full of cameras” is coming!

Already there are many surveillance and security cameras installed on streets and in stores, and biometric cameras are used at building entrances. People on the street wear smart glasses. Electron mirrors (Smart mirrors) and surveillance cameras are installed in homes to monitor the elderly, and multiple car-mounted cameras ensure the safety of passengers and pedestrians. In addition, many machine-vision cameras are used in industry, so many products are manufactured without any human intervention.

Without exception, all these modern cameras make use of artificial intelligence technology, because these products need to process images and videos to operate. Image recognition has many applications, for example: public cameras for monitoring pedestrian traffic and surveillance; biometric authentication using the iris of the eye, fingerprints, and even veins for biometric cameras; recognizing people’s falls and dropped items; detecting pedestrians and dangerous objects by car-mounted cameras.

A new challenge caused by camera + AI

“Camera + AI.” A new challenge is surfacing through their fusion: AI cannot always make the correct judgment because of large differences between camera modules, which cause color texture, brightness, and signal-to-noise ratio of images to differ. As a result, even if different cameras photograph the same subject, differences can occur between the images  ̶ which will cause differences in the results of AI image recognition.

Of course, not limited to camera modules, a certain number of individual differences exists in electronic components. However, according to THine Electronics, “individual differences of camera modules are much larger than those of semiconductor chips and passive components.”

Why? There are two reasons. First, unlike general semiconductors and passive components, the characteristics of camera modules and measurement methods have not been standardized in the industry. The other reason is that the manufacturing environment of camera modules varies with the manufacturer. Some of them manufacture these modules in clean rooms while others do not, in an ordinary environment with people coming and going.

Camera modules consist of image sensors, a lens, and actuators (voice coil motors) (Fig. 2).
Fig. 2. A platform for developing camera system firmware

Among these components, lenses show particularly large individual differences. Furthermore, the production tolerance in assembly is a major factor that can exaggerate these differences.

Start out problem solving

Problems are caused by individual differences in camera modules. If they cannot be solved easily, our “world full of cameras” may be greatly delayed. This would also be bad for manufacturers of camera modules and also companies that make semiconductor image signal processors (ISPs). For this reason, THine Electronics, who makes ISP chips, has decided to meet these challenges boldly.

Of course, there is already a method to compensate for such differences to some extent; it is a two-step process. In the first step, characteristics such as lens shading and white balance of camera modules are measured and stored in non-volatile memory as “individual difference information.” Then, using this information, correction processing is done using dedicated circuits integrated into the ISP and firmware. This method reduces the effects of such differences.

However, this method is tough to apply reliably. One reason is that firmware for the ISP has to be developed for making corrections, which requires specialized knowledge of both cameras and software. Another reason is that the particular characteristics that need to be corrected are different for each application, and subtle “seasoning on image quality” is required.

This requires shooting quality images where the quality can be easily authenticated by combining visible light and infrared light to do iris and vein authentication. With digital still cameras and smartphones, creating images that people consider beautiful is required. Moreover, each camera manufacturer has its own idea of what is a beautiful image, and these differences must be accounted for.

Offering “CDK + CAO”

Therefore, THine Electronics has developed a solution to solve these problems by combining the Camera Development Kit (CDK) and the Camera Application Option (CAO). We intend to provide a definitive solution that works together with our ISPs for use by manufacturers that develop camera modules and by independent design houses (IDH) for contracted development.

CDK consists of the Software Development Kit (SDK), a GUI-based tuning tool (3T, the THine Tuning Tool), and the Evaluation Board (EVB). With CDK, firmware can be developed with almost no programming. Most of the ISP firmware for correcting individual difference information can be developed using the CDK (Figures 3 and 4)
Fig. 3. Create images that are friendly to people and technology
 
Fig. 4. Create AI-friendly images

CAO is used in with CDK, with positioning as an “add-on tool.” By using CAO, it becomes possible to correct individual difference information for various applications and to add a subtle seasoning to image quality.

However, what users require from “CDK + CAO” will depend on the application. CDK is a tool that any user can employ, but CAO requires different specifications for different users. It is not realistic for THine Electronics to develop the most appropriate CAO for each user company.

Aiming to be the Linux of camera systems

Therefore, THine Electronics is considering building an “ecosystem” for camera system development by involving IDH.

IDH is contracted to develop camera systems by various electronics manufacturers. If THine Electronics' ISP and CDK are adopted, the CAO corresponding to the application should be developed. IDH is a company that excels in camera system development. Any CAO developed by IDH is expected to have a high level of quality.

Then, THine Electronics discusses details with IDH so they understand that the CAO will be provided as an add-on tool for CDK for other manufacturers as well as to IDH. It means that the CAO will be rolled out. 

On the other hand, if IDH approves a version of the CAO, it will be certified by THine Electronics as an “ISP Gold Partner (IGP).” Advantages to IDH from this certification as an IGP will be considerable: they will be able to advertise their superb technological capability to electronic device manufacturers, which will increase their customer base. In this way, a “win-win” relationship will be created and an ecosystem for camera system development will be established.

THine Electronics aims to make CDK the “Linux in camera systems.” Linux is an open-source operating system (OS) for computers that has been distributed free of charge. However, later on, various companies have developed application software and add-on tools that run on Linux and are offering them for free or for a fee. Thus, the Linux ecosystem is now growing on a very large scale and is widely adopted for personal computers, servers, supercomputers and embedded devices.

As for CDK, THine Electronics will expand their partner companies and expand the CAO portfolio in cooperation with IDH. In this way, they are trying to increase the scale of the ecosystem consisting of user companies and IDH. In the future, THine Electronics will become a semiconductor manufacturer offering ISP as well as a coordinator for the completed ecosystem.

Change the camera development environment

To position CDK as the Linux in the camera system is a major goal for THine Electronics, while aiming for the larger picture: to change the camera development environment in Japan.

Japan was once a great power in the field of cameras, the top runner in the world in terms of engineering capability, product planning ability, and business. But what about now? It is not easy to say that Japan is still a world leader in any of them. Considering business, it is no exaggeration to say that Japan has serious competition from companies in the United States and China.

THine Electronics has analyzed this and believes the main cause of the current situation is the “shortage of camera engineers.” THine Electronics researched the ISP business examining camera system development companies in China and the United States and found they have many camera engineers and can develop a project more quickly than companies in Japan. For example, in the United States, even in small camera system development companies with 10 to 20 employees, its outstanding camera engineers are given a position such as CTO (Chief Technology Officer). These companies groom their young, talented engineers with great care.

On the other hand, it seems that the number of engineers in camera system development in Japan are decreasing. Highly skilled, experienced engineers are still active, but it seems difficult to obtain a large number of young engineers to assign jobs that lead to smaller opportunities for technology transfer. High-quality camera technology in Japan has to be transferred before experienced engineers leave their workplace.

Therefore, THine Electronics aims to change the development environment for camera systems in Japan by widely spreading CDK to allow Japan to regain its position as a great power in the field. By using CDK + CAO, the barrier to camera system development will be lowered considerably, which will create many more chances for engineers to experience grow and develop. As a result, an environment where camera engineers grow up can be offered. Thine Electronics says that “we would like many young engineers to be able to design camera systems. We will support them. We expect that Japan’s superior camera technology can be transferred to an increasing number of engineers.”