Artificial neural network based color identification and detection to help color blind people

Abstract

Background

Color vision defects (CVD), another name for color blindness, is a condition characterized by signs of failure of the optic nerve, which connects to the brain. People who suffer from color blindness have trouble identifying colors both physically and perceptually. Thus, that those who suffer from it are unable to appropriately differentiate between colors.

Objective

In order to recognize colors, victims require a tool. A technology that can assist color blind people in recognizing colors is being tried in this investigation.

Methods

The Artificial Neural Network (ANN) algorithm is embedded into the hardware of the system, which is based on artificial intelligence (AI). A light sensor based on the Red, Green, and Blue (RGB) filter is employed. RGB data will be sent into the ANN. An Arduino Due microcontroller is equipped with an ANN program that functions as a data processor. This system’s outputs are the color speech on the speaker and the color text on a dot-matrix LED display.

Results

This system’s output is represented by the color that is shown on the LED dot-matrix display board. The method is successful in identifying all 16 of the training colors. Targets and outputs have an MSE of roughly 0.0001.

Conclusions

The findings of this study should assist those who are color blind in identifying colors by listening to the speaker’s sound and reading the display board that the system displays.

Keywords

CVD color identification ANN RGB sensor

Introduction

A particular spectrum present in ideal light (white) is called color or hue. The wavelength of the light determines a color’s identity. Blue, for instance, has a wavelength of 460 nm. Often called the visible spectrum or visible light, the wavelength of color that the human eye can still detect falls between 380 and 780 nm. Color in optical equipment can also refer to how the brain interprets a combination of the three fundamental hues of light—red, green, and blue—in a particular composition. For instance, the color magenta can be created by combining 100% red, 0% green, and 100% blue. The degree of brightness or darkness of a color determines its value. The inclusion of black or white affects its value. This value is established in the RGB system by adding red, blue, and green components in precisely the same proportions, though not always in that order. Color is a component of light that an object reflects and that the eye interprets according to the light that strikes the thing.¹

Color vision diseases (also called color vision deficiencies, CVD, or color blindness) refer to disorders in which a person has difficulty distinguishing certain colors or perceiving color differences normally. These conditions occur when the cone cells in the retina, the photoreceptor cells responsible for detecting color, are absent, not functioning properly, or detect colors differently than typical. In color blindness sufferers, the cells that detect color pigments are damaged or not functioning. As a result, the eye cannot detect certain colors or even all colors.² Symptoms of color blindness in each patient can be different, depending on the pigment cells that are damaged or not functioning. These symptoms are divided into three types, there are red-green, blue-yellow, and total.³

A device or system that can identify and characterize the color of items in front of color blind persons is required in order to help them overcome their inability to recognize existing colors. Several issues and system-related concerns are taken into account when implementing this system, such as sensors that identify things with specific colors, processors capable of processing sensor-derived input data to generate output in the form of precisely defining hue, users can read and see actuators in the form of system output displays, and algorithms are employed to process data.

A system using RGB-based sensors, an Arduino Due microcontroller processor, and an output in the form of a dot-matrix display that can show both running and still text is proposed in this study as a solution to these issues. Artificial intelligence (AI) based on artificial neural networks (ANN) with iteration and error correction via the backpropagation method is the methodology to be utilized.^4,5

Some articles have been discussed about color blind people assistant, but there are leak in some features, like in artificial intelligent utilizations, or in small display that causes misinterpretation in reading result display of the system. Some studies also use mobile applications and hardware with various features, but in their implementation they require large memory and high-frequency processors, thus requiring higher resources and costs.^6,7

The method

In order to assist color blind individuals in differentiating and identifying the colors of visible items, this project aims to develop a system that can detect object colors. The system is constructed with sensors that can identify the fundamental hues red, green, and blue, also known as RGB (red-green-blue). The TCS3200 sensor type is intended to be chosen. A processor in the form of an Arduino Due microcontroller with an inbuilt artificial neural network algorithm will process sensor input in the form of RGB intensity values. Determining the hue of the observed object is the system’s output. The maximum number of colors that can be identified in this investigation is 16.

The hardware

With multiple components that have been identified, the hardware design adheres to the color identification system block depicted in Figure 1 so that the block is also referred to throughout manufacture. A description of the implementation of hardware manufacturing utilizing sensor, processor, and actuator components is depicted in Figure 2. As displayed in Figure 3, the actuator consists of four MAX7219 type dot-matrix displays, the sensor is a TCS3200 type color sensor, and the processor is an Arduino Due microcontroller.

Figure 1.

System diagram.

Figure 2.

Hardware diagram.

Figure 3.

The hardware.

The purpose of the sensor is to record the amount of light that a particular hue reflects. The sensor is internally illuminated and shielded from external light intensity by a black tube. The Red, Green, and Blue (RGB) color filter, which is the fundamental color that creates additional colors, allows the sensor to filter the colors that are recorded. Digital gray level values are created from each RGB color that is absorbed and sent to the sensor. The processor unit will subsequently process these values in order to train and identify the observed color. Training and testing are the two phases of the processor’s processing, which include training and implementation.

Real data on color values acquired by the sensor is gained through training. Training data is obtained by repeatedly training each color to determine its RGB value. Following the acquisition of training data, the data is fed into an artificial neural network (ANN) system that is integrated into a processor program and has a target in the form of a color that has been converted to four-bit binary. The ANN system is used to continue training in order to get network weight data. The implementation method will make use of network weight values.

For testing and implementation, the sensor is connected immediately to the ANN system, which has been weighted based on the training results, eliminating the need for additional training. A dot-matrix arrangement of LEDs will make up the actuator unit’s display of the color identification findings. The function of the dot-matrix display, which consists of four 8 × 8 LED units, is to show the processor’s processed results as text names of colors that were successfully detected. The training provided and the number of color categorization bits (4 bits, 2⁴ = 16) allow for the display of a total of 16 colors.

The software

The artificial neural network algorithm is designed to have three neurons in the input layer, six neurons in the hidden layer, and four output layers.⁸ The output layer is in the form of one code encoding that forms a four-bit binary code so that the output can form a variation of 2⁴ (16) color outputs. The configuration of network is described in Figure 4.

Figure 4.

Artificial neural network design.

The formula of neural network is designed by referring to:

y_{j} = σ (\sum_{i = 1}^{n} w_{i j} x_{i} + b_{j})

(1)

And the sigmoid activation function used in this system is:

σ (x) = \frac{1}{1 + e^{- x}}

(2)

Backpropagation refers to the process of updating weights by changing old weights to new weights. The variation is determined by the difference between the output and target deviations.

w_{n e w} = w_{o l d} - α \frac{\partial E}{\partial w}

(3)

It is also calculated to the bias value,

b_{n e w} = b_{o l d} - α \frac{\partial E}{\partial w}

(4)

After updating the weight and bias values, the signals are sent back to the network and repeating the process that has been done previously.⁹ This process will be repeated continuously until the optimum weight value is reached when the minimum error value is obtained, or after reaching a predetermined number of iterations. The iterations of this process depend on the learning rate of ANN in 0.5. This iteration will be monitored by the mean square error (MSE) formula to determine the error value.

To find the error, the formula used is:

E = \frac{1}{n} \sum_{1}^{n} {({t a r g e t}_{i} - {o u t p u t}_{i})}^{2}

(5)

The design of this software consists of: Data Acquisition, ANN Training, ANN Testing, and Implementation. The flowcharts of the software are detailed in Figure 5.

Figure 5.

Software Design Flowchart: a. Data Acquisition, b. ANN Training, c. ANN Testing.

The data acquisition flow follows the flowchart in Figure 5(a). Data is taken by reading color samples carried out by the sensor, and data recording is done by viewing the computer screen display in the form of serial monitor data. Table 1 shows the sample data of color training.

Table 1.

Sample of color training.

As seen in Figure 5(b), Training process is done using artificial neural networks with backpropagation algorithm. The training process runs by comparing the training output results with the given target.¹⁰ Iterations are given 1000 times, and will stop when it has reached the iteration or the error between the output and the target has been achieved is smaller or equal to 0.0001.

ANN testing and implementation directly uses hardware units in the form of sensors, processors, and actuators, without being connected to a computer unit.¹¹ The power source uses a separate battery and does not use a battery from a laptop/computer USB. And the process is shown in Figure 5(c).

The program starts with system initialization and given inputs-targets as shown in Table 1 to the system,

The activation is using sigmoid activation formula,

Before getting the weights from the network, it must be declared in the program to use random value in range. The network weights are from input layer nodes and hidden layer nodes.

The weight will be processed in ANN formula for forward,

And then the backpropagation algorithm run to update the weights.

The program loops until 1000 iterations or the minimum error is reached. After the loops are ended, the final weights are saved in the memory.

The final weights will be used in testing or implementation process.

Then, the colors are declared and being initiated.

So, the forward ANN processes start to read values of the sensor and definite certain color detected.

The training process

4. A number of preparations pertaining to the training objects were made prior to the training. To enable the sensor to cover the existing color circle region, the training objects were colored circles printed on paper with diameters tailored to the sensor area.^2,12 The color circles prepared were 16 as can be seen in Figure 6. The training colors will be used to obtain data on the values produced by the sensor when faced with the color circle.

Figure 6.

Color prints for training.

A serial monitor is used to link the hardware to the computer running the Arduino IDE software in order to retrieve the values. The monitor screen of the laptop or computer shows the sensor identification data, as shown in Figure 7. Changes and fluctuations in each color are recorded on the monitor. As indicated in Table 1, each color’s RGB value change of 10 units is recorded, for a total of 160 color value variation data. In order to train artificial neural networks (ANNs), these data constitute input data. In the meantime, one hot decoding with 4-bit variation data is used to extract the target data from the color data that has been encoded.

Figure 7.

Serial monitoring data acquisition.

Artificial neural networks with the backpropagation method are used for training. By comparing the training output results with the specified target, the training process is carried out. Iterations are performed 1000 times, and they end when the iteration is completed or the output-to-target error is less than or equal to 0.0001.

Before being implemented and tested in the system, the output results are checked to see if they are in accordance with or close to the target value. The results of the check can be seen in Table 2. Output values above 0.9 are rounded and considered as a value of 1, while values below that are categorized as a value of 0. From the table, it shows that the output of the system processing results can be successful, although it does not reach absolute 1 or absolute 0.

Table 2.

Output of training.

The Testing and Implementation process

To make sure the system has operated as intended by the manufacturer and design, testing is done on the constructed system. Hardware components such as sensors, processors, and actuators are used in direct testing; these components are not connected to the computer unit. The power source does not use a battery from the laptop/computer USB instead, it uses a separate battery.

The Testing and Implementation involved seven respondents to ensure that the system worked well and could be interpreted correctly by the respondents.

Result

The test results show that the system runs well, as seen in Table 3 which contains real images of the test results for each given color. All colors can be detected as indicated by the text displayed on the dot-matrix screen on the actuator unit.

Table 3.

Color detection results.


Black	Blue	Cyan	Green

Grey	Lime	Magenta	Maroon

Navy	Olive	Purple	Red

Silver	Teal	White	Yellow

The system is tested to seven respondents and the result is shown in Table 4.

Table 4.

Respondents’ result.

Respondent	Black	Red	Lime	Blue	Yellow	Cyan	Mgnta	Silver	Gray	Mroon	Olive	Green	Purple	Teal	Navy	White
A	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
B	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
C	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
D	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
E	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
F	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
G	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓	✓	✓	✓

Note. ✓ = success, x = failed.

Almost all respondents could interpret well to the system output. Only three respondents failed to recognize the teal and maroon colors, possibly due to the imprecise placement of the sensor on the color object. By these results, the system performed well with 97.32% accuracy.

Conclusions

(1) The system for identifying and recognizing colors can function effectively overall. The system can identify all trained colors, and the dot-matrix display can display text based on the color that the sensor detects.

(2) The artificial neural network can correctly recognize colors from data supplied by the sensor based on the training results between input and target, such that the network weight has been conditioned when there is input fed to the system.

(3) This approach can help color blind persons distinguish existing colors, particularly fundamental colors and key complementary colors (R, G, B, C, M, Y), as well as various other colors totaling 16 colors because the text display can be read clearly, with 97.32% accuracy number.

Footnotes

Acknowledgments

We thanks to LPPM of Universitas Islam Sultan Agung for the support.

Author Contributions

Conception: ENB

Performance of work: ENB and MI

Interpretation or analysis of data: ENB and JPH

Preparation of the manuscript: ENB

Revision for important intellectual content: MI and JP

Supervision: JPH

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Valdoria

JCJC

Natividad

Jader

, et al. An android based Mobile application for color recognition assistance for colorblind individual through color segmentation using color threshold, J Educ Teach Trainers 2023; 14: 788–797.

Bhargav

. Image processing for colour blindness correction or colour detection. Int J Res Publ Rev 2022; 3(11): 954–957.

Costa

MCJ

Amarasinghe

Jayasekara

, et al. Third eye - a voice assistant-based Mobile application for blind and colorblind people. International Research Journal of Innovations in Engineering and Technology 2023; 7(6): 12–19.

Budisusila

Arifin

Prasetyowati

SAD

, et al. Artificial neural network algorithm for autonomous vehicle ultrasonic multi-sensor system. EECCIS 2020 - 2020 10th electr. Power, electron. Commun. Control Informatics Seminar 2020: 128–131.

Budisusila

Khosyi’In

Prasetyowati

SAD

, et al. Ultrasonic multi-sensor detection patterns on autonomous vehicles using data stream method. Int Conf Electr Eng Comput Sci Informatics 2021; 2021b: 144–150.

Natsir

. Identification system for color blind puppets output text and sound, pp. 5–9.

Alfiansyah

Gozali

Natakusuma

. Color blind assistant app based on computer vision using openCV. Jurnal Mantik 2023; 7(2): 588–596.

Budisusila

Arttini

Prasetyowati

, et al. Neural network training for serial multisensor of autonomous vehicle system. Int J Electr Comput Eng 2022; 12: 5415–5426.

Budisusila

Prasetyowati

Arifin

, et al. Real time system handling using multi fixed weight artificial neural network. Int Conf Electr Eng Comput Sci Informatics 2023: 504–508.

10.

Budisusila

Prasetyowati

SAD

Arifin

, et al. Advanced artificial neural network for steering and braking control of autonomous electric vehicle. Int J Electron Eng Res 2024; 12(3): 870–880.

11.

Wang

Zhang

. Artificial intelligence for visually impaired. Displays 2023; 77: 102391.

12.

Yanto

ABH

Fauzi

Maulana

, et al. Color blind detection application design using the Ishihara method using the android OS. J Infokum 2022; 10(5): 227–236.

Respondent	Black	Red	Lime	Blue	Yellow	Cyan	Mgnta	Silver	Gray	Mroon	Olive	Green	Purple	Teal	Navy	White
A	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
B	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
C	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
D	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
E	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
F	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
G	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓	✓	✓	✓

Respondent	Black	Red	Lime	Blue	Yellow	Cyan	Mgnta	Silver	Gray	Mroon	Olive	Green	Purple	Teal	Navy	White
A	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
B	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
C	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
D	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
E	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
F	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
G	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓	✓	✓	✓

Respondent	Black	Red	Lime	Blue	Yellow	Cyan	Mgnta	Silver	Gray	Mroon	Olive	Green	Purple	Teal	Navy	White
A	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
B	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
C	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
D	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
E	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓
F	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
G	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	x	✓	✓	✓	✓	✓