Abstract
Illustrations referring to the visible aspects of objects in tactile books may be difficult for children with visual impairments to understand. To solve this problem, a new illustration design was developed based on modalities other than vision. The objective of the study is to evaluate the comprehension by children with visual impairments of a multimodal book whose illustrations refer to haptic and audio modalities. To examine the role of residual visual experience in comprehension, the book was tested in children with two profiles of visual impairment: 5 blind children and 6 with low vision, aged 5–12 years. To assess their comprehension as they discovered the book, we observed a matching between the text and the exploration of the illustrations, and the production of expected manual explorations by children. After discovery, we asked the children about how they handled the objects in the book and the sounds they heard. The main results showed no significant difference between the two groups of children on the matching score, manual exploration expected, and description of sounds perceived or handling performed. The book was very well understood and appreciated by both groups. These results could indicate that residual visual experience is not necessary for this task. Results suggest that exploring other sensorial experiences could be a more inclusive way to represent objects in tactile books that are more suitable for all children, regardless of their visual experience.
Introduction
Tactile illustrated books are devices that combine text in large print and in Braille with tactile illustrations. The presence of tactile illustrations in books is beneficial for the development of language and emergent literacy in blind and sighted children (Carney & Levin, 2002; Stratton & Wright, 1991). Moreover, they play a role in children’s comprehension and memorization of stories (Bara et al., 2018; Gentaz et al., 2003; Heller & Gentaz, 2013). For children with visual impairments, tactile illustrations are important as they develop tactile interest in Braille and the processes of reading by helping children to take a more active role in reading (Stratton & Wright, 1991).
Nowadays, tactile illustrations in tactile books often represent the formal aspects of objects based on visual reality, for instance, through texturing the different parts of the image or highlighting the contours of the shape. Nevertheless, several studies have shown that blind adults and children have difficulty identifying the objects depicted in two dimension (2D) whose formal aspects are highlighted (Lederman et al., 1990; Theurel et al., 2013; Thompson et al., 2003).
Among the factors that could explain this difficulty, studies have pointed to subjects’ limited ability to generate visuospatial imagery (Lebaz et al., 2012; Lederman et al., 1990; Thompson et al., 2003) and their lack of familiarity with visual conventions (for a review, see Valente & Gentaz, 2019). Indeed, figurative proprieties on drawings depend on factors, many of which are related to the visible aspects of the objects but are also subject to cultural adjustments and cognitive categorization activities (in the drawing, the length of the dog’s snout distinguishes it from the cat’s for example) (Valente, 2015). For someone who is blind from birth and unfamiliar with these conventions, it is difficult to make an unambiguous link between the tactile forms touched and the object they represent. Other studies on the identification of tactile images (Vinter et al., 2012) and tactile graphics (Zebehazy & Wilton, 2021) have shown that these comprehension problems could be reduced with prior learning of perceptual strategies to explore tactile contents. Vinter et al. (2012) have shown that identification difficulties result from the nature of the explorations deployed by the subject when interacting with tactile images. More recently, Zebehazy and Wilton (2021) have shown that prior experience with tactile graphics and content knowledge can positively affect the interpretation performance of these educational devices.
In youth literature, forms and styles of illustrations are highly varied, as their primary purpose is to awaken and maintain the interest of young readers and/or to support their understanding and memorization of the story (Valente & Gentaz, 2019). Recently, studies have shown that replacing visual content with relevant textures, three-dimensional (3D) elements or manipulations would improve recognition of illustrations and story processing in tactile books (Bara et al., 2018; Mascle et al., 2022; Valente et al., 2021). In an exploratory study, Bara et al.(2018) showed that the processing of information in a story was enhanced through the manipulation of miniature objects, especially in congenitally blind children. Indeed, the use of 3D illustrations should lead children to use more varied exploratory procedures than classical illustrations and therefore to collect more information about the objects represented. Three-dimensional illustrations could also be used to reactivate real motor components involved in interaction with objects. In this line, Valente et al. (2019, 2021) developed and tested a new illustration technique that explores finger simulation of the body’s experiences in interaction with objects. The illustrations are 3D mini-scenarios that children explore with their two fingers simulating two legs representing actions (such as climbing the stairs or jumping on a trampoline). In a preliminary study, Valente et al. (2019) examined whether finger gesture patterns simulating actions performed on objects (action simulations by fingers gestures [ASFGs]) are the same for blind and sighted people. Researchers asked blindfolded sighted adults, early blind adults, and late blind adults to produce ASFGs for 18 action concepts (i.e., climbing stairs or sliding on a toboggan). ASFGs produced by the three groups were videotaped and analyzed by sighted decoders in a visual recognition task. Results showed that gestures produced by the three groups were very well recognized by sighted decoders. Indeed, the same motor pattern was found in the sighted and blind adults’ simulations. Few differences were found referring to the illustrative components of ASFG (i.e., how the gesture is seen by the interlocutor), which are unknown to those blind from birth. Despite these differences in the appearance of the gesture, the results have shown that the procedure for simulating a real action by finger gestures has symbolic relevance for people regardless of their visual experience. In a later study, Valente et al. (2021) examined the identification of 3D miniature action models in comparison with classical tactile illustrations of the same action objects. Results showed that action objects had higher recognition scores when they were illustrated with 3D miniature action models, both for blind and sighted children. Results suggest that the ASFGs procedure associated with 3D miniature models could be an illustration technique suitable for children, regardless of their visual experience.
These results encourage children’s book publishers to explore other forms of illustration that refer to haptic and sensorimotor experiences with objects. Another promising track is the addition of sound to enrich the sensory experience of reading. As a spatial modality, hearing provides information about the location of sounds (direction and distance) and their other properties (pitch, timbre, etc.). More generally, it is specialized in capturing and processing temporally successive data and is particularly important for learning and using spoken language.
There is clear evidence that combining auditory and tactile information can improve performance in a variety of tasks. For example, Ro et al. (2009) reported that sound enhances touch perception. They reported that a simultaneous sound stimulus aided the detection of a cutaneous stimulus. However, the sound had to be on the same side of the body and the effects depended on frequency. Different auditory frequencies did not provide the benefit that they tested for. Ro et al. (2009) concluded that there may be similar coding mechanisms in both sensory modalities.
It is interesting that the effects of interactions between sound and touch may not be identical in blind as in sighted individuals. Many blind people rely on sound for cues about the distance of objects, their identification, obstacle detection, and even shape information.
The aim of the study is to evaluate comprehension by children with visual impairments of a multimodal book in which illustrations are linked to haptic and audio perception. Two profiles of children with visual impairments are included in this study: congenitally blind children and children with low vision working without sight. Following studies that have shown a better recognition of haptic content at the expense of visual illustrations in relief, independent of the reader’s visual experience, our objective here is to examine whether this multimodal book is adapted to any profile of visual impairment by the fact that it appeals to perceptions other than vision. If visual experience has an effect on book comprehension, we expect children with low vision to perform better on this task. On the contrary, if the visual experience is not necessary for the comprehension of the book, one should expect the same performance from both the groups.
Method
Participants
Tables 1 and 2 present the characteristics of the 11 children who participated in the study. The group of blind children (Table 1) was composed of one girl and four boys (M = 9.4, SD = 2.3, range = 6–12 years), and the group of children with low vision (Table 2) was composed of three girls and three boys (M = 8, SD = 2.37, range = 5–11 years).
Characteristics of blind children who participated in the study.
ICD-10: International Statistical Classification of Diseases and Related Health Problems, 10th Revision.
Characteristics of children with low vision who participated in the study.
ICD-10: International Statistical Classification of Diseases and Related Health Problems, 10th Revision.
Blind participants belonged to one of the three categories of blindness (Categories 3, 4, or 5) of the International Statistical Classification of Diseases and Related Health Problems, 10th Revision, and those with low vision were included in one of the two categories of low vision (Categories 1 or 2). The children were recruited in two institutes for children with visual impairments in France. They attended regular classes or localized units for school inclusion classes (i.e., Special classes for pupils with disabilities in a regular school in France) and received specialized support from specialist teachers who provided intervention in the classroom or a special educational center. All the children have a congenital visual impairment. No child has a diagnosed associated disorder that prevents the completion of the task. This study respects the ethical principles for research involving human subjects (World Medical Association Declaration of Helsinki) and was approved by the Cantonal research ethics commission on Swiss Ethics Committee (project ID 2020-01571).
Stimuli
The prototype contained four pages in which electronic components were inserted and managed by a tiny digital microcontroller (Arduino Nano) embedded in the back cover of the book. In order to have good-quality sound feedback, the book was connected to an external device – a computer. This allowed the children to hear different sounds in response to their tactile exploration of the book. Thus, they had a direct sound response depending on the elements they were exploring on the page, and they could hear instructions on each page that were written as well. In this version of the prototype, the text was only transmitted by a voice-over. Figure 1 presents technical components of the multimodal book.
Technical components of the multimodal book.
Figure 2(a) presents the book’s cover. The four pages of the book are inspired by the French album Petite main petit pouce (Small hand small thumb) of Martine Perrin (original publisher: Seuil Jeunesse, publisher of the tactile version: Les Doigts Qui Rêvent, LDQR). Solène Négrerie and Dannyelle Valente decided together on the pages and choice of design. The illustrations of the multimodal prototype were designed by Les Doitgs Qui Rêvent with specific materials and format to accommodate the electronic components. Thus, the electronic components of the book are designed by Umar Farood, master student, and Dominique Archambault. Illustrations were made with materials that were pleasant to the touch and frequently found in tactile books (wood, foam board). They were attached to the page so that they would resist the expected manipulations. The publishing house partner of this study, specialists in editing series of books with multiple materials intended to be handled, attached the materials to the pages.
Book cover (a) and Page 1 of the multimodal book (b).
Figure 2(b) presents the first double-page of the book and included a 3D miniature model of a stair on the left-hand page and the written instruction “Now you climb the stairs with your fingers” on the right-hand page. This instruction was announced to the child using a voice-over (a woman’s voice). When the children touched the steps of the stair with their fingers, footsteps were produced.
The second double page consisted of several sets of fabric forming two large flowers and a small flower on the left-hand page. On the right-hand page, the written instruction expressed by a voice-over was “Caress the field flowers.” When the children touched the flowers, a soundscape was created (sea, wind and birds sounds).
The third double page included a small cardboard door that could be opened with a handle on the left-hand page. On the right-hand page, the written instruction expressed by a voice-over was “knock on the door, knock, knock, knock and come in.” A squeaking sound was produced when the door was opened and a slamming sound when it was closed.
The last double page consisted of a small cotton mouse that could be attached to two different parts of the left-hand page with a Velcro fastener. The mouse could be hung in its original position at the bottom right of the page or the top left, under a cloth representing a net. On the right-hand page, the instruction expressed by the voice-over was ‘Oh, a little mouse! Quick, hide it so that the cat doesn’t find it!’ Squeaks were made when it was placed under the net and if it was pressed down after being hidden.
Procedure
The children were first asked by an experimenter whether they had practice with tactile books (see Tables 1 and 2 about participants’ practice). Then, the experimenter introduced the book and explained that a voice would say what they had to do. The children were always asked to wait until the end of each instruction to begin exploration. On each page, the children could move on to the next page after having done what the instruction asked. The experimenter could tell a child to go to the next page if he/she had succeeded or asked to turn the page. A blackout curtain attached to the table allowed children with low vision to discover the book without the use of their visual remains. This device made of possible to study the effect of visual experience by preventing the results from being influenced by the use of sight during the task.
Sessions were recorded on video (only the children’s hands were filmed). Two research assistants, unaware of the purpose of the experiment, coded the videos. A Cohen’s Kappa was calculated to measure their inter-reliability which was considered as a strong agreement, κ = .923, p < .001. For the data the two coders disagreed with, the video was watched again and discussion between the two coders led to an agreement.
In order to measure the children’s comprehension, two variables were analyzed during the discovery of the book (matching score and the production of manual exploration expected) and two other variables after discovery (description of handling performed and sound perceived). In particular, for the matching score, assistants coded whether the children carried out explorations in response to the instruction (e.g. if a child tried to climb the stairs with his/her fingers on Page 1 in response to the instruction “Now you climb the stairs with your fingers”). Assistants coded 1 for matching and 0 for not matching. Then, for each page, assistants noted 1 if a child performed the manual exploration expected and 0 if not (for example, if children use their index and middle fingers to climb the steps on Page 1 or if they perform the exploratory lateral motion procedure to caress the flowers on Page 2. Immediately after the discovery, we asked the children what they should do on each page of the book (handling performed). Then, the experimenter came back to each page to ask them what sounds they had heard following the handling performed (sound perceived). The score on handling performed and sounds perceived was 1 if the answer was correct and 0 if not. Supplemental Appendix 1 presents the rating criteria for matching, manual exploration expected, description of handling performed, and description of sound perceived.
Finally, the experimenter asked some additional questions in order to determine whether children enjoyed the book (coded “1” if they did and “0” if not) and if they had any difficulties on a page. The children were also asked what page they preferred (they could choose up to four pages).
Results
Since the sample of our data was still small and not normally distributed, Mann–Whitney U tests were chosen to conduct our analyses.
First, Mann–Whitney tests were used to control for the effect of practice with tactile books on our variables. The effect of practice was not significant on matching scores U = 10, p = .50, manual exploration expected U = 9, p = .39, handling performed U = 10, p = .50, or sound perceived U = 11.5, p = .70.
Comparing the two groups of children, no significant difference was found between blind children (M = 3.4, SD = 0.89) and those with low vision (M = 2.66, SD = 1.5) on the matching score (U = 10.5, p = .73). Nor did blind children (M = 3.2, SD = 0.84) and those with low vision (M = 3, SD = 1.09) show any significant difference on the manual exploration expected (U = 14, p = .92). No significant difference was found between either blind children (M = 3, SD = 1) or those with low vision (M = 2.5, SD = 1.38) on sounds perceived (U = 11.5, p = .58). Finally, analyses showed that the description of handling performed was not significantly different between blind children (M = 3.8, SD = 0.45) and those with low vision (M = 3.33, SD = 0.82) (U = 10, p = .41).
Table 3 shows the number of participants by profile (Blind = B and Low Vision = L) who succeeded in matching, expected manual exploration, description of handling performed, and description of sounds perceived as well as the total success score (%) for each page. Results show that the two groups succeed very well in the matching (between 64% and 91%) and performed manual explorations expected (between 73% and 100%) on the flowers, door, and mouse pages. The results were lower for the stairs page (55% for the matching rate and 55% for manual exploration expected). Specifically, results were poorer for half of the children with low vision and only two of them succeeded in matching and produced the expected manual exploration on the stairs page. Moreover, children with low vision performed less well on matching and expected finger gestures on that page (half of the children succeeded in matching and only 2 of 6 produced the manual exploration expected). Three out of the five blind children succeeded in matching and produced the manual exploration expected. This non-significant difference reflects the fact that the percentage of blind children who produced maximum scores on the matching, the manual exploration expected, handling performed and sounds perceived tended to be non-significantly higher than that of children with low vision (for every rate on all four pages except the expected manual exploration on the second and third pages and the sounds recall on the third page).
Number of participants by profile (Blind = B and Low Vision = LV) who have succeeded in matching, expected manual exploration, description of handling performed, description of sounds perceived and total score (%) for each page.
On the door page, the children did not dare to open the door spontaneously; some of them even asked the experimenter before if they could lift the handle because they were afraid to break it. Moreover, some of the children did not knock on the door with their fist but only with their fingers, thus not knocking as they would actually do on a real door. This page (73%) and the stairs page (55%) had the lowest rate of expected manual exploration. Indeed, these were the ones that required imitating an action by embodying a character – the first page by imitating two legs climbing stairs with fingers and the third page by imitating the same gesture children would make when opening a door (knocking and opening the door).
On the mouse page, no difficulty was noted but children verbally stated that they were afraid of breaking the book if they pulled the mouse from the Velcro. Thus, the experimenter had to encourage them to detach the mouse.
The page that had the lowest rate of description of sound perceived was the mouse page (55%) although it was the one that had the highest rate of handling performed (100%).
Finally, it should be noted that the book was appreciated by all the children. When asked by the experimenter why they liked the book, 64% of the children indicated that they liked being able to touch and manipulate the book while hearing sounds. The children’s most favorite page was the door page (73%), while the stairs page and the flowers page were tied for their least favorite (36% in both cases).
Discussion
The aim of this study was to evaluate the comprehension of a multimodal book associating manipulation and sounds by two profiles of children with visual impairments. Our results did not show a significant difference between blind children and those with low vision on the matching score, manual exploration expected, description of sounds perceived, or handling performed. These results could indicate that visual experience is not necessary for the comprehension of this book.
Qualitative analyses revealed that most of the children in both groups succeeded in matching and produced the expected manual exploration for the flowers, door, and mouse pages. The few children who did not succeed in these three pages did not dare to manipulate the illustrations – even with instructions that guided them. Indeed, for the door and mouse pages, some children expressed that they were worried about breaking the illustrations. Indeed, it was those illustrations where it was necessary to completely unhook the element of the book (mouse) or to completely open a shutter (door) that caused concern. The children needed the approval of the experimenter to be sure that they could actually perform these actions. This finding is important for book creators (such as publishing houses) because it indicates that for some blind children, adult guidance in discovering a book would be essential for them to explore it freely.
Concerning the first page that required an action simulating climbing stairs by finger gestures, the matching and expected manual exploration rates were lower. Nevertheless, given that recognizing a model of a stair and producing the finger gestures have been tested in previous studies (Valente et al., 2019, 2021) and shown to be successful, we believe that the low rates reflect children’s difficulties in familiarizing themselves with the book procedures. Indeed, we suppose that an introduction page to the procedures of the book would have allowed the children to project themselves more easily in the book from the first page (e.g., a page that introduces children to ASFG and asks them to simulate the legs of a character with their two fingers). Moreover, the matching rate and manual exploration expected on that stairs page were non-significantly lower for children with low vision than for blind children. This observation applied for every rate on all four pages except the expected manual exploration on the flowers and door pages and the sounds perceived on the door page. This may result from the nature of the task. The exercise of tactile exploration of a book without the use of residual sight was something new and special for the children with low vision, whereas the blind children were exploring a book in the usual way. However, as visual experience and practice of tactile books were not significant on the matching score, manual explorations expected, handling described or sounds perceived, results still suggest that the comprehension of this book would not depend on either the haptic or the visual experience of the reader.
The pages that had the lowest rates of expected manual exploration were those that asked children to imitate real gestures, either the same gestures they would perform when opening a door or using their fingers to imitate the legs of a character climbing stairs. One hypothesis could be that children with visual impairments were not used to discovering illustrations that ask them to embody a character or to simulate the real gestures they would perform in a certain context. Indeed, some studies showed that symbolic play is less familiar to children with visual impairments (Valente et al., 2020). Moreover, the door page was the children’s favorite page, while the stairs page was one of their least favorite. Thus, it cannot be argued that they did less well on these pages because they liked them less. Furthermore, these pages had high rates of the description of handling performed, meaning that children understood what they had to do on the page.
Finally, the majority of the children managed to describe the sounds perceived and handling performed on all the pages. However, the mouse page had the lowest rate of sounds perceived and also the highest rate of handling described. Thus, it would seem that when children focused on the sounds, their responses would be lower for the handling. Nevertheless, this does not call into question the choice of a book mobilizing touch and hearing, since the majority of children said they liked the fact that they could manipulate the book and have corresponding sound feedback.
It should be noted that our results are based on a small sample size and any generalization should be taken with caution. This limitation is explained by the great heterogeneity of the profile of visual impairment and therefore the challenge of constituting homogeneous groups of participants (Heller & Gentaz, 2013). In this study, we had to select participants with a congenital visual impairment and no associated disorders, which further reduced the sample size.
For future research, it would be interesting to evaluate a group of sighted children to determine if the book is also comprehensible to this population. On the other hand, it is also important to conduct new studies to assess book comprehension by children with additional disabilities who represent a significant part of the population of children with visual impairments (Hatton et al., 2013). If so, it would be essential to encourage the development of inclusive multisensory books that would create a common reading universe among blind, sighted children, and children with low vision.
The prototype version connected to the computer by a USB cable was used for this first test, given that the objective here was to evaluate children’s comprehension of the multimodal illustrations and to determine the relevance of integrating this content into future books. Following these encouraging results, the project team has been working this last year on a new prototype ‘Kapi Capitaine’. In this prototype, conductive fabrics and foams associated with electronic circuits are used to trigger sounds and sound atmospheres according to the readers’ gestures. In this new prototype, the book is no longer connected to a computer. It will have an integrated box for externalizing the sound elements so that the book will gain in robustness, sound quality, and portability. Another interesting track would be to imagine software, which could enrich already existing books with multimodal aspects, allowing parents and teachers to enhance the books they want in the way they want.
In parallel with the multimodal dimension, it is important to give continuity to studies that focus on the relevance of the texture chosen in books to maximize the link between the material touched by the child on the pages and the real properties of the object (Mascle et al., 2022). This point is important to avoid pitfalls such as ‘textured coloring’ (Valente & Gentaz, 2019) that are often present in tactile illustrations created by sighted people (for example, one will choose a texture that is green to represent the leaves of a tree but which is very far from the real tactile sensation of actual leaves). This issue of texture has been highlighted in our previous writings (Valente & Gentaz, 2019). We also seek to answer questions about texture in two new research projects concerning parent-blind child co-adaptation of books (Tactilivre project) and in the field of emotion (Emoti-sens project).
In conclusion, identifying how well children with visual impairments comprehend new illustrations is essential to creating appropriate content for them. Indeed, not only does this method allow for the development of content that better meets the needs of these children, but it also allows for a process that includes end users of different visual status in the co-design of illustrations.
Supplemental Material
sj-docx-2-jvi-10.1177_02646196231172071 – Supplemental material for Comprehension of a multimodal book by children with visual impairments
Supplemental material, sj-docx-2-jvi-10.1177_02646196231172071 for Comprehension of a multimodal book by children with visual impairments by Dannyelle Valente, Lola Chennaz, Dominique Archambault, Solène Négrerie, Sophie Blain, Anna Rita Galiano and Edouard Gentaz in British Journal of Visual Impairment
Research Data
sj-xlsx-1-jvi-10.1177_02646196231172071 – Supplemental material for Comprehension of a multimodal book by children with visual impairments
sj-xlsx-1-jvi-10.1177_02646196231172071 for Comprehension of a multimodal book by children with visual impairments by Dannyelle Valente, Lola Chennaz, Dominique Archambault, Solène Négrerie, Sophie Blain, Anna Rita Galiano and Edouard Gentaz in British Journal of Visual Impairment
Footnotes
Acknowledgements
We would like to thank the children who took part in the study and professionals who helped us to meet the children. We would also like to thank Psychology Master’s students Maureen Cosnard, Thomas Trivier et Camille Richard for their participation in the collection of data during their research internship in the DIPHE Laboratory (University of Lumière Lyon 2) and research trainers Manon Andrieu, Olivia Gentilhomme and Laurène Vitte-Blimer for their contribution to the coding of videos.
Author contributions
The first two authors performed the conceptualization, methodology, data collection, analysis, and writing of the study. The first, third, fourth, and fifth authors participated in the design of the prototype. The first, sixth, and seventh authors supervised the study. All authors participated in the revision of the final paper.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is part of the TIBonTAB project carried out by Les Doigts Qui Rêvent publishing house and was funded by the Ministère de la Culture et de la Communication in France under the SNI framework ‘Innovative digital service’ call for projects and by the Région Bourgogne-Franche-Comté in France under the fund for the development of innovative digital uses.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
