Image Clustering: An Unsupervised Approach to Categorize Visual Data in Social Science Research

Self-Supervised learning

Although standard deep learning models outperform traditional methods in their ability to extract a meaningful representation of raw images, they cannot be directly applied to unsupervised image analysis because researchers do not have labels for input images. Self-supervised learning of image features has been recently proposed in the deep learning community to address this limitation (Caron et al. 2018, 2019; Gidaris, Singh, and Komodakis 2018; Yang, Parikh, and Batra 2016) ⁶ It has also been used to study social scientific problems already (Valensise et al. 2021). The intuition behind self-supervised learning is that we can perform data augmentation and create pseudo-categories (Gidaris, Singh, and Komodakis 2018). For instance, we can rotate each image in a dataset by 10°, 20°,…, 340°, 350°, effectively expanding the size of the dataset by 35. Then a standard supervised deep learning model can be trained that takes the augmented dataset as input and classifies different rotations of the same image into the same group. After this initial step, the last layer can be extracted as the initial vector of images.

We then apply a predetermined clustering algorithm such as k-means on the initial vectors to group them into K clusters. The new cluster assignment is then used as the new pseudo-category, with which we can refit the supervised deep learning model. We iterate between applying a clustering algorithm on the last hidden layer to generate new cluster labels as pseudo-category, and using pseudo-category to train supervised deep learning models. This iterative process will continue until some convergence threshold is reached. Upon convergence, we obtain both the low-dimensional vector representation of raw images and its cluster assignment for an unlabelled image dataset. The process is called self-supervised because the entire dataset is used as the training dataset. In contrast, standard supervised learning requires an independent training dataset that has expert-labelled images.

Most studies on self-supervised learning of image representations follow the above iterative process that alternates between training supervised learning algorithms on pseudo-category and then clustering the last layer of the learned models. Their differences lie in the choice of deep learning architecture and clustering algorithms. In practice, we choose the method called “DeepCluster,” developed by Caron et al. (2018), because it relies on two widely used deep learning architectures (AlexNet and VGG) and the simplest k-means algorithm for clustering. ⁷

Transfer learning

Another deep-learning-based method of mapping an image to an intermediate representation is through transfer learning. Transfer learning “borrows information” from existing deep learning models trained on external datasets and repurposes that model to the datasets at hand (Pan and Yang 2013). The external dataset and the dataset at hand typically are not sampled from the same population. The existing deep learning models are often called pretrained models; popular choices of pretrained models include the AlexNet, VGG, and ResNet models trained on the ImageNet dataset.

Transfer learning has been widely used in supervised tasks where researchers have labelled data to predict. Previous research has proposed two types of transfer learning (see Sarkar, Bali, and Ghosh (2018, Chap. 4), TensorFlow (2021) for more details). In one approach called “fine-tuning,” researchers keep most layers intact, while allowing the last few layers (i.e., weights) to adapt to the new data. In another approach called “feature extraction with pretrained models,” researchers do not need to train their algorithm directly. Instead, researchers feed images into a pretrained deep learning model, without changing the model's already learned functional forms and weights, and extract features from one of the last few layers. Researchers can use these extracted features in a machine learning model to predict certain labels (Ha et al. 2020).

While these two methods of transfer learning have often been adopted in supervised tasks, we suggest that the second method, “feature extraction with pretrained models,” can also be applied to unsupervised image clustering. We perform “feature extraction” by taking the results from one of the final hidden layers as the intermediate representations. Therefore, each image can be transformed into a vector that can be fed into a clustering algorithm. Using the previous notations, imagine someone has fit the deep learning model Y = h_d(h_d−₁(···(h₁(XW₁))W_d−₁) using X which we refer to as “pretrained dataset.” We have a new dataset X_new and we used the estimated W₁,···W_d−1 to calculate the last hidden layer, and obtain the extracted low-dimensional representation Z_new = h_d−₁(···(h₁(X_newW₁)W_d−₁) as the extract feature vector for the new data.

Why can we repurpose pretrained models that are trained on a dataset other than scholar's own to obtain an intermediate representation? As we explained earlier, deep learning models map images into low-dimensional vectors in a hierarchical fashion (LeCun, Bengio, and Hinton 2015). The lower layers in a pretrained model have already “memorized” how to extract basic features from images. Moreover, popular pretrained models, such as those with good performance in the ImageNet Challenge, are trained on gigantic datasets with at least millions of images. The sheer data size allows these models to learn how to extract basic features very well.

Where do social scientists find pretrained models? Computer scientists will often release their trained models alongside their publications. Alternatively, popular software for deep learning research (e.g., TensorFlow (Abadi et al. 2015), Keras (Chollet et al. 2015), and PyTorch (Paszke et al. 2019)) contain off-the-shelf, pretrained models based on popular architectures (AlexNet, VGG, and ResNet) and on the ImageNet dataset. Social scientists have also been using transfer learning models in their research, such as Zhang and Pan (2019) (VGG) and Sobolev et al. (2020) (ResNet).

When applying transfer learning, researchers can choose from a myriad of available pretrained models. We highlight three aspects scholars should consider when choosing their pretrained models.

Supervised vs. Self-supervised Pretrained Models: Researchers first must choose between pretrained models based on supervised or self-supervised tasks. Self-supervised pretrained models learn to distinguish images based on their innate differences (e.g., different rotations) (Caron et al. 2018, 2019; Gidaris, Singh, and Komodakis 2018). On the other hand, supervised models learn a representation of images that are optimal for predicting labels in the training dataset. Even for the same set of images, models trained for predicting one set of categories may not be optimal for predicting another set of categories. For instance, the same images in the ImageNet dataset can be grouped according to whether they contain the same objects (e.g., cat or tree) or whether their scene differs (e.g, outdoor or indoor scenes). Even using the same ImageNet Dataset, a CNN model trained to detect objects is unlikely to produce image representations suited for detecting scene differences, and vice versa (Krizhevsky, Sutskever, and Hinton 2012; Simonyan and Zisserman 2015). Therefore, transfer learning using selfsupervised pretrained models should provide more general representations. ⁸ However, we note that self-supervised learning is still relatively new, while standard supervised deep learning has a long and mature history. Therefore, most off-the-shelf pretrained models are the results of some supervised tasks.

Training Dataset in Pretrained models: If researchers’ dataset differs significantly from the dataset on which pretrained models are trained, pretrained models will likely produce less meaningful presentations (Azizpour et al. 2015). For instance, many popular pretrained models were trained on the ImageNet dataset. If one wants to cluster a dataset full of human faces, a pretrained model (either supervised or self-supervised) trained on ImageNet is unlikely to provide meaningful representations because ImageNet almost does not contain human classes (Zhang and Pan 2019). In this case, it is better to find a pretrained model based on human faces, such as the VGGFace Dataset that contains 2.6 million images and over 2.6 thousand people (Parkhi, Vedaldi, and Zisserman 2015).

Architecture of Pretrained Models: Finally, researchers need to consider what architecture the pretrained model used. A more complex network is usually more powerful in learning image representations. For instance, AlexNet has 8 layers, VGG has 16 or 19 layers, and ResNet has 50, 101, or 152 layers. The more complex model generally performs better on object classification tasks than simple models. In the main results, we used VGG mainly because the self-supervised transfer learning model we used (DeepCluster) is build upon the VGG architecture (Caron et al. 2018). We can compare the self-supervised and supervised transfer learning model trained on the same VGG architecture. We showed the result using ResNet (with supervised transfer learning) in Appendix F.

Data and methods

Our first dataset comes from CASM-China (Zhang and Pan 2019), as described in the Introduction.

From CASM-China, we selected 2,742 offline protest events in the first half of 2016 that have social media images associated with them. ¹¹ The 2,742 protests contained 14,127 images. We rescaled each image to be of size 224 ∗ 224 because this is the size of the input used in standard VGG pretrained models.

Method 1 used the bag-of-visual-words model to extract features from images. We expected its performance to be worse than the deep learning-based methods, as described below. Method 2 trained the self-supervised representation learning algorithm (Caron et al. 2018) on the same 14,127 protest Chinese protest images. ¹² Method 3 used transfer learning (feature extraction in particular). We used two pretrained models. The first used a standard supervised pretrained model using VGG and ImageNet, and the second used self-supervised pretrained model, “DeepCluster” (Caron et al. 2018). ¹³

All three methods used the k-means clustering algorithm. K-means requires a critical parameter, the number of clusters K. Below, we visualize the cluster assignment in collated pictures using K = 6 based on our proposed visualization strategy and then present the human validation results with within-cluster consistency, which justifies our choice of K.

Findings from collated images

Figures 4–7 are the collated images. Each row represents a cluster identified by the corresponding method. To highlight the main theme of each category, the images in each category were chosen to be a representative sample by selecting those that are closest to the centroid of the cluster.

OPEN IN VIEWER

Figure 7.

Method 3, transfer learning using self-supervised pretrained models.

Method 1 used bag-of-visual-words model (Figure 4). It produced three clusters of images with textual elements (Clusters 3, 4, and 5). Cluster 6 picks up trees and grasses, which is not an theoretically interesting category for understanding contentious politics. Only Cluster 1 and 2 recognize crowd gathering, but still there are incorrectly classified pictures (e.g., 1.F and 1.G, which display police violence; 2.B, and 2.J, which display hand-written banners; 2.I, which is a Wi-Fi signal irrelevant to protests at all). Considering the fact that we already used images that are closest to the cluster centroids, such that these images should be the most representative, it suggests that Method 1 does not perform very well.

OPEN IN VIEWER

Figure 5.

Method 2, self-supervised learning of image representations (Caron et al., 2018).

Method 2, which used self-supervised learning (DeepCluster), also did not produce satisfactory clustering solutions (Figure 5). Although it recognized that trees should not be a separate category, it conflated protester's banners with crowd gathering (Cluster 1), and letters with buildings (Cluster 3). Furthermore, protesters with banners were assigned to different clusters (1, 2, 4, and 5).

OPEN IN VIEWER

Figure 6.

Method 3, transfer learning using supervised pretrained models.

Figures 6 and 7 reveal that the transfer learning models (Method 3) both significantly outperformed the two previous methods. Both transfer learning models picked up categories such as gathering of people (Cluster 1 and 5 in Figure 6; Cluster 3 and 5 in Figure 7), hand-written or typed petition letters (Cluster 2 in Figure 6; Cluster 6 in Figure 7), screenshots of text (Cluster 3 in Figure 6; Cluster 4 in Figure 7), and protester with banners.

However, transfer learning based on different pretrained models exhibited differences. Transfer learning using supervised pretrained models treated photos with black backgrounds as a separate category. Transfer learning using self-supervised pretrained models recognized that black borders do not have intrinsic meaning; it did not treat photos with black background as a separate class. Moreover, transfer learning using self-supervised pretrained models recognized that photos about protesters holding banners should be put in the same class (Cluster 4 in Figure 7), while transfer learning using supervised pretrained models did not. Overall, transfer learning based on selfsupervised pretrained models produces more meaningful categories. This is expected since selfsupervised learning aims to learn more general features instead of specific features that map images to particular categories in the ImageNet Dataset.

The best model—transfer learning based on self-supervised pretrained models—produced theoretically relevant clusters. Cluster 1 in Figure 7 concerns protesters gathering at the gate of government offices; some protesters held banners. Cluster 2, on the other hand, is relevant to protesters blocking streets. Cluster 3 are also crowd gathering, with a zoomed-in view. Cluster 4 contains screenshots of text, an approach to avoid censorship on Chinese social media, which heavily relies on detecting texts (King, Pan, and Roberts 2013). Cluster 5 is also crowd gathering, but with police presence. Cluster 6 contains petition letters, printed or hand-written. Researchers can potentially apply OCR techniques to extract text from letters and infer protesters’ goals.

Quantitative validation

Because visual inspection reveals that the traditional bag-of-visual-words model and self-supervised learning does not yield satisfactory performance, we only instructed research assistants to compare the results produced by the two transfer learning models. We present a total of six clustering solutions, which vary by the number of clusters (6, 8, and 10 clusters), and two transfer learning models, based on supervised and self-supervised pretrained models. For each cluster produced by a pretrained model, we randomly selected 20 images and had research assistants give a label to each image. We then calculated the within-cluster consistency defined earlier.

Figure 8 shows the human validation results. We proceed in two steps. First, we require the minimum threshold of within-cluster consistency to be 0.5—namely, at least half of the images need to belong to the majority category in that cluster. After this step, only the solutions produced by self-supervised model (K = 6 or 8) and supervised model (K = 8) that met this minimum threshold were kept and other solutions are discarded. Second, we compared the remaining three clustering solutions with respect to their average within-cluster consistency. Among the three remaining clusters, the self-supervised model with six clusters produced the highest average withincluster consistency score. Its minimum within-cluster consistency is also significantly larger than the other two remaining solutions (self-supervised model with K = 8 and supervised model with K = 8). Therefore, we chose the self-supervised model with K = 6 as the final clustering solution. This choice is the same as our visual inspection in that self-supervised model slightly outperforms the supervised models. In Appendix C we provided statistical tests to show that our final clustering solution indeed has a statistically higher within-cluster consistency compared with the other five solutions.

OPEN IN VIEWER

Figure 8.

Within-cluster consistency for Chinese protest dataset. Average within-cluster consistency (M) is highlighted in red and the exact values of the average within-cluster consistency is shown on the top of each bar. Six clustering solutions are shown, varying by the number of clusters and the pretrained models.

Overall, study 1 shows that traditional feature extraction methods (bag-of-visual-words) were less satisfactory. Self-supervised learning using our own dataset, although seemingly attractive and currently popular in computer science, also did not perform well. One possible reason why self-supervised learning did not perform well is that our dataset is still not large enough (n around 10 K), being two scales smaller than the training data used by the two pretrained models (ImageNet dataset, n ¿ 1 M). Moreover, both the bag-of-visual-words and self-supervised learning models took considerable amounts of time to train. In our practice, it took over 24 h to train the selfsupervised learning model (Method 2) ¹⁴ and over 3 h to train the bag-of-visual-words model (Method 1). On the contrary, the transfer learning model finished within 5 min.

Moreover, transfer learning is also easier to implement with respect to coding. Python code for implementing Method 3—using the standard supervised pretrained model to extract features—is available in Appendix A. The code demonstrates that we are able to extract vectors within 50 lines of code. Due to the space restriction, we cannot put all codes in the appendix; we will make them public available. Here, we will disclose that Methods 1 and 2 needed significantly more time to implement and could not be finished within 50 lines of code. As traditional approaches and selfsupervised models are slow, harder to implement, and do not have good performance, we do not recommend them to social scientists who plan to apply image clustering methods at this time.

We also compared transfer learning based on supervised and self-supervised pretrained models and found that self-supervised pretrained models were slightly better than supervised pretrained models in terms of average within-cluster consistency. However, self-supervised learning is still a burgeoning area, and there are fewer off-the-shelf pretrained models. On the other hand, standard software for deep learning, such as Keras, TensorFlow, and PyTorch already have built-in support for using supervised transfer learning models. Therefore, if scholars cannot find pretrained selfsupervised models, we recommend using pretrained supervised models.

How image clustering is useful in empirical research

While the present research is a methodological piece, it should be useful to demonstrate how image clustering can be used in empirical research (Kang et al. 2020b). We thus offer some examples on how image clustering can inform understanding theoretically interesting questions. First, image clustering might be useful to provide quantitative evidence for existing theories. For instance, one established argument in the qualitative studies of protests in China is that protesters often used disruptive tactics to gain leverage over governments (Cai 2010; Chen 2009, 2012). In particular, blocking government doors and blocking streets are two widely used disruptive tactics. Blocking government doors often occurs regardless of whether the targets of the protests are the government itself or other nonstate entities, such as private real-estate developers. Protesters use this tactic to force the government to react (if the government is the target) or intervene and punish the private companies (if the state is not target) (Chen and Cai 2019). The second tactic—blocking streets—is powerful because doing so will attraction attention of the bystander public, who may show sympathy toward protesters and add pressure on the government. In comparison, violent tactics in China are often found to be highly ineffective because they bear much higher risk for government repression (Cai 2010); protesters will also consciously limit their actions, trying not to incur physical confrontation with the police (Fu 2017).

Yet, many of the theoretical studies are drawn from qualitative observations but have not gone through scrutiny of quantitative evidence. We plot the number of images in each category in Figure 9 based on our preferred solution (Method 3, transfer learning using self-supervised pretrained models). Confirming the theoretical predictions, protesters gathering at government offices (Cluster 1) and blocking streets (Cluster 2) are the two most prevalent image categories. Crowd gathering with police presence (Cluster 5), which often involves violent confrontation, are considerably less popular than the two categories suggesting tactics. The comparison between images using disruptive tactics versus violent tactics provides quantitative evidence supporting the observations from qualitative studies.

OPEN IN VIEWER

Figure 9.

Number of images in each class.

Second, image clustering results may inform new empirical research. As Cluster 4 and 6 have shown, protesters in China often use photos or screenshots of their hand-written/printed petition letters to avoid platform censorship. These texts embedded in the images contain valuable information, especially the claims that were believed to be removed by the censorship machine, such as criticisms of the government. Traditionally, when researchers study the use of images during protest mobilizations, they mostly examine visual elements such as crowd gathering, fire, and police (Casas and Williams 2019; Oleinik 2015; Steinert-Threlkeld, Chan, and Joo 2021). These text information embedded in information is a neglected aspect (Goebel and Steinhardt 2019). Comparing the text from images to those that they write in text might reflect different strategies the protesters used to mobilize and attract attention from bystanders.

As our discussions have shown, image clustering can be used to measure theoretically interesting concepts (e.g., protest tactics) and to inform new research ideas that have been neglected in previous research (text embedded in images). These two directions will also be our future empirical research plans. These are certainly not the only ways researchers can use image clustering; depending on the dataset and the problem, researchers may discover patterns they have not expected before running image clustering.

Selection of datasets

We selected two datasets that can illustrate how the choice of pretrained models impacts the clustering solutions on these two datasets. The first dataset contains images about climate change on Instagram. Previous research has shown that visual messages about climate change often contain a broad range of content themes, including protests, scientific activities, natural landscapes, animals, data visualization, and satellite images (O’Neill and Smith 2014). In February 2020, we curated a list of eleven popular Instagram accounts that extensively publish content related to climate change by using the search keywords “climate change” and “global warming” on Instagram. We then retrieved these accounts’ posts using a Python library instaloader. ¹⁵ We kept images published before January 31, 2020 for analysis (N = 11,873).

The second dataset is about profile images from disinformation accounts associated with the Russian Internet Research Agency (IRA) on Twitter. In October 2018, Twitter released a dataset that contained accounts it identified as affiliated with IRA. The archive contained these accounts’ tweets and associated media. We focused on the profile images (N = 3,709) from these accounts. This dataset is very different from climate change visuals: the majority of the images are human faces, whereas a small subset of the images are icons or logos.

Selection of pretrained models

We selected three pretrained models that used the same architecture (i.e., VGG16) but had been trained on three distinct datasets: ImageNet, Places365, and VGGFace. ImageNet features a diversity of content categories, such as animals (e.g., magpie, jellyfish), natural scenes (e.g., coral reef, lakeside), places (e.g., cinema, restaurant), foods (e.g., pretzels, cheeseburger), vehicles (e.g., aircraft carrier, speedboat), and everyday objects (e.g., joystick, balloon, envelope, volleyball). As visual messages of climate change contain a variety of content, features learned from a diverse dataset like ImageNet should be able to find some categories in climate change visuals. Yet, the ImageNet dataset has almost no categories specifically related to humans, so it may not reveal meaningful categories in the IRA profile images that predominately feature human faces.

In comparison, the Places365 dataset contains about 1.8 million images of 365 scene categories, which include a broad range of indoor (e.g., conference center, classroom, legislative chamber, kitchen) and outdoor settings (e.g., street, skyscraper, forest, ocean) (Zhou et al. 2017). As pictures of climate change communication often feature natural landscapes, urban environments, industry, protests, and conferences, this model should help us discover meaningful categories particularly related to scenes and setting. Still, the Places365 dataset does not contain categories specifically about humans, so we expect that this pretrained model is not very suitable for the IRA profile images.

The VGGFace dataset contains about 2.6 million facial images of over 2.6 thousand people (Parkhi, Vedaldi, and Zisserman 2015). The features extracted should predominantly pertain to human faces. This pretrained model should be less capable of revealing visual categories relevant to the dataset about climate change than the same model trained on ImageNet and Places365. Nevertheless, as the majority of images in the IRA dataset contained human faces, this pretrained model might have better performance for this particular dataset.

To facilitate the comparison among the three pretrained datasets, we retrieved three pretrained models using the same model architecture, VGG16, but trained on different datasets: ImageNet (accessible at https://keras.io/api/applications/), Places365 (https://github.com/ GKalliatakis/Keras-VGG16-places365), and VGGFace (https://github.com/rcmalli/ keras-vggface). We extracted the first dense layer after the convolutional blocks in all three models as our intermediate representations of images. ¹⁶ These vectors have 4,096 dimensions. For each model, we performed a principal component analysis and kept the first 200 dimensions for clustering. ¹⁷ We then ran k-means clustering with the number of clusters ranging from 5 to 20.

Findings from collated images

We first present the collated images of clustering results to demonstrate how the same deep learning architecture on different pretrained datasets affects the categories machine identified.

Climate Change Visuals: Figures 10–12 show results with K = 8 using ImageNet, Places365, and VGGFace, respectively. We randomly selected 20 images from each cluster for inspection.

OPEN IN VIEWER

Figure 10.

Eight-cluster solution in the climate change dataset based on features extracted from a pretrained model trained on imageNet dataset.

OPEN IN VIEWER

Figure 11.

Eight-cluster solution in the climate change dataset based on features extracted from a pretrained model trained on the Places365 dataset.

OPEN IN VIEWER

Figure 12.

Eight-cluster solution in the climate change dataset based on features extracted from a pretrained model trained on the VGGFace dataset.

For climate change visuals, the solution using ImageNet revealed a variety of visual categories (Figure 10). Three clusters (3, 5, and 7) were related to textual messages and graphics, with some variations in design aesthetics. Three clusters (1, 2, and 4) were related to human activities. Cluster 1 captured people, usually several individuals, in outdoor environments, mostly natural scenes or rural areas. These people were also viewed from a distance. Cluster 2, in comparison, featured mostly crowds, which were in mostly urban environments such as conferences and streets (protesters). Cluster 4 featured close-ups or medium shots of people, and these images also had a limited number of individuals. Two other clusters also emerged. Cluster 8 featured landscape pictures, usually without people. Cluster 6, however, did not seem to show a clear cohesive theme. It contained a combination of still objects, animals, and satellite images, potentially because these pictures shared similar visual characteristics. It is also possible that some types of images (e.g., animals, satellite images) only comprised a small percentage of the dataset. If we chose a large number of clusters, this mixed cluster might break down into smaller categories.

The solution using Places365 showed a pattern different from the solution using ImageNet. Four clusters (2, 4, 5, and 7) were related to textual messages and graphics, with variations in design aesthetics. Interestingly, Cluster 4 contained images featuring dense text on a white background, which had not been found in the ImageNet solution. The crucial difference came in how this solution clustered pictures of people. While the ImageNet solution distinguished between pictures of crowds and pictures of several individuals, this Places365 solution mostly grouped pictures of people based on settings. Cluster 3 featured people in indoor settings, such as conference rooms and lecture halls. Cluster 6 mostly featured people in outdoor settings, such as streets. Cluster 1 featured pictures of landscapes. Importantly, pictures of satellite images were also mostly categorized into this cluster. This cluster had both pictures featuring people and pictures without people. This could be because the scene categories in Places365 only concerned the different types of scene, regardless of the presence of people. Finally, Cluster 8 exclusively featured wind farms, a frequently used symbol in climate change messages. This is likely because the Places365 dataset has a scene category—wind farms—and this niche category has distinct visual characteristics, so this type of image was rediscovered in our dataset regarding climate change. This particular category was absent in ImageNet and VGGFace solutions.

In comparison, the VGGFace solution performed suboptimally. Not surprisingly, this solution discovered two clusters that were entirely related to human faces (Cluster 3 and 7). However, its ability to find other cohesive and meaningful categories was limited. Two clusters (Cluster 1 and 4) predominantly featured images of textual messages and graphics, although these two clusters also mixed in a few other types of images. Cluster 6 featured mostly outdoor scenes, mixing pictures of protests and landscapes. There seemed to be no cohesive themes from the other three clusters (Clusters 2, 5, and 8).

IRA Profile Images: Similarly, we observed that solutions from the three pretrained models also performed differently for the IRA profile images. Figures 13–15 show results with K = 8 using ImageNet, Places365, and VGGFace, respectively. We randomly selected 20 images from each cluster for inspection.

OPEN IN VIEWER

Figure 13.

Eight-cluster solution in the IRA dataset based on features extracted from a pretrained model trained on imageNet dataset.

OPEN IN VIEWER

Figure 14.

Eight-cluster solution in the IRA dataset based on features extracted from a pretrained model trained on the Places365 dataset.

OPEN IN VIEWER

Figure 15.

Eight-cluster solution in the IRA dataset based on features extracted from a pretrained model trained on the VGGFace dataset.

For ImageNet, there was a clear division between human (Cluster 1, 2, 3, 7, and 8) and nonhuman clusters (Cluster 4, 5, and 6). Among the clusters featuring humans, the algorithm was able to formulate a cluster featuring solely men (Cluster 1), mostly in suits, a cluster featuring solely women, mostly with long hair (Cluster 7), a cluster featuring people with sunglasses or dark glasses (Cluster 8), a cluster featuring people in environments, usually with smaller faces and showing bodies (Cluster 3). There seemed no clear patterns regarding gender, ethnicity, or age differences in Cluster 2. Cluster 4, 5, and 6 mostly featured nonhuman subjects such as icons, logos, or illustrations.

For Places365, the clustering solution was less ideal. First, some clusters even did not have over 20 images, for example, Cluster 4 only had one image, indicating that this pretrained model might result in unbalanced categories. Again, there was a distinction between human (Cluster 1, 3, 4, 5, and 7) and nonhuman clusters (Cluster 2, 6, and 8). Among the clusters featuring humans, the algorithm was able to formulate a cluster featuring mostly women (Cluster 1) and a cluster showing people in outdoor environments (Cluster 3). There seemed no clear patterns regarding other clusters (5 and 7). Cluster 2, 6, and 8 mostly featured nonhuman subjects such as icons, logos, or illustrations.

For VGGFace, the clustering solution was able to find more nuances among categories. Again, there was a distinction between human (Cluster 1, 3–8) and nonhuman clusters (Cluster 2). Among the clusters featuring humans, the algorithm was able to formulate a cluster featuring mostly white men (Cluster 1), two clusters showing mostly white women (Cluster 5 and 6), one cluster featuring mostly African Americans (Cluster 7), and a cluster showing people in environments (Cluster 8). Therefore, compared to the other two pretrained models using ImageNet and Places365, the solution based on VGGFace was more able to pick up visual patterns related to gender and race in human faces. Only one cluster was about nonhuman objects (Cluster 2).

Quantitative validation

Next, we calculated within-cluster consistency for the two datasets, three clustering algorithms (using different pretrained datasets), and K ranging from 6, 8, and 10. Figure 16 shows the human validation results for the climate change dataset. Regardless of K, VGG model trained on Places365 dataset always gives the highest within-cluster consistency, while the same algorithms trained on VGGFace always give the lowest within-cluster consistency.

OPEN IN VIEWER

Figure 16.

Within-cluster consistency for climate change dataset. Average within-cluster consistency (M) is highlighted in red and the exact values of the average within-cluster consistency is shown on the top of each bar. Nine clustering solutions are shown, varying by the number of clusters and pretrained models.

Figure 17 shows the human validation results for the IRA dataset. Confirming our expectation, VGG model trained on the Places365 dataset yields the lowest within-cluster consistency, regardless of the choice of K. VGG model trained on the ImageNet dataset gives a higher within-cluster consistency than the VGG model on Places365 dataset. VGG model trained on the VGGFace dataset always gives the best within-cluster consistency across different choices of K. It suggests that if one should choose a pretrained model that is trained on a dataset similar to the final dataset used for clustering. ¹⁸

OPEN IN VIEWER

Figure 17.

Within-cluster consistency for IRA dataset. Average within-cluster consistency (M) is highlighted in red and the exact values of the average within-cluster consistency is shown on the top of each bar. Nine clustering solutions are shown, varying by the number of clusters and pretrained models.

To conclude, Study 2 reveals the choice of pretrained datasets can critically shape the categories found in the clustering solution. Among the three solutions for climate change images, the solution using VGGFace had the worst performance. It was only able to discover categories related to human faces and missed other visual themes. Both the solution using ImageNet and the solution using Places365 were able to find more meaningful categories, yet they differed on the specific emphasis and categories in the solutions. The solution using Places365 also yields the highest within-cluster consistency based on human coding.

Among the three solutions for IRA profile images, the solution using VGGFace found the most interesting patterns and resulted in the highest within-cluster consistency. For example, this solution was able to formulate categories featuring certain racial or gender groups. Nevertheless, we included this dataset to demonstrate how the choice of pretrained models, and in particular, VGGFace, could influence the categories revealed in image clustering. However, there have already been many well-developed algorithms for facial detection and facial analysis. Therefore, instead of using an unsupervised image clustering approach, it might be more appropriate to apply a supervised approach to directly analyze facial attributes that might be theoretically meaningful, such as gender, age, and race.

Because our goal in this section is to see how the choice of pretrained dataset matters, we leave the detailed steps of how we choose the best model (which K) in Appendix D. We also employed two coders and calculated the intercoder reliability of these clustering solutions. Appendix E show the intercoder reliability results. For the ones that have high within-cluster consistency (e.g., VGG model trained on VGGFace dataset), the intercoder reliability (Cohen's Kappa) is also high, typically higher than 0.9. Moreover, to add more confidence to our previous observations, for the IRA dataset, VGG models trained on VGGFace dataset generally gave higher intercoder reliability compared with other pretrained dataset choices; VGG models trained on Places365 dataset also have higher intercoder reliability on the climate change dataset.