Attention based sequence-to-sequence framework for auto image caption generation

Abstract

Caption generation using an encoder-decoder approach has recently been extensively studied and implemented in various domains, including image captioning and code captioning. In this research article, we propose one particular approach for completing a capture generation task using an “attention”-based sequence-to-sequence framework that, when combined with a conventional encoder-decoder model, generates captions in an attention-based manner. ResNet-152 is a Convolutional Neural Network (CNN) based encoder that generates a comprehensive representation of an input image while embedding that into a fixed size length vector. To predict the next sentence, the decoder uses LSTM, a Recurrent Neural Network (RNN), and an attention-based mechanism to concentrate attention on certain sections of an image selectively. Define a set of epochs to 69, which should be enough for training the model to generate informative descriptions, and the validation loss has reached its minimum limit and no longer decreases. We present the datasets as well as the evaluation metrics, as well as quantitative and qualitative analysis. Experiments on the MSCOCO and Flickr8k benchmark datasets illustrate the model’s efficacy in comparison to the baseline techniques. On MSCOCO, evaluation scores included BLEU-1 0.81, BLEU-2 0.61, BLEU-3 0.47, and 0.33 METEOR. For Flickr8k BLEU-1 0.68, BLEU-2 0.49, BLEU-3 0.41, METEOR 0.23, and 0.86 on SPICE. The proposed approach is comparable with several state-of-the-art methods in terms of standard evaluation metric, demonstrating that it can produce more accurate and richer captions.

Keywords

Image captioning CNN LSTM sequence-to-sequence neural network

1 Introduction

In Artificial Intelligence, image captioning is a fundamental goal that explains objects, attributes, and relationships in an image in a natural language sense. It has several applications, semantic image search, supporting chatbots with visual insight, or allowing people with visual impairments to see the world around them [1 –7]. The problems related to image caption generation have emerged as a greater spectrum of research in the field of Natural Language Processing (NLP) and computer vision (CV) [8, 9]. To understand the salient and semantic content of an image [10, 11], this process requires a machine, the object type, and the relationship between those objects, etc. Regardless of current shortcomings, for human life, the method is significant to direct visually disabled people to understand the visualized environment by organ or to create an actual sense of intellectual robot that exposes language, sensory and visual networks [12, 13] For content-based retrieval or in media platforms communications, dynamically generated descriptions may also be used. Classifying the leading strategies into two different main streams is important. One stream takes an end-to-end, encoder-decoder architecture introduced by machine translation [13]. They use a CNN to achieve the best image attributes, which they then feed into RNNs to generate target sentences.

We illustrate in this comprehensive study that the task of image captioning is both an active and mature deep learning topic. The goal is to define the content of an image by mapping it to a sequence of words. Introduction of sampling temperature to the Decoder RNN for image captioning. We’re using self-temperature sampling to scale output. This allows the LSTM to be more selective but cautious with its samples. - Beam search caption sampling [15], which involves keeping track of the most likely k hypothesis at each decoding point while generating the hypothesis Beam search caption sampling is added, which keeps track of the k most likely hypothesis at each decoding phase while creating the hypothesis. By adding an attention mechanism to our framework, can produce another significant improvement. The research study’s key contributions can be summarized as follows:

To use the attention-based model for feature extraction, a completely implemented image detection model that offers higher performance on the COCO and Flickr8k collection, an Attention-Based Sequence-to-Sequence System for Image Caption Generation is implemented.

Providing an encoder-decoder network model depends on the fact that the encoder correctly summarizes the important features of the input image. Attention splits the input into smaller sections and allows redundant parts to be pruned by the decoder network.

Adoption of visual attention and insightful ResNet-152 based encoder and the robust LSTM based language model helps to increase the performance in most of the evaluation metrics including BLEU 1 - 3, METEOR, ROUGE-L, and SPICE, such as the MS COCO and Flickr8k datasets [16].

As follows, the remainder of the paper is structured. Section 2 Summary related work in image captioning and attention approach. The proposed strategy is defined in Section 3. The findings of our experiment on the benchmark dataset are seen in Section 4 and contrasted with state-of-the-art approaches. Finally, a discussion followed by a conclusion is given in section 5.

2 Related work

This section demonstrates an overview of exciting research work on the generation of image captions and attention. Template-based approaches, retrieval-based methods, and neural network-based methods are the three types of contemporary work on image captioning. Using object detection and attribute classification approaches, the template-based methods [23], [50] first recognize essential visual conceptions of objects, attributes, and actions from images. These strategies divide a sentence into several sections based on specified language templates, subject, verb, and object. Lastly, using statistical approaches, these methods connect the discovered visual concepts with the parts of a linguistic template [17, 19]. Since these systems rely heavily on established language templates, they can only create syntactically accurate phrases at the expense of natural language flexibility. The retrieval-based approaches [19, 20] often assess the similarity of an input image to external phrases or the similarity of an input image to other visually comparable images. These approaches can select the least semantically comparable sentences from an external sentence source or candidate phrases derived from visually similar images based on similarity. As a consequence, because all of the phrases were created by humans, these approaches may generate human-level sentences. These algorithms, however, are difficult to apply to diverse datasets and cannot produce novelty in sentences for given images.

DNNs are used in the neural network-based algorithms [1, 54] to exploit conditional probability distributions given visual material and create words. Following the success of encoder-decoder architectures in neural machine translation [18, 21], these techniques treat image captioning as a translation task that involves generating the caption for a given image [17]. An instance, with a CNN encoding the image to a feature vector and LSTM decoding that to caption [1], an end-to-end system is proposed. An early fusion of encoded visual cues and combined textual features is used to perform multimodal representation. This data was used to train a baseline caption generator based on Long Short-Term Memory (LSTM) [54]. A spatial and channel-wise attention framework is suggested in [18]. An adaptive attention technique is deployed to determine when to decide when and how to activate visual attention. Most significantly, to provide clearer descriptions, more complicated information such as objects, attributes and relationships is introduced [22 , 50].

A new attention-based encoder-decoder framework has recently been proposed to detect a collection of salient image regions through the bottom-up attention approach and then address the salient regions with the sentence generation top-down attention mechanism [21]. Other mechanisms of attention include multi-layer attention, as well as spatial and channel-wise attention. In multi-layer feature maps,Spatial and channel-wise attention (SCA-CNN) dynamically adjusts the sentence generating context [18], The attention layers visualization depicts the Stacked attention Networks’s (SAN) layer-by-layer progress in locating relevant visual clues that lead to the explanation [25], multi-level attention CNN produce semantic concepts from high-level semantics, and a bidirectional recurrent neural network encodes region-based middle-level CNN outputs into spatially-embedded representation [26], multi-head and attention proposed the Transformer, an attention-based in sequence transduction model that replaces the recurrent layers most typically used in encoder-decoder designs with multi-headed self-attention [27]. Furthermore, recent advances in image processing and object recognition [53] show highly efficient and promising results for surface texture analysis [51] image dehazing [22] using deep learning.

Many other research studies expand the concept of using self-attention in the field of computer vision to certain tasks inspiring us to apply self-attention to image captioning generation to model associations in an image with objects [28 , 48]. Throughout the decades of studies, the image captioning problem has remained an active and mature deep learning topic. The ultimate goal is to transfer the content of an image into a sequence of words and present a method for solving a capture generation task by combining an “attention”-based sequence-to-sequence framework with a conventional encoder-decoder model to generate captions in an attention-based manner. ResNet-152 is a convolutional neural network-based encoder that produces a full representation of an input image while embedding it in a fixed-length vector. The decoder employs LSTM, a recurrent neural network, and an attention-based mechanism to selectively focus attention on key regions of an image to predict the next sentence. The concept of artificial neural networks based on Euler polynomials can be expanded in the future for accuracy of caption generation [49].

3 Proposed framework

In this research paper, construct a structure that can produce a more specific English sentence that defines an object, action, or event in an RGB input image. Using a recurrent neural network encoder, this model learns an interactive representation of the basic sentence and using another recurrent neural network in the target language to decode the representation. We adhere to techniques that allow an “attention”-based sequence-to-sequence model to carry out a capture generation task. The general architecture of the proposed attention-based sequence-to-sequence model is illustrates in Fig. 1. To top of the figure, there is the encoder block, which takes an image as input and extracts a feature map. To the bottom, the decoder has to produce a captioning from the input image, that use information from input, its hidden state, as well as the context information obtained from attention block. The attention mechanism is located between the encoder and the decoder. It is responsible for determining which parts of the image (which features) to focus at each time step while. The attention mechanism takes the image feature and the decoder’s hidden state as input and generates a context vector consisting of the features weighted according to some learned weights that represent the relevance given to each feature at a given moment.

Fig. 1

Overview of the proposed model architecture for attention-based sequence-to-sequence image captioning generation. Encoder, decoder and attention mechanism.

3.1 Methodologies adopted for the caption generation

Our suggested image captioning attention framework, which consists of CNN, Attention Module, and RNN. Each of the activities described is elaborated on. In terms of Z input/output pair can be written as $B, C = {B^{i}, C^{i}}_{i = 1}^{Z},$ (1) where, we consider supervised processes where the training set is provided. Using Bⁱ is the input, and either constant (like for an image) or dynamic (like for a Sequence), whereas the Cⁱ the output of the target has become a sequence in Equation (2), a variable number of tokens belonging to a specified set dictionary. $C^{i} = q_{1}^{i}, q_{2}^{i}, \dots, q_{T_{i}}^{i} .$ (2)

3.1.1 Deep CNN encoder

As a feature extractor consisting of convolution building blocks embedded with shortcut connections, the latest implementation (Fig. 1) assumes Resnet-152. In contrast to the framework that was used to analyse an image to be represented by a single fc feature vector, to serve image regions, and obtain conv-features. It is because fc-type features don’t maintain spatial data. Assuming parts of the feature vector cannot be associated with regions in an image. The source image is fed into the CNN encoder, and features with a size of 7 × 7 ×1048 are extracted. The attention block receives these features. to generate the context vector. This results in 49 regional local vectors with 2048 lengths in Fig. 1 these regions are denoted by K₁, K₂, …, K₄₉ .

3.1.2 Attention block

The mechanism of attention [28, 30], which is formed from human experience, has been mainly utilized and has resulted in substantial improvements for different sequence learning tasks. We emphasize the most significant pixels in the image using the attention mechanism in our proposed framework. The attention network outputs the background vector, which is the weighted sum of the output of the encoder, to concentrate on relevant parts at each decoding stage (features) described in Fig. 2.

Fig. 2

Attention block representation for sequence-to-sequence process based captioning generation.

Initially, with the scoring function, score each of the outputs (features) of the Encoder transmitted to the attention network, and get probabilities, implementing the scores with the Softmax function. The significance of each feature vector input to the Decoder is represented by these values. Determine the weighted sum, multiplying the features with sufficient probabilities.

Attention scores t, j attention scores are measured by using the feed-forward method and v and f_j are vectors and U and Ware matrices. These are all trainable parameters, as in Equation (3). ${score}_{t, j} = v_{a}^{T} \times tanh (U_{a} \times f_{t - 1} + W_{a} \times f_{j}),$ (3)

where, f_t-1 is a Decoder’s hidden state at time t - 1 and f_j is the Encoder’s output (feature maps).α_ij is the output score of a feedforward neural network described by the function a that attempts to capture the alignment between input at j and output at i. Then, apply the Softmax to measure Atten-weights probabilities as in Equation (4). $α_{tj} = \frac{e^{score (f_{t - 1}, f_{j})}}{\sum_{k = 1}^{T_{x}} e^{score (f_{t - 1}, f_{k})}} .$ (4) After this, extract the context vector, C_t is the context vector, Tx number of “annotations”.where f_j is the encoder hidden state at the j^th encoder time step (the hidden state vectors) i.e., the weighted sum of input in Equation (5). $C_{t} = \sum_{j = 1}^{T_{x}} α_{tj} f_{j} .$ (5)

3.1.3 Decoder’s architecture

The RNN’s task is to decode the process vector and transform it into a sequence of words [17]. Thus, this part of the framework is also a decoder. Let’s formalize the image captioning task before switching to the architecture of a Decoder. The model learns to accurately predict sequence C,computing the log-likelihood P (C ∣ B),given the input function maps B and target captions C with the length T as in Equation (6), $\begin{matrix} log P (C ∣ B) = \sum_{t = 1}^{T} log P (q_{t} ∣ y_{1}^{t - 1}, B), \end{matrix}$ (6) where C is a length T sequence described by tokens q₁, q₂, …, q_T. In the above Equation (6), the latter term is calculated by a recurrent neural network with parameters θ by adding a state vector, ft, which is a function of the former state, f_t-1 And the former target token, q_t-1. i.e. $\begin{matrix} log S (q_{t} ∣ q_{1}^{t - 1}, B; θ) = log P (y q_{t} ∣ f_{t}; θ), \end{matrix}$ (7) where qt is token at time step t, And ∣q_1:t-1 = (q₁, q₂, …, q_t-1) is previous output tokens. If the ft is determined as follows by a recurrent neural network as in Equation (8). $f_{t} = {\begin{matrix} p (X; θ) & if t = 1 \\ p (f_{t - 1}, q_{t - 1}; θ) & otherwise \end{matrix} .$ (8)

A set of parameters θ^* that enhance the log probability of the right sequence is learned by the model. After that use the LSTM cell that outputs hidden state (short-term memory) h and cell state (long-term memory) c to process the sequence. To compute probabilities for all tokens in a dictionary [31], then feed the hidden state h to a fully connected layer, followed by Softmax. Decoder’s architecture is demonstrated in Fig. 3. The loss function applies Softmax to outputs and performs logarithmic operation afterward while practice with Cross-Entropy loss. Thus, the model only learns to produce the next token in a sequence given a hidden state and a previously token.

Fig. 3

Decoder’s architecture that reads the encoded image and generates the textual description output.

Develop embeddings from the captions of the target. Here, Fig. 4 demonstrates the vector size (batch, t, em - bed - dim). Where a sequence length corresponds to t^- using the fully connected layers, hidden states (h, c). Through each phase during the training, perform the scheduled sampling [3]. As a condition of selecting the next input token, and set the sampling probability, either from the target sequence (so-called teacher forcing) or from the output sample. The key concept behind the scheduled sampling is to minimize the distance when the targeted tokens are not available between teacher forcing training and inference level. Convolve the embedding and context vector into the LSTM cell’s single, eventually for a hidden state h, apply dropout regularization and distribute it to a fully connected layer.

Fig. 4

An embedding layer that generates word embedding.

3.1.4 Training

Finally, require to mention here that For the CNN encoder, and initialize it with The ResNet-152 model pre-trained on the Image-Net classification task. For the LSTM, attention block, and word embedding, were randomly initialized. During the training phase, the input image enters into the CNN encoder, and features are extracted with the size of 7 × 7 ×10488. These features are fed into the attention block. To produce the context vector at each step, and the ground truth caption’s tokens are embedded and used during the LSTM training. Use cross-entropy loss function is used to train the model, and extract the features from the input image using the CNN encoder and enter them into the attention block, and the start token along with the context vector is used to predict the next word of the caption, which is used as an input word in the next step.

4 Experiments and analysis

Resnet-152 is assumed to be a function extractor that contains convolution building blocks, including shortcut connections in the proposed system in Fig. 1. It takes a 224 × 224 randomly cropped image sample and applies transformers to obtain 2048 feature vectors, each measuring 7 × 7 pixels. Without using the fully connected layer, to get the feature vectors from the last convolution block. During encoding, this enables the attention mechanism to be selective and concentrate on different image features. By using the attention framework to draw attention to the image’s most appropriate pixels. The context vector and which is the weighted sum of the Encoder’s output, is output by the attention network to concentrate on relevant parts from each decoding phase (features). During the training, perform the scheduled sampling at each stage. Concatenate that embedding through context vector it into single input to LSTM. The number of epochs is set to 69, which should be enough to train the model to generate informative descriptions, and the validation loss has reached its lowest minimum limit and is no longer decreased. The proposed model has a total of 23, 578, 721 trainable parameters. We present the dataset and evaluation metrics, quantitative and qualitative analysis in this section, that use in our experiments. Quantitative outcomes are based on the BLEU-(1 -4) matrix of n-gram, METEOR, CIDEr, and ROUGE-L ratings. Furthermore, our experimental setup and methods are described. The experimental outcomes are eventually explained in detail.

4.1 Data set

Availability of the benchmark datasets. We measure the performance of our proposed framework on two different benchmark datasets MS-COCO and Flickr8k dataset [16]. The MS COCO dataset includes 123, 287 images, including 82, 783 training images and 40, 504 validation images, each labelled with different five captions. For Flickr8k is a popular dataset that contains 8000 images. The training data is made up of 6000 images, while the test and development data are made up of 1, 000 images each. Humans have annotated 5 reference captions for each image in the Flickr8k dataset. The same datasets section used for evaluation containing 5000 for evaluation, was used for the test phase, and the same quantity was used for the validation process [6].

4.2 Evaluation metrics

Different indexes are used to compute the efficiency of the proposed framework to check the performance. Two different criteria should be met by evaluation metrics [30]. First, elevated scores can be obtained by captions which are considered highly by a human. Secondly, human beings should consider captions that obtain high scores to be successful. The captioning of images is often compared with the translation of languages or with the summary of text [1 , 35], which inspired the adaptation of metrics originally established for the assessment of language tasks [36 –38].

4.2.1 BLEU

Bilingual Evaluation Understudy (BLEU 1 - 4) is one of the first metrics used to evaluate image descriptions and is a common metric for machine translation performance. To prevent short sentences, it determines the geometric mean of n-gram precision scores multiplied by a BP (Brevity Penalty) as in Equation (9). BLEU score estimates for an image in our experiment, according to the NLTK (Natural Language Toolkit) execution [36]. $BLEU = BP \times exp (\sum_{n = 1}^{N} w_{n} log p_{n}),$ (9) where, p_n is geometric of the modified g-gram precision, N = 4 baseline of 4 gram is used in the original implementation, and W_n is weight, adding up to 1 and adjusted BP to 1.0 if a candidate’s length is just like any translation length as in Equation (10). $BP = {\begin{matrix} 1 & if c > r \\ e^{(1 - r / c)} & if c \leq r \end{matrix} .$ (10)

The length of the closet reference term is r, and the length of the candidate is c. The ranking behavior is more immediately apparent in the log domain, $log BLEU = min (1 - \frac{r}{c}, 0) + \sum_{n = 1}^{N} w_{n} log p_{n} .$ (11) In our baseline, we use N = 4, and uniform weights w_n = 1/N.

4.2.2 METEOR

Metric for the Evaluation of Machine Translation Output (METEOR) is another metric of machine translation [37]. It uses stemmers, WordNet [41] related terms, and tables of paraphrases to classify similarities among candidate sentences and reference phrases.

4.2.3 ROUGE

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of measures for evaluating textual summaries initially created [38]. A ROUGE-L variant is generally used for image captioning tasks, which measures F-measure based on the Longest Common Subsequence (LCS), i.e., a collection of words shared by two sentences that occur in the same order without requiring consecutive matches.

4.2.4 CIDEr

Consensus-based Image Description Evaluation (CIDEr) is a metric developed for evaluating image captions created automatically [39]. By conducting a Term Frequency Inverse Document Frequency (TF-IDF) weighting for each n-gram, it tests the correlation between the candidate sentence and a set of human-written sentences.

4.2.5 SPICE

Semantic Propositional Image Caption Evaluation (SPICE) is a metric developed for evaluating image captions tasks [40]. By computing an F-measure based on the candidate’s propositional semantic content and reference sentences described as scene graphs [42], it measures the quality of produced captions.

4.3 Quantitative analysis

To evaluate the hypotheses methodology, and employ evaluation metrics, including BLEU-(1 -4) [36], METEOR [37], ROUGE-L [38], CIDErD [39], and SPICE [40], to evaluate the proposed method and compare that to another state of the art the strategies. On the MSCOCO, present the results of our proposed attention-based sequence-to-sequence architecture in Table 1, and found that the findings outperformed the baseline models.

Table 1
BLEU 1-4, METEOR, ROUGE-L, CIDEr, and SPICE metrics compared to other methods using the MS COCO dataset

Models BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr SPICE

LSTM-A5[21] 0.73 0.56 0.43 0.32 0.25 0.54 0.97 -

Spatial Attention [20] 0.73 0.56 0.41 0.30 0.25 - 102.9 -

ARNet [43] 0.74 0.58 0.44 0.33 0.26 0.55 103.4 0.19

Attribute Driven Attn [44] 0.74 0.58 0.44 0.33 - - 104.4 -

Context-aware attention [14] 0.76 0.60 0.46 0.36 0.28 0.56 103.1 -

GCN-LSTM [23] 0.77 - - 0.36 0.27 0.56 113.5 -

GL-JSAM [25] 0.79 0.60 0.46 0.38 0.30 0.59 119.5 -

Attention Seq-to-Seq (our) 0.81 0.61 0.47 0.22 0.33 0.60 117.4 0.92

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
LSTM-A5[21]	0.73	0.56	0.43	0.32	0.25	0.54	0.97	-
Spatial Attention [20]	0.73	0.56	0.41	0.30	0.25	-	102.9	-
ARNet [43]	0.74	0.58	0.44	0.33	0.26	0.55	103.4	0.19
Attribute Driven Attn [44]	0.74	0.58	0.44	0.33	-	-	104.4	-
Context-aware attention [14]	0.76	0.60	0.46	0.36	0.28	0.56	103.1	-
GCN-LSTM [23]	0.77	-	-	0.36	0.27	0.56	113.5	-
GL-JSAM [25]	0.79	0.60	0.46	0.38	0.30	0.59	119.5	-
Attention Seq-to-Seq (our)	0.81	0.61	0.47	0.22	0.33	0.60	117.4	0.92

When compared to the CNN+LSTM baseline Spatial Attention, [20], LSTM-A5 [21], AICRL-VGA16 [30], ARNet [43], Attribute Driven Attention and Soft-Attention [28, 44], can see how the adoption of visual attention and powerful ResNet-152 based encoder and the robust LSTM based language model helps to increase the performance in most of the evaluation metrics. We assume that the proposed approach captures the spatial information related to the caption better than the attention-based models GCN-LSTM [23], and GL-JSAM [25]. The above illustrates how using image features improves output results. On all evaluation methods, our attention-based sequence-to-sequence framework outperforms the competition. As a result of our system, interacting visual and semantic meaning via an attention mechanism may aid in caption development. As shown in Tables 1, and 2, the achieved results of our proposed model are the height in most of the Evaluation metrics, based on the MS COCO and Flick8K datasets. On MSCOCO, evaluation scores included BLEU-1 0.81, BLEU-2 0.61, BLEU-3 0.47, and 0.33 on METEOR. For Flickr8k BLEU-1 0.68, BLEU-2 0.49, BLEU-3 0.41, METEOR 0.23, and 0.86 on SPICE. In terms of standard assessment measure, the suggested methodology is comparable to numerous state-of-the-art methods, suggesting that it can create more accurate and richer captions.

Table 2

BLEU 1-4, METEOR, ROUGE-L, CIDEr, and SPICE metrics compared to other methods using the Flickr 8k dataset

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
m-RNN[53]	0.58	0.28	0.23	-	-	-	-	-
AICRL-VGA16[30]	0.67	0.43	0.33	0.22	0.18	-	0.74	-
Soft-Attention [28]	0.67	0.44	0.30	0.19	0.18	-	-	-
Hard-Attention[28]	0.67	0.45	0.31	0.21	0.20	-	-	-
AICRL-ResNet50[30]	0.61	0.45	0.36	0.26	0.20	-	0.80	-
Attention Seq-to-Seq (our)	0.68	0.49	0.41	0.19	0.23	0.53	0.98	0.86

4.4 Qualitative analysis

This model’s key results are very insightful. The model produces descriptive and grammatical captions for several images. Figures 6 illustrates some promising outcomes on MSCOCOC and Flickr8k dataset. Although many captions are informative, some describe a scene that is entirely different from the one depicted in the image, and others are completely incomprehensible. Although it is impossible to distinguish between the two tasks performed by the network, may terminate the former as a failure in image recognition and the latter as a failure in text generation. Figure 7 depicts several examples of such failures.

Fig. 5

Examples of image captions generated by our proposed framework MS COCO dataset. This demonstrates that our proposed model can generate more reasonable descriptions (e.g., “a yellow train at the station”, “a man riding a bike on a street”) by using an attention-based sequence-to-sequence process.

Fig. 6

Examples of image captions generated by our proposed framework on Flickr8K Dataset. This demonstrates that our proposed model can generate more reasonable descriptions (e.g., “a person riding a motorcycle on a street”, “a little boy in a field with a frisbee”) by using an attention-based sequence-to-sequence process.

Fig. 7

Example of bad captions predicted by the model.

5 Conclusion and future improvements

In summary, this paper has proposed one particular approach for completing a capture generation task using an attention-based sequence-to-sequence framework that is based on Resnet-152 as a feature extractor that consists of convolution building blocks embedded with shortcuts connections. After that LSTM, a language framework was chosen as the decoder to generate the description sentence. In the meantime, when combined with a conventional encoder-decoder model, it generates captions in an attention-based manner. Experimental results including BLEU-1 0.81, BLEU-2 0.61, BLEU-3 0.47, METEOR 0.33, Rouge-L 0.60, and Spice 0.92 on the MS-COCO dataset demonstrate the model is capable of automatically producing strong captions for images. Increasing the number of datasets and adding more variation to the data will improve performance by allowing the model to learn more nuances. To train the model using labeled occluded object data and to deal with noisy images in various ways. Transform the model’s architecture; larger models require a large amount of data to learn their parameters. Also, with the simpler seq-to-seq attention-free models, however, you can get good results. We’d like to investigate methods for generating multiple sentences with varying content in the future.

Data and system availability

The NVIDIA GeForce GTX 1660Ti was used in all of the trials. This model was developed and tested on a GPU with 6GB of memory and an Intel i7-4790 CPU. PyTorch was used to complete the development process. Dataset collected from the authorized website of MS-COCO. The data used in this research is included in the paper.

Data: The data presented in the figures within this paper and other finding of this study are available from the corresponding authors upon reasonable request.

Code: The custom codes used to produce the results presented in this paper are available from the corresponding authors upon reasonable request.

Footnotes

Acknowledgment

This research is supported by the Fundamental Research Funds for the Central Universities. (Grant no. WK2350000002).

Conflicts of interest

The authors indicate that the publishing of this paper does not constitute a conflict of interest.

References

Vinyals

, Toshev

, Bengio

and Erhan

, Show and tell:Aneural image caption generator, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164, 2015.

Xu, , Kelvin, , Jimmy Ba, , Ryan Kiros, , Kyunghyun Cho, , Aaron Courville, , Ruslan Salakhudinov, , Rich Zemel, and Yoshua Bengio, , Show, attend and tell: Neural image caption geneation with visual attention, International conference on machine learning, pp. 2048–2057, PMLR, 2015.

You

, Jin

, Wang

, Fang

and Luo

, Image captioning with semantic attention, arXiv preprint arXiv:1603. 03925, 2016.

Donahue

, Anne Hendricks

, Guadarrama,

, Rohrbach

, Venugopalan

, Saenko

and Darrell

, Long-term recurrent convolutional networks for visual recognition and description, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2625–2634, 2015.

Yu , Niange , Xiaolin Hu , Binheng Song , Jian Yang and Jianwei Zhang , Topic-oriented image captioning based on order-embedding, IEEE Transactions on Image Processing 28(6) (2018), 2743–2754.

Karpathy

and Fei-Fei

, Deep visual-semantic alignments for generating image descriptions, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137, 2015.

Johnson

, Karpathy

and Fei-Fei

, Densecap: Fully convolutional localization networks for dense captioning, arXiv preprint arXiv:1511.07571, 2015.

Tan

Y.H.

and Chan

C.S.

, Phrase-Based Image Caption Generator with Hierarchical Lstm Network, , Neurocomputing 333 (2019), 86–100.

Fan

, Zhang

and Crandall

D.J.

, Deepdiary: Lifelogging Image Captioning and Summarization, Journal of Visual Communication and Image Representation 55 (2018), 40–55.

10.

Yuan

, Li

and Lu

, 3g Structure for Image Caption Generation, , Neurocomputing 330 (2019), 17–28.

11.

Chen

, Zhang

, Wang

, Zuo

, Li

and Yang

, Leveraging Unpaired out-of-Domain Data for Image Captioning, , Pattern Recognition Letters 132 (2020), 132–140.

12.

Guan

and Wang

, Repeated Review Based Image Captioning for Image Evidence Review, , Signal Processing:Image Communication 63 (2018), 141–148.

13.

, Shi

, Bai Xia

, Zhang

G.-S., Z.

and Dong

, Image Caption Generation with Part of Speech Guidance, , Pattern Recognition Letters 119 (2019), 229–237.

14.

Wang , Junbo , Wei Wang , Liang Wang , Zhiyong Wang , David Dagan Feng and Tieniu Tan , Learning visual relationship and context-aware attention for image captioning, , Pattern Recognition 98 (2020), 107075.

15.

Liu , Xihui , Hongsheng Li , Jing Shao , Dapeng Chen and Xiaogang Wang , Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 338–354, 2018, Springer, 2015, 133–140.

16.

Lin

T.Y.

, Maire

, Belongie

, Hays

, Perona

, Ramanan

, Dollá

and Zitnick

C.L.

, Microsoft coco: common objects in context, In: European conference on computer vision, Springer, pp. 740–755, (2014).

17.

Goyal

, Singh

, Tiwari

, Punia

and Kumar

, Intelligent skin cancer detection mobile application using convolution neural network, (7(SI)), Journal of Advanced Research in Dynamical and Control Systems (JARCDS) 11 (2019), 253–259.

18.

Long Chen , Hanwang Zhang , Jun Xiao , Liqiang Nie , Jian Shao , Wei Liu and Tat-Seng Chua , Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning, In CVPR, 2017.

19.

Kulkarni , Girish , Visruth Premraj , Vicente Ordonez , Sagnik Dhar , Siming Li , Yejin Choi , Alexander Berg

and Tamara Berg

, Babytalk: Understanding and generating simple image descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12) (2013), 2891–2903.

20.

Yang , Yezhou , Ching Teo , Hal Daumé III and Yiannis Aloimonos , Corpus-guided sentence generation of natural images, In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 444–454, 2011.

21.

Peter Anderson , Xiaodong He , Chris Buehler, , Damien Teney , Mark Johnson , Stephen Gould , Lei Zhang , Stephen Gould and Lei Zhang , Bottom-up and top-down attention for image captioning and visual question answering, In CVPR, 2018.

22.

Deeba , Farah , Fayaz Dharejo

, Muhammad Zawish , Fida Memon

, Kapal Dev , Rizwan Naqvi

, Yuanchun Zhou and Yi Du , A novel image dehazing framework for robust vision-based intelligent systems, International Journal of Intelligent Systems (2021).

23.

Li , Siming , Girish Kulkarni , Tamara Berg , Alexander Berg and Yejin Choi , Composing simple image descriptions using web-scale n-grams, In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, pp. 220–228, 2011.

24.

Naqvi , Nuzhat and ZhongFu Ye , Image captions: global-local and joint signals attention model (GL-JSAM), Multimedia Tools and Applications 79(33) (2020), 24429–24448.

25.

Zichao Yang , Xiaodong He , Jianfeng Gao , Li Deng and Alex Smola , Stacked attention networks for image question answering, In CVPR, June 2016.

26.

Dongfei Yu , Jianlong Fu , Tao Mei and Yong Rui , Multilevel attention networks for visual question answering, In CVPR, July 2017.

27.

Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan Gomez

, Lukasz Kaiser and Illia Polosukhin , Attention is all you need, In NeurIPS, 2017.

28.

Xu, Huijuan and Kate Saenko , Ask, attend and answer: Exploring question-guided spatial attention for visual question answering, In European Conference on Computer Vision, pp. 451–466. Springer, Cham, 2016.

29.

Zhang , Hongbin , Diedie Qiu , Renzhong Wu , Donghong Ji , Guangli Li , Zhenyu Niu and Tao Li , Novel model to integrate word embeddings and syntactic trees for automatic caption generation from images, Soft Computing 24(2) (2020), 1377–1397.

30.

Chu , Yan , Xiao Yue , Lei Yu , Mikhailov Sergei and Zhengkui Wang , Automatic image captioning based on ResNet50 and LSTM with soft attention, Wireless Communications and Mobile Computing 2020 (2020).

31.

Bengio , Samy , Oriol Vinyals , Navdeep Jaitly and Noam Shazeer , Scheduled sampling for sequence prediction with recurrent neural networks, arXiv preprint arXiv:1506.03099, (2015).

32.

Liu

, Zhu

, Ye

, Guadarrama

and Murphy

, Improved Image Captioning via Policy Gradient Optimization of SPIDEr, arXiv preprint arXiv:1612.00370, 2016.

33.

Elliot

and Keller

, Comparing automatic evaluation measures for image description, in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Short Papers, 2014, pp. 452–457.

34.

Kulkarni

, et al., Babytalk: Understanding and generating simple image descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12) (2013), 2891–2903.

35.

Yang

, Teo

C.L.

, Daumé III

and Aloimonos

, Corpusguided sentence generation of natural images, in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011, pp. 444–454.

36.

Papineni

, Roukos

, Ward

and Zhu

W.J.

, BLEU: a method for automatic evaluation of machine translation, in Proceedings of the 40th annual meeting on association for computational linguistics, 2002, pp. 311–318.

37.

Denkowski

and Lavie

, Meteor universal: Language specific translation evaluation for any target language, in 9thWorkshop on Statistical Machine Translation, 2014, pp. 376–380.

38.

Lin

C.Y.

, Rouge: A package for automatic evaluation of smmaries, in, Text summarization branches ot: Proceedings of the ACL-04 workshop, 2004, vol. 8.

39.

Vedantam

, Zitnick

C.L.

and Parikh

, Cider: Consensus based image description evaluation, in CVPR, 2015, pp. 4566–4575.

40.

Anderson

, Fernando

, Johnson

and Gould

, Spice: Semantic propositional image caption evaluation, in ECCV 2016, 2016, pp. 382–398.

41.

Miller

G.A.

, Beckwith

, Fellbaum

, Gross

and Miller

K.J.

, Introduction to WordNet: An on-line lexical database, International Journal of Lexicography 3(4) (1990), 235–244.

42.

Johnson

, et al., Image retrieval using scene graphs, in CVPR, 2015, pp. 3668–3678.

43.

Chen , Xinpeng , Lin Ma , Wenhao Jiang , Jian Yao and Wei Liu , Regularizing RNNs for caption generation by reconstructing the past with the present, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7995–8003, 2018.

44.

Chen

, Ding

, Lin

, Zhao

and Han

, Show, observe and tell: attribute-driven attention model for image captioning, In: IJCAI, pp. 606–612, (2018).

45.

Ye , Zhongfu , Rashid Khan , Nuzhat Naqvi and Shujah Islam

, A novel automatic image caption generation using bidirectional long-short term memory framework, Multimedia Tools and Applications (2021), 1–26.

46.

Mishra , Santosh Kumar , Rijul Dhir , Sriparna Saha and Pushpak Bhattacharyya , A Hindi Image Caption Generation Framework Using Deep Learning, Transactions on Asian and Low-Resource Language Information Processing 20(2) (2021), 1–19.

47.

Islam

M.S.B.

, Khan

K.R.

, Islam

M.M.

and Ye

, Single and two-person (s) pose estimation based on R-WAA, Multimedia Tools and Applications (2021), 1–14.

48.

Islam Khush Bakhat

M.S.

, Rashid Khan , Mansoor Iqbal , Mattah Islam

and Zhongfu Ye , Action recognition using interrelationships of 3D joints and frames based on angle sine relation and distance features using interrelationships, Applied Intelligence (2021), 1–13.

49.

Khan , Naveed Ahmad , et al., Application of Euler Neural Networks with Soft Computing Paradigm to Solve Nonlinear Problems Arising in Heat Transfer, Entropy 23(8) (2021), 1053.

50.

Kulkarni , Girish , Visruth Premraj and Vicente Ordonez , Babytalk: Understanding and generating simple image descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12) (2013), 2891–2903.

51.

Aggarwal

, Alshehri

, Kumar

, Sharma

, Alfarraj

and Deep

, Principal component analysis, hidden Markov model, and artificial neural network inspired techniques to recognize faces, Concurrency and Computation: Practice and Experience 33(9) (2021), e6157.

52.

Aggarwal

and Kumar

, Image surface texture analysis and classification using deep learning, Multimedia Tools and Applications 80(1) (2021), 1289–1309.

53.

Mao

, Xu

, Yang

, Wang

and Yuille

A.L.

, Explain images with multimodal recurrent neural networks, University of California, Los Angeles, 2014.

54.

Akilan

, Thiagarajan

, Venkatesan

, Thirumeni

and Chandrasekaran

S.G.

, Quantifying the Impact of Complementary Visual and Textual Cues Under Image Captioning, In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 389–394). IEEE, (2020, October).